So I'm recording a hybrid drumset with acoustic drums and MIDI from a Roland SPD-S Sampling Pad.
I have to have the buffer size quite high, since I want to be sure not to have dropouts, it's a lot of tracks in 96kHz, my laptop is not the newest and I don't need software monitoring.
And that's fine for the audio material, because the buffer size doesn't affect the timing of the recorded material, however obviously that isn't true for the MIDI notes. The picture shows a recording of me hitting the acoustic snare and a pad on the SPD-S at the same time which ends up horrendously out of sync!

What can I do? Yeah I could lower the buffer size but this will only lower the offset and not solve the problem. Also I might run into performance trouble. Quantizing is also not an option because then the MIDI notes will be on the grid but still not in sync with the audio material, also I don't always play with click.
Gear: Focusrite Clarett 4 Pre Interface, Macbook Pro from 2015, Ableton 10 Suite newest version.