Strange Buffer Issue

[asarghmatey]

I do a lot of live instrumentation, both midi and organic. Everytime I open a Live Set there is s some latency, mostly noticeable if I play fast. It seems to be about the same with midi and with live audio. If I change the buffer size the latency goes away. Even if I increase the buffer size it fixes it (so long as I don't have too high).

When performing, I'm in the habit of switching between 64 and 128 each time I load the next set. This ads extra down time that I'd like to eliminate if possible.

Setup:
MacBook Pro 10.14.2 Mojave
2.2 GHz Intel Core i7
16 GB 2400 MHz DDR4
Zoom TAC-8 Audio Interface
Ableton Live 10.1.5

Thanks in advance for any help!

[yur2die4]

If you have a set with zero plugins, you will have minimal latency for your buffer settings.

When you add plugins, some have a higher processing tax than others and will result in added latency. Common latency inducing plugins are high quality reverbs, guitar fx processors, mastering plugins, vocal effect and tuning plugins. You can see if a plugin adds latency by looking at the lower edge of the Live screen while hovering a mouse over various devices.

[asarghmatey]

If you have a set with zero plugins, you will have minimal latency for your buffer settings.
...

I'm well aware of this, but that doesn't explain the issue I'm having. It certainly wouldn't explain the fix. Also some of my sets aren't using plugins at all.

[asarghmatey]

I'm realizing that I could probably simplify my explanation of the issue, as it's an odd one. Here is what happens.

Open live set one. Notice latency. Switch buffer to 64, latency goes away.

Open live set two. Notice latency. Switch buffer to back 128, latency goes away.

Open live set three. Notice latency. Switch buffer back to 64, latency goes away.

It's the switching of the buffer size that makes the latency go away, not the actual buffer size.

What could cause this?