I can work at 256 or 512 with no problems. 128 or 64 is not reliable. I have a Midas XL48 on internal clock driving the clock on the Apollo. That's all my digital gear.
I did some experiments running a large session (for me): 12 inputs recording, 12 backing tracks playing back, and 4 headphone mixes. Below are screenshots of the Resource Monitor at 256, 128, and 64 samples of buffer. I can see that the CPU usage is increasing with decreasing buffer, so I guess I should upgrade my CPU if I want to run at 64 samples buffer without any dropouts.
256
128
64