A wild guess......
since it goes between the native plugs (leaving the DSP world and coming back), which puts them into different "processing streams" so to speak, they get handled on a different core (or virtual core?), whereas when all the native processes happen in serial, they are in the same stream and are concentrated/ confined to the same core?
I have no idea what I am talking about, does it show???!!?