Some things to consider:
1-Having "power machine here with 32 GB Ram, upper class Intel processor, SSDs" means nearly nothing because these specs, while fine at first glance, might be worthless if any hardware isn't compatible(I know this has gotten much better in recent years, but posting a Sandra report is way more revealing)
2-Try freezing your Kontact track when you need to run a low buffer(immediately unfreeze once you can go back to a higher buffer).
3-running at 96K might allow you to use a slightly higher buffer setting(depending on your interface connection) so maybe try not going as low as you think you need
4-(don't hate me for this one) Would the world come crashing down if you did your recording at 48K and then used SRC to deliver the 96K product?(would anyone but you notice)
I mention #4 strictly due to my own experience with 96K(99% of my work is happy at 48K). I had a client bring in a 12 song CD project that was recorded at another studio(at 96K) and it brought my system(considered pretty potent at the time) to its knees. I ended up converting everything to 48K and delivered finals that made the client very happy. As it turned out, the recordings were so poorly done, that SRC was the absolute least important of the issues I had to deal with. Turns out the recording engineer was a really good sax player...........
