Back to Disk cache. I'm assuming you tested over the session size. All I want to make sure is you have looked at the Disk Performance meter and opened a session and watched that session fully load into memory (Disk meter bar go all green). Then one it is fully loaded start using Pro Tools.
The question of memory... well if enabling really large disk cache helped you (to hold a really large session) or you were running out of memory for VIs etc. then sure its' a no-brainer to increase the memory size. But is there something more subtle going on here? With say mismatched DIMMs,... possible, but I doubt it. If you were seeing other errors, crashes, then I'd worry about the more. Unless you need more for other reasons, messing with the memory right now would be near the end of my list.
I've been turning Dynamic Processing off by habit, after very early issues when this was introduced, but now on 2019.5 I've show to myself it can help so its staying on for now.
Since you came from Pro Tools 11 I want to mention Track Freeze/Commit again. I would sure be trying that out as much as I could if are having -9171 errors trying to get this Project done.
I have not worked on a Mac Pro Cheesegrater in a while (sometimes wish I had one again) so I am not up on GPU recommendations etc. for them.
BTW I want to mention this recent thread
http://duc.avid.com/showthread.php?t=404737 where James Welch reported great improvements by moving to Mojave from Sierra on his Mac Pro Cheesegrater (Mac Pro 5,1, six core, Radeon R9 280X). I'd look that over (ignore my useless comments there) and consider trying what James did. I am curious if you also see the CPU meter behavior with closing mix and edit windows that he saw on Sierra, if so I'd run not walk to test Mojave
