Audio continues playing while PT5.3.1 freezes

[o2post]

We are running a 40+ channel mix (for the first time in this set-up) on a TDM Core card in a 800Mhz G4 with 512MB RAM allocated to Pro Tools HD 5.3.1 on OS 9.2.2. We are reading from an internal IBM 120GB AV rated drive with 8MB cache. We have never had this problem before with fewer track counts.
The Pro Tools finger-tapping hand comes up randomly, the operator loses control of the software and audio mysteriously continues to play. Pro Tools then becomes inoperable even to the point of a force quit, and the computer must be hard booted. The problem keeps occuring after multiple reboots/allocating more RAM to PT/and 'zapping' the PRAM.

Has anyone EVER encountered this? Any recommendations on where to look in the Digidesign support world?
We would appreciate any suggestions. This ofcourse comes about with a big deadline looming...but when doesn't it, right?

Thanks,
SM