Disk Cache changes randomly after freeze

[[email protected]]

Attached below, are two screenshots.
The first, I start printing a mix (CONSTANT 9093, 9172- 2018.7 with High Sierra has been a nightmare over here) Disk Cache shows it's 39% full. I start printing and no 20 secs in, protools freezes with a 9093. I select the partial take and move it to the trash. Disk Cache now shows 81%. Anyone has some thoughts on that?

[Darryl Ramm]

When/how does the disk usage meter change? You are scanning through the entire session reading more content into the cache to produce the print so it growing as you print is not a surprise. What happens if you just play the entire session though in real-time in one go vs. printing it.

Oh hang on you mean print to track or bounce to disk?

Hopefully the bounce itself is not ending up in the cache, so I would hope deleting it makes no difference to cache utilization (and the Pro Tools disk cache would need to be watching that file to see if its deleted from the filesystem to know to delete it's own cache copy, likely a really bad idea).

Print to track, should hopefully up in the cache and it may be the cache meter is lagging reality for a while.

---

But in general with caches like this I would not get overly worried about what is going on. You can't tell how theses things are supposed to work, how they evict or garbage collect data. Lots of sensible approaches to doing these things will have them fill up and not start evicting stuff until pretty full.

[[email protected]]

Thanks, Daryl. The cache just gets to 100% eventually whether I print to not- which is what worries me...

Originally Posted by Darryl Ramm

Yep. You are scanning through the entire session reading more content into the cache to produce the print. What happens if you just play the entire session though in real-time in one go vs. printing it.

Hopefully the print itself is not ending up in the cache, so I would hope deleting it makes no difference to cache utilization (and the Pro Tools disk cache would need to be watchign that file to see if its deleted from the filesystem to know to delete it's own cache copy, likely a really bad idea)

---

But in general with caches like this I would not get overly worried about what is going on. You can't tell how theses things are supposed to work, how they evict or garbage collect data. Lots of sensible approaches to doing these things will have them fill up and not start evicting stuff until pretty full.

[Darryl Ramm]

And a trivial test confirms bounces are not hitting the cache. Set the disk cache really small. Record a signal gen to a track until the cache is 25% full and then bounce (real time or fast) and nothing increases in the cache as I'd hope.

[Darryl Ramm]

Originally Posted by [email protected]

Thanks, Daryl. The cache just gets to 100% eventually whether I print to not- which is what worries me...

I don't know what other people see, I have such huge cache relative to my usually smallish session sizes I don't see it anywhere near full. And I'm normally not watching meters anyhow.

I suspect it's really hard to see what is going on with the tools given. The % cache occupancy is likely not that useful, you want it to be high so stuff is cached. Stats like the rate of data pages into and out of the cache and similar data would give more useful information for serious debugging... but I expect Avid wants to keep it simple. Along the lines of just letting folks know if the cache fills that it might help to increase the size of buy more DRAM.

BTW this type of thing needlessly worries people with computer memory... noting that say Windows or macOS reports very little free memory... when in reality that's the demand paging system doing what is expected and there might be lots of pages the OS can free at very low cost anytime it needs them. And again other rate based metrics may be more useful.

[[email protected]]

Maybe I'm not explaining clearly-
First, I'm looking at the meters because things aren't stable and I'm trying to find a pattern. (there isn't one- sometimes a restart just fixes things for a while and sometimes it doesn't. Sometimes the same session will play fine, and sometimes it's days full of 9093, 9172 and other gems)
Secondly, I'm explaining that when I open a session and the cache is loaded, let's say 39%- after a few plays, it'll just make its way to 100% without adding ANY new info to session. That's not right and something is getting calculated wrong...

Originally Posted by Darryl Ramm

I don't know what other people see, I have such huge cache relative to my usually smallish session sizes I don't see it anywhere near full. And I'm normally not watching meters anyhow.

I suspect it's really hard to see what is going on with the tools given. The % cache occupancy is likely not that useful, you want it to be high so stuff is cached. Stats like the rate of data pages into and out of the cache and similar data would give more useful information for serious debugging... but I expect Avid wants to keep it simple. Along the lines of just letting folks know if the cache fills that it might help to increase the size of buy more DRAM.

BTW this type of thing needlessly worries people with computer memory... noting that say Windows or macOS reports very little free memory... when in reality that's the demand paging system doing what is expected and there might be lots of pages the OS can free at very low cost anytime it needs them. And again other rate based metrics may be more useful.

[Darryl Ramm]

How big is your disk cache set to? How much total memory on your system?

[[email protected]]

Originally Posted by Darryl Ramm

How big is your disk cache set to? How much total memory on your system?

I play around with the size, depending on mix session size.
Currently, around 5GB. System has 24GB of ram.

[Darryl Ramm]

Some comments, not sure they really help.

So playing with this tonight I occasionally saw the disk cache "zoom along" from say 70% or 80% to 100% in one quick smooth motion taking what seems one or two seconds, the meter stayed green. I was recording several tracks being fed from as signal generator. Onto say a session that had content in it and opened with 50% cache use. With a disk cache of 512MB the cache was filling at about 1% every several seconds, then it just takes off. I saw this twice out of ten or more tests. So sure I'm willing to believe that something occasionally gets funky with the cache meter and it might shows more cache in use than there is data to cache. I have no other problems actual problems using Pro Tools. The meter has an occasional GUI glitch when closing the session it leaves a few pixel green or amber vertical bar in the meter, just where it's sometimes not redrawing correctly what looks like the right hand edge of the previous meter bar. If I can reproduce these more I'll throw them over the fence to Avid.

The other thing I played with was just peeking at Pro Tools disk IO with the fs_usage utility (derived from the low level dtrace instrumentation). Works like a charm, I can watch the Pro Tools process open all the media files and use non-cached asynchronous IO (aio_read()) to fill its disk cache.... just like I'd expect for a high IO-performance app doing its own cache management. And then see Pro Tools use pwrite() to write new audio content out.

It's not something normal users are going to want to play with, and it requires disabling SIP (System Integrity Protection). If I was really trying to debug something I'd might play with dtrace more directly. With enough playing that could capture the meter zooming along faster event and the disk IO might coordinate that going on, or might not.

I'm running 2018.10 on a maxed out 2017 MBP, 16 GB on macOS Mojave.

[[email protected]]

Thanks for spending time with it, Darryl.
One of the weird things is that I could open the session, it'll stop at around 50% (green) then play a bit, print NOTHING, (nor create ANY new audio) and it'll just make its way to 100%

Originally Posted by Darryl Ramm

Some comments, not sure they really help.

So playing with this tonight I occasionally saw the disk cache "zoom along" from say 70% or 80% to 100% in one quick smooth motion taking what seems one or two seconds, the meter stayed green. I was recording several tracks being fed from as signal generator. Onto say a session that had content in it and opened with 50% cache use. With a disk cache of 512MB the cache was filling at about 1% every several seconds, then it just takes off. I saw this twice out of ten or more tests. So sure I'm willing to believe that something occasionally gets funky with the cache meter and it might shows more cache in use than there is data to cache. I have no other problems actual problems using Pro Tools. The meter has an occasional GUI glitch when closing the session it leaves a few pixel green or amber vertical bar in the meter, just where it's sometimes not redrawing correctly what looks like the right hand edge of the previous meter bar. If I can reproduce these more I'll throw them over the fence to Avid.

The other thing I played with was just peeking at Pro Tools disk IO with the fs_usage utility (derived from the low level dtrace instrumentation). Works like a charm, I can watch the Pro Tools process open all the media files and use non-cached asynchronous IO (aio_read()) to fill its disk cache.... just like I'd expect for a high IO-performance app doing its own cache management. And then see Pro Tools use pwrite() to write new audio content out.

It's not something normal users are going to want to play with, and it requires disabling SIP (System Integrity Protection). If I was really trying to debug something I'd might play with dtrace more directly. With enough playing that could capture the meter zooming along faster event and the disk IO might coordinate that going on, or might not.

I'm running 2018.10 on a maxed out 2017 MBP, 16 GB on macOS Mojave.