Opening Session with lots of audio files takes too long

[HugoC]

We are a film post production company and we have been having a problem for some time with sessions containing a lot of audio files. We create these sessions by importing all the rushes of a film in order to proceed to auto-conform later on. This is a lot of data.

We noticed that sessions took much longer to open from version 2020.9 onwards, for sessions created in version 2019.12. To give an order of magnitude, for a session containing 760 GB of rushes, it takes 1 minute to open in 2019.12 and 12 minutes in 2020.9 and beyond (we tested up to version 2021.12 and found the problem with each version).

The problem is even wider as importing audio files takes much longer than before. The files are simply added to the session, not copied.
We conducted tests in the same versions of Pro Tools, importing 160 GB of rushes. In 2019.12, importing audio files takes about 15 minutes, including the time to stamp the files with a new unique ID. In 2020.9 and beyond, we stopped the processes after 3 hours before they finished, so the files were not even imported into the session.

The files are stored in a USB 3.2 connected hard drive, and all the tests were done on this same drive. It is a Seagate BarraCuda 4Tb HDD (ST4000DM004). We work with these drives all the time, they are very fast and we never have any speed problems with them.

We ran these tests on :
- Version 2019.12: Mac Pro 5.1 with MacOS 10.12.6, 2x2.4Ghz Quad, 24GB RAM, GC ATI Radeon HD 5770 1024MB
- Version 2020.9 : Mac Pro 5.1 with MacOS 10.12.6, 2x2.4Ghz Quad, 48 Go RAM, GC Radeon RX 560 4Go
- Version 2021.12: Mac Pro 6.1 with MacOS 10.15.3, Intel Xeon E5 2.7 Ghz 12Core, 64 GB RAM, GC AMD FirePro D700 6GB, HDx 3

We would like to know if other film post-production studios are experiencing this kind of problem and if any solutions have been found, and also to alert Avid to the problem, so that a solution can be found.

Thank you very much and have a nice day!

[BScout]

Left out an important part: where are all these files stored? NAS, SAN, thunderbolt connected drive, internal drive?

Pro Tools changed the way they index and catalogued files between your versions. There are some tweaks that can be done depending on storage location.

[HugoC]

Indeed, this is important information. The files are stored in a USB 3.2 connected hard drive, and all the tests were done on this same drive. It is a Seagate BarraCuda 4Tb HDD (ST4000DM004). We work with these drives all the time, they are very fast and we never have any speed problems with them.
I will edit the original post with this information.

[albee1952]

As part of troubleshooting, I would take a session that you have been using and copy it to an internal drive(even the system drive will be okay if its an SSD) and see what happens

[The Weed]

AFAIK, that's a 5,400 RPM. Was Disk Cache set to something in the older versions of Pro Tools and not the later versions?

[Darryl Ramm]

Originally Posted by The Weed

AFAIK, that's a 5,400 RPM. Was Disk Cache set to something in the older versions of Pro Tools and not the later versions?

Yes. Just not a drive I'd be doing anything performance critical on. That under spec slow drive may make differences between Pro Tools versions here much more significant.

And back to the OP

What file system is on the drive?

Lots of potential things to think about with troubleshooting. Disk Cache size already mentioned. And it's not just the size of the cache... but you should look at the amount of cache in use before and during the import and try to see if it's low and therefore maybe having to evict content from the cache. You will only see stuff load into the disk cache if you import clips to tracks. With content not moved you would hope only the header block is rewritten - It would not surprise me that Pro Tools might do some inefficient handling here. And just try with cache "normal" to see if it makes any difference.

How come you are importing this stuff on an external 5400 RPM drive and then not moving it during import? e.g. into the session folder on a fast SSD?

Have tried trashing prefs before each import and starting with sessions in exactly the same state (ie. totally empty, not from a template) before import?

You mention audio but is there video on the session and/or in the import?

If this is important you maybe also want to file an incident with Avid customer support, have them look over this with you, reporting it on DUC is unlikely to get something looked at by support/tracked as an incident or escalated to bug. If you already have an incident number posting it here may help get attention on it.

[audiolex1]

Originally Posted by Darryl Ramm

Yes. Just not a drive I'd be doing anything performance critical on. That under spec slow drive may make differences between Pro Tools versions here much more significant.

And back to the OP

What file system is on the drive?

Lots of potential things to think about with troubleshooting. Disk Cache size already mentioned. And it's not just the size of the cache... but you should look at the amount of cache in use before and during the import and try to see if it's low and therefore maybe having to evict content from the cache. You will only see stuff load into the disk cache if you import clips to tracks. With content not moved you would hope only the header block is rewritten - It would not surprise me that Pro Tools might do some inefficient handling here. And just try with cache "normal" to see if it makes any difference.

How come you are importing this stuff on an external 5400 RPM drive and then not moving it during import? e.g. into the session folder on a fast SSD?

Have tried trashing prefs before each import and starting with sessions in exactly the same state (ie. totally empty, not from a template) before import?

You mention audio but is there video on the session and/or in the import?

If this is important you maybe also want to file an incident with Avid customer support, have them look over this with you, reporting it on DUC is unlikely to get something looked at by support/tracked as an incident or escalated to bug. If you already have an incident number posting it here may help get attention on it.

USB 3.2 is meaningless if the drive is slow.

There are plenty of USB 3 drives with faster HDD drives that will perform way better.

Optimally, moving the SSD drives will change everything.

[HugoC]

The file system is Mac OS extended (journaled).

As for the files, they are stored on the hard disk and we create the session on the same disk. We do not import video into these sessions. The audio files are not copied but just linked into the session. We don't want to copy or move these files, as this is usually several hundred GB of data, and moving/copying them would be very time consuming. We use these sessions to stamp the files with a unique ID so that the whole post production chain is sure to relink the right media to the right place, and also to auto-conform the rushes from the shoot.

Some in this thread mention that the discs used are too slow, however as I said in the original post, in earlier versions of Pro Tools we didn't have these problems with import speeds etc. So the problem is not with our disks.

We will test the solutions you give, but moving these sessions to internal SSDs or similar solutions would involve a big change in our workflow and significant costs for possible disk purchases, so we are hoping for a correction from avid.

[Darryl Ramm]

That it worked OK before is unlikely to get anybody’s sympathy here when you are running on such a slow underspec drive. You are just very likely to be sensitive to changes that you would not be on a modern SSD. And you won’t know until you do more investigation.

Nobody from Avid is likely to be going to “correct” anything here, nobody has a clue what to correct. Avid staff may not even read this, this is not Avid customer support, it’s a user forum, if you want to reach support you need to file a support case. And the first thing they will likely note is you are running on an under spec audio HDD and likely ignore the case. So if nothing else reproduce the problem on a modern SSD and then file a support case with Avid.

Good SSDs cost ~$150/TB, hard to see that being a huge burden. And should be no burden at all, at least for testing/troubleshooting to see if that helps.

[TOM@METRO]

^^^ this ^^^