NAS users (ISIS, NEXIS, SNS owners): you PT performance experience?

[DLpres]

I'm supporting a facility running 12 PT bays (doing foreign language dubbing, think iMacs in rooms, not sound stages). They eagerly wanted centralized storage that everyone worked directly from, and we ended up choosing a Small Tree shared storage system several months ago, based on their PT experience and competitive price point.

After plenty of troubleshooting and ironing, the system is deployed, stable and everyone is working directly off the storage. However, we're seeing sporadic performance issues: PT would beachball for 1-10 seconds. It usually happens after hitting record, stop, save, or render.

1. What has been your experience, as far as responsiveness? Have you experienced such performance "hiccups"?

2. Do you know any PT / NAS engineering gurus in the US that could provide consulting or 'tier 3' support? Ideally someone who helped develop PT, has hands-on experience debugging PT code, who can analyze SMB/AFP networks stacks, and help figure out the bottleneck.

It may be worth spending hundreds of $ on that before I tell the owners "sorry, gotta shell out $50K on a beefier system and hope it works better this time around."


I can give a whole lot of detail about the system's configuration, but wanted to keep the initial post short.

Thanks!

[Darryl Ramm]

I have to ask... Disk Cache turned on and large relative to session content? Dedicated Gbit Ethernet network for storage?

[mtung]

Hi!
I'm running a multi room facility as well using a hybrid setup (Infortrend iSCSI RAID appliance with Studio Network Solutions software; globalsan iSCSI initiator and SANmp clients). In my experience, the only time I get beach balls is when I forget to turn on the "iSCSI writethrough cache" in the SANmp client. And it only happens when recording and I press stop. These setups are sometimes complicated and don't alway work as expected. It would be possible to try the SNS globalsan initiator on one machine and see if it behaves better.

[DLpres]

Yup: dedicated, physically separated, non-internet 1GbE network to all the PT machines. Running off a new low-latency switch with trunked dual 10GbE headend connections to the NAS server.

Disk Cache is on, green and doesn't help.

NAS functions normally otherwise. It's been very hard to diagnose because the behavior literally comes and goes, we just know it's facility-wide. So users will have good days and bad days, and it's not correlated to the work load that day.

[Inglesworth]

We have 6 PT rigs running large mixes, records, and sfx edit sessions including videos from a NAS system. We are using the SNS EVO. It works pretty well. It had it's bumps in the road in getting it set up and ready to use constantly. But it's been rocking straight through many sessions for over a year now.

There are several things to consider in diagnosing your issue. Specifically for this reason, it makes a lot of sense to go with something like the VEO, rather than any old NAS storage box. It has it's own OS and browsing system, which allows for users to log in. I think this helps the system itself keep track of who is accessing which files at any given time. Like maybe too many users are trying to access certain directories all at once, and the NAS box doesn't know quite how to handle it? ALso, it's a good idea to make sure that the NAS box and all of your computers are accessing the same exact time server so that the time is the same on all machines if possible. This could potentially be an issue if one machine sees something that was created in the future and doesn't know how to handle it either. That actually was a cause of an issue where one of our systems would get the beach ball and spin a few seconds every time the system did an auto save. Getting the times figured out helped a good deal there.

[Darryl Ramm]

Originally Posted by DLpres

Yup: dedicated, physically separated, non-internet 1GbE network to all the PT machines. Running off a new low-latency switch with trunked dual 10GbE headend connections to the NAS server.

Disk Cache is on, green and doesn't help.

NAS functions normally otherwise. It's been very hard to diagnose because the behavior literally comes and goes, we just know it's facility-wide. So users will have good days and bad days, and it's not correlated to the work load that day.

I would hope that your vendor could help diagnose a system wide problem like that.

Comes and goes... one thing there is check any backup/archive jobs, to this or other systems on the same LAN. which I expect you have. But I've found some pretty interesting rogue stuff running on storage systems.

iSCSI initiator stuff was already mentioned but what protocol does your Spanning Tree box already support? Is that a ZenStor box? I'm not sure they support iSCSI.

SMB?... I hope you are not running CIFS/Samba, that would be the last choice for live storage that was needed to perform.. Have you tried AFP (which you mention as well) and NFS to test if the problem is tied to protocol?

I'd also think about doing simple testing to exclude whole layers of stuff, like maybe run a 10GbE (Thunderbolt adapter on the Mac if needed) as direct from the SAN box as you can, stick it in the machine room next to the SAN if needed, and see if that is also affected while all the rest of the system is struggling.

Sorry I don't have leads to network folks in LA. I would hope your vendor does even if they are struggling to help you.

[Inglesworth]

Originally Posted by DLpres

Yup: dedicated, physically separated, non-internet 1GbE network to all the PT machines. Running off a new low-latency switch with trunked dual 10GbE headend connections to the NAS server.

Disk Cache is on, green and doesn't help.

NAS functions normally otherwise. It's been very hard to diagnose because the behavior literally comes and goes, we just know it's facility-wide. So users will have good days and bad days, and it's not correlated to the work load that day.

Also in thinking about this, your NAS box is hooked up 10GbE to the switch. I think one issue you might have, is can that switch output 10Gb through multiple 1Gb lines all on the same subnet? Does the NAS box have any other available ports on it? Even 1Gb lines. Running some of your systems on different ports might lighten . up the traffic a bit.

I would also like to edit something I mentioned earlier about the Evo and it's browsing system. The software on it is very clunky, and I would rather use the finder and treat them as network connected drives mounted on the desktop. But we do use some of the SAN functionality, and I haven't found a convenient way to mount the SAN volumes without it.