NAS- use SMB or NFS shares to connect?

[lucienpalmer]

Curious to see what other people are using SMB or NFS to connect from a Mac to your server? I have a 64TB server based on TrueNAS (free BSD). Been running it for 2 years, and mostly happy with it, but always a little disappointed that file transfers never hit close to the full 10Gb connection. The main 'work' drive on it are a pair of enterprise SSDs in striped RAID.

I have been using NFS shares. Large files (ie 20 GB video), copy at about 450 MB/sec, which is half what it should. I decided to experiment with SMB shares this week, and big files copy insanely fast, 900 MB/sec and over.

I was ready to switch everything to SMB, but then I tried opening a Pro Tools session. A whole film takes 5-10 min to open. Maybe SMB can't handle the tiny files as well and large? The same film opens over NFS in about 45 seconds.

My big questions is wether or not to use SMB or NFS for shares between the server and the user. In my tests, there seem to be trade offs for each, so I was curious if anyone else had an option of SMB vs NFS for Pro Tools.

SMB
-extremely fast large file copy (900MB/sec+)
-extremely slow session open (5 min for a feature film)

NFS (asynchronies writes on)
-slower speed large file copy (450MB/sec average)
-fast session open (45 sec for the same feature film)

I'm not seeing any difference in playback or stem recording. They both seem to be equally stable.

[Darryl Ramm]

How exactly are you measuring file copy/write times? Is that copy to or from the client of copy across the server mount?

I would be careful of any async mounted volumes, even with any performance hit... which should be less with high-performance SSD. I know that might not be a popular opinion. The data you are writing is presumably precious. At a minimum I'd want the server and network gear and ideally client on a UPS. And disk cache if it's not being hammered should help offset using async/but keep critical writes the Pro Tools thinks have hit the disk actually do so. And SMB uses async writes by default.

There are multiple tuning parameters here, increasing NFS block size may help sequential performance.

nfsstats may give clues what is going on, esp. with memory/cache info.

If you are mostly worried about sequential file/big block I/O performance striping won't help you. And helps reduce systems reliability. But I assume the session open time is really the more important goal here?... but also how are you measuring that? esp. what is your disk cache size... I'd normally measure the time it then takes with a large disk cache size to load the cache (i.e. the meter turns green) Along with some sanity tests like save/exit times are not awful. Auto backup works OK (if you use that).

My preferred way of working with NAS is to stage stuff off the NAS to local SSD and then restage back to the NAS (esp. in a zip format so it can't be run from the NAS or accidentally mangled). I know that won't work for everybody, especially with video.

Thinking about this stuff now as I need a SSD/server on the net here, but it will likely be NFS for a lot of personal history/bias reasons and most of the use here will not be Pro Tools.

[BScout]

I run SMB. Specifically SMB 3 multichannel.

Read/Write speeds are 1500MB/s read, 1100MB/s write (this is 8 x 18TB RAID6) for the 5GB file test (blackmagic disk speed test; the 1GB test clocks in at the same for write but 1380MB/s for read)

Pro Tools sessions open nearly as fast as on my local drive (which of course clocks in much faster at 5400MB/s read, 6730MB/s write as they do on the larger Apple M processor drive options.)

Pro Tools is clearly the bottleneck (otherwise the local would open even faster.) Usually I work on my local and then archive to the NAS (though Soundminer libraries, sample libraries, etc. are only on the NAS.) Today I had to go into some completed episodes from the archive for a studio. 5 episodes, each had about 5000 files or so (hour long programs) plus picture all opening off of the NAS alone. It was definitely under 30 secs each time.

Though I'm running multichannel SMB, that really doesn't apply to my main system using the NAS as the limiting factor is the drive array speed now and not the connection. (Theoretically I could be at 3100+MB/s read and write with the connection type so clearly not hitting that.) I have dual NAS units that are all-flash, 12 drive systems of the same type that do those numbers at a different studio so it is possible but the capacity on those is far less.

Checking my smb.conf, I have asynchronous i/o enabled, sendfile enabled, aio read size 1, aio write size 0, and for all the shares, aio read size on sequential = 4097, oplocks enabled, strict sync = no
(and, yes, everything is on backup power supplies)

Should mention if anyone is looking at my hardware, the three NAS mentioned above are using ext4 (QTS operating system.) The 4th recent one runs QuTS Hero operating system (which is ZFS based.)

[Darryl Ramm]

BScout: Out of interest who's server hardware/software are you running and are you happy with them? Edit: And now I'm on a computer I see the QNAP TVS-872XT 148TB TB3.

[BScout]

Originally Posted by Darryl Ramm

BScout: Out of interest who's server hardware/software are you running and are you happy with them?

QNAP
I had 1 backplane die within 6 months of owning it and they fedex box, repair, return all within 24 hours (all drives stayed with me.) That was years ago. Haven't had an issue since. On the flash system (dual -- one backs up to the other), recently had a virtual volume conversion issue where the volume wouldn't mount after conversion (thick to thin.) Over 3 year old system, a developer from them (escalated beyond customer service quickly) had it working in hours/mounted. (it was a real weird issue -- I could confirm the volumes were fine and the drives were fine but couldn't get it to mount through ssh. At some point with virtual volumes it starts to get a little worrisome to tinker more -- especially when you know the data is intact so I called them in.) I don't know if it was a system OS bug that I ran into but the fix time (and no charge on any of this, of course) made me feel confident with them. That flash system has 3 systems reading and writing Pro Tools to it plus carrying terrabytes of samples that are read from 2 systems (they are duplicate rigs.) Those are all on a 10Gbe switch plus the 2 duplicate rigs are via thunderbolt bridge (which can get you a real world 25Gbe connection.)

So overall, 80-90% satisfied with QNAP. Enough that a 4th one is coming online at a different studio. Would be 100% satisfied if there were zero issues over all this time but I get that things happen...

[Frank Kruse]

Hi All,

I'm running a TrueNAS Scale (stayed away from QNAP because of the frequent security issues in the past) with an 8-drive Z2 array, 64GB RAM and a 1TB cache NVME attached to the array. And SMB is very snappy. However I am not running PT sessions off of it. It's only for backups.
One of my Mac's NICs is for Eucon and Dante only so I can't run multi-channel SMB but single channel 10GB is more than I need.

Keep in mind speed tests can be misleading for small test files like 5GB as ZFS tries to use all of your RAM as cache first before even accessing the drives so if you copy a 50GB QT to the NAS and someone else pulls it off the NAS after, it will most likely come straight from RAM or other cache (if installed) so you're not seeing the speed of your RAID. Redundant arrays (Z1,2,3) will only give you the total IOPS of a single drive which is around 300MB with a spinner or around 450MB/s with SATA SSDs once the cache is full or it has to read meta-data and data straight from the array.
My guess is that PT pulls a ton of meta-data when opening a session which is a very different load than streaming audio from the array. Hence the slower speed when opening sessions. SMB isn't great at that so some loss towards overhead can be expected I think.

All that said OSX is notoriously terrible with SMB implementation and a moving target across the various OSX variants, so YMMV even across different OSX versions.

What I wanted to say: Just becasue BM disk speed tool gives you 10GB speed doesn't mean copying or looking at 10k tiny files will be done at that speed...

And yes: turn off strict sync, it will help a bit. Unless your area is hit by power outages, in which case better leave strict sync on :-D

[Frank Kruse]

I just did a quick test opening by session backup which is quite big from the TNscale NAS and it opened very fast too. Only marginal difference between the NAS and the local NVME drive. Relinking wasn't as fast as from the local NVME but not overly slow I would say... about 30 seconds for a 5-reel FX super session.

This is from an 8x8TB RAID Z2

Hope this helps.

[Frank Kruse]

Oh and also:

I found that NFS is MUCH faster at displaying (reading meta-data) for very large folders on OSX. SMB sometimes takes a couple of seconds to display folder content (folders with 5-10k items for example) and bulk-deleting files can sometimes take a long time.

But sequential reads/writes are MUCH faster via SMB over here too.

@Lucian: I'd highly discourage setting up a mixed SMB/NFS share on TN. Make them either NFS or SMB only. I'd stick with SMB. Try a clean dataset and see if that works. One thing I can say: You'll never hit near-10Gb transfer speeds for folders with ten thousands of small files. The speed will always be significantly different for 100GB of a single QT or 100GB of 100k tiny audio files.

[lucienpalmer]

Thanks everyone for the insight.

Originally Posted by Frank Kruse

@Lucian: I'd highly discourage setting up a mixed SMB/NFS share on TN. Make them either NFS or SMB only.

Absolutely. I only have one share pointing to each dataset, and set them up according to the intended share. And as a general housekeeping rule, even name the share and the dataset the same. In the case of these tests, I have the same two film projects copied to each dataset.

Originally Posted by Darryl Ramm

I would be careful of any async mounted volumes, even with any performance hit... which should be less with high-performance SSD. I know that might not be a popular opinion. The data you are writing is presumably precious. At a minimum I'd want the server and network gear and ideally client on a UPS.

I'm ok with asynchronis writes. SMB is async by default, and I've turned it off for NFS for testing. IF the UPS's happen to fail during a power outage, the only thing in RAM that wouldn't have a copy on disk already are stems being recorded. And those can be recorded again. If the power goes out here, I'll have more important things to worry about!

Originally Posted by Frank Kruse

My guess is that PT pulls a ton of meta-data when opening a session which is a very different load than streaming audio from the array. Hence the slower speed when opening sessions. SMB isn't great at that so some loss towards overhead can be expected I think.

Opening that film this morning over SMB took 10 min. Painfully slow. The same session just opened over NFS in 1 min 20 sec. That might be the deciding factor for me unless I can speed up SMB.

I found some info about speeding up SMB's handling of Metadata. Also turning off packet signing, etc. I'm going to try that this morning and do a few more tests.

Frank - I just came across this tweak to speed up finder browsing with SMB shares that might help your large folders displaying....https://support.apple.com/en-us/102064

[Frank Kruse]

Originally Posted by lucienpalmer

Opening that film this morning over SMB took 10 min. Painfully slow. The same session just opened over NFS in 1 min 20 sec. That might be the deciding factor for me unless I can speed up SMB.

I found some info about speeding up SMB's handling of Metadata. Also turning off packet signing, etc. I'm going to try that this morning and do a few more tests.

Frank - I just came across this tweak to speed up finder browsing with SMB shares that might help your large folders displaying....https://support.apple.com/en-us/102064

What's odd is: Over here sessions open really quick even from the NAS (via SMB). I wonder what's different. I just have default settings active on that dataset. But: I use TNscale so it doesn't compare 100%. I even have everything encrypted.