NAS- use SMB or NFS shares to connect?

[lucienpalmer]

Originally Posted by Frank Kruse

What's odd is: Over here sessions open really quick even from the NAS (via SMB). I wonder what's different. I just have default settings active on that dataset. But: I use TNscale so it doesn't compare 100%. I even have everything encrypted.

Yeah, that is very interesting. As you alluded, it sounds like the big difference is Scale vs. Core. TrueNAS Core is based on FreeBSD, so maybe its handling of SMB is different?

Ironically, TrueNAS Scale is based on Linux, so I would assume that NFS would be the best fit since its the natural Unix/Linux share format.

Regardless, sounds like NFS is performing best in my situation. Thanks for sharing your experience.

[lucienpalmer]

Well, I spoke too soon.

I switched back to NFS this morning, and received several -9093 and -9073 errors during playback. Then remembered that was the whole reason why I started exploring SMB in the first place. I got so distracted with the file copy and load times, I had forgotten what prompted this experiment in the first place.

During the playbacks over SMB I received no playback errors at all. I might stick with SMB after all and research more on performance tweaks. (and get used to making a cup of coffee during session open)

[Darryl Ramm]

Originally Posted by lucienpalmer

Yeah, that is very interesting. As you alluded, it sounds like the big difference is Scale vs. Core. TrueNAS Core is based on FreeBSD, so maybe its handling of SMB is different?

Ironically, TrueNAS Scale is based on Linux, so I would assume that NFS would be the best fit since its the natural Unix/Linux share format.

Maybe you mean that NFS performance is expected to be better on both NAS platforms than SMB? I think SMB performance gets a lot of attention by these vendors and the broader community contributing key parts here, and the story here is going to be much more complex than just Linux vs FreeBSD.

(Old UNIX geek here)

FreeBSD has a long history that goes back to the BSD UNIX distributions, which kinda what Sun Microsystems developed NFS on top of and then a port of NFS was included in BSD (although effectively all of us at different UNIX systems companies were licensing NFS code directly from Sun). Linux has no historical claim/preference on NFS vs FreeBSD, if anything historically it's the other way around, but it does not really matter. All modern UNIX-like systems have a lot of work put into them on TCP/IP and NFS performance, by lots of folks, many of who are familiar with different implementations, and I expect any performance differences will likely come down to specific reasons, not really some broad historical reason. Linux may well have been likely more optimized for NFS and SMB (see below) over time, but there has also been a lot of work done on FreeBSD networking performance. I just don't have near the memory, skills, knowledge to unroll what the signifcant ones have been for either camp.

TrueNAS Scale has had less time being hammered on than TrueNAS Core, and the focus of Scale seems to be more Linux/container deployments and it sounds like Core has had more of an SMB focus with multiple folks claiming TrueNAS Core has better performance that TrueNAS Scale for SMB, but I don't easily seen any hard numbers. I have not kept up with TrueNAS Core and Scale but they at least used to use the NFS Gashea user space server, quite different to any standard Linux or UNIX implementation of NFS. And it's a potential cause of performance differences. I thought at one time Scale was going to use the Linux kernel NFS implementation, maybe they moved to that eventually, or maybe it's options I'm not sure I don't run it and cannot quickly check.

The support for SMB in FreeBSD and Linux is (or is derived from) Samba. Andrew Tridell in Australia reverse engineered SMB/CIFS and implemented Samba starting in the early 1990s, IIRC the first releases were for SunOS, and then it took off on Linux, growing alongside and helping out with it's popularity, it was also available on many other UNIX OSes. I helped get Tridge some Silicon Graphics hardware to work on and later helped him out with use of VMware products. Microsoft had a typical slimy attempt to embrace and extend IBM's SMB protocol and unjustly protect CIFS from reverse engineering, they had their hat handed to them by Tridge and the broader CIFS/SMB reverse engineering and open-specification community, all very amusing to watch happen.

Recent Linux kernel versions provide ksmbd, a kernel implementation for SMB support. Ultimately parts of Samba can/will coexist with ksmbd, so be careful of claims of "using Samba" may or may not mean using ksmbd and visa versa. And depending on needs ksmbd may not be suitable. The adaption of ksmbd was not pain free, with significant security problems. In general with all these NAS toys you want a vendor who is really on top of security issues... and take their security updates. ksmbd came from Samsung folks very much focused on Linux. FreeBSD seems to have lagged Linux with SMB support enhancements, and I'm not clear on where things are at there. I could ask some of the FreeBSD kernel dev folks I know there or maybe just look at the code sometime.

Good NAS/server companies do a lot of optimization and custom code work, from filesystems, network stack, administration tools, etc. They might not even be using components that come from the OS distribution. I think TrueNAS and QNAP both use SMB support provided by Samba, not sure if they also implement ksmbd. Some high-end enterprise NAS boxes sure reimplement the whole SMB stack, including having kernel space SMB support, they don't use Samba (or ksmbd) (and legal factors may also come into play as Samba is GPL licensed). Licensing and legal factors also affect the choice of the underlying operating system. This is especially true where vendors might select the FreeBSD software with its BSD license, or Illumios for its CDDL license vs a Linux based system with the GPL license and its strong copy-left terms.

And Apple removed the Samba client and server code from Macs in the OS X 10.5-10.7 time frame because of GPL licensing concerns and replaced it with their own SMB implementation. Which caused some problems, hopefully they also eventually significantly increased performance, I can't remember comparisons at the time.

Hopefully this gives a flavor of just some of the potentially significant differences there can be between different vendors systems, nothing is likely to be as simple as just prescribing differences because of what OS kernel or distribution some NAS box might use.

Filesystem aside: My background makes me want to pick a platform built on the ZFS filesystem, just such an impressive piece of engineering, but I'd want the NAS vendor to have a team of folks with ZFS internals experience on board. (I'd love an Oxide Computer rack as a NFS Server, from some of the developers of ZFS and maintainers of Illumos, but they won't give me one and I can't cough up the few hundred $k they (very fairly) cost, and PG&E won't run a 3 phase service to the house anyhow ).

[BScout]

Originally Posted by Darryl Ramm

Good NAS/server companies do a lot of optimization and custom code work, from filesystems, network stack, administration tools, etc. They might not even be using components that come from the OS distribution. I think TrueNAS and QNAP both use SMB support provided by Samba, not sure if they also implement ksmbd. Some high-end enterprise NAS boxes sure reimplement the whole SMB stack, including having kernel space SMB support, they don't use Samba (or ksmbd) (and legal factors may also come into play as Samba is GPL licensed).

QNAP supports ksmbd but enabling it disables SMB encryption (nice little warning if you want to enable it.)

[Darryl Ramm]

Originally Posted by BScout

QNAP supports ksmbd but enabling it disables SMB encryption (nice little warning if you want to enable it.)

Thanks! Yes encryption is one of the big losses with ksmbd.

And it looks like they timed stuff well to avoid supporting it too early and suffering from a bunch of security CVEs. Which all gives some confidence in their approach. (edit: becasue of wide concerns abouy QNAP security, I meant this was at least a good sign).

I just noticed that the recent (affordable mid-tier not super high end) QNAP boxes support PCIe 4 x 4 M.2 cards, IIRC when I looked in the past they had low-end M.2 PCIe lane configs that turned me off. I need to spend more time looking at them, I am currently rebuilding my 12TB M.2 Thunderbolt JBODs to get more space and should really move some of that to a SSD/HDD NAS.

[Frank Kruse]

For what it's worth:

I was on TNcore for 2 years before moving to scale. All on the same RAID Z2 array and same hardware (Supermicro server i3, 64GB RAM nothing high end really) and I could open sessions on core just as fine via SMB (but I only did that very rarely for quick looks at backups). Definitely no waiting for 10 Minutes for them to open even when I was still on the 5,1 mac and 1Gb. I don't even use Jumbo frames or SSDs (except for the L2ARC NVME cache) Nothing fancy really.
ZFS as been rock solid. Snapshots and being able to roll the entire Dataset back in time is awesome.
Must be something else that's off I think. Current scale does use a different version of SMB than the current core version. No idea...could be a ton of reasons.
What we do have in common is that NFS is WAY slower than SMB for sequential transfers.

[lucienpalmer]

Originally Posted by Frank Kruse

What's odd is: Over here sessions open really quick even from the NAS (via SMB). I wonder what's different. I just have default settings active on that dataset. But: I use TNscale so it doesn't compare 100%. I even have everything encrypted.

Interesting turn of events- TrueNAS just announced that they are phasing out CORE (FreeBSD based) and moving to SCALE (Linux). I can't say I didn't see it coming, but surprised it's happening so soon.

https://www.msn.com/en-us/news/techn...on/ar-BB1k5Bv1

The good news for me is they have a tool to migrate an existing system. I'll give them time to work out the bugs, of course, but maybe I'll finally get that SMB performance like you are seeing on your system.

[Frank Kruse]

Originally Posted by lucienpalmer

Interesting turn of events- TrueNAS just announced that they are phasing out CORE (FreeBSD based) and moving to SCALE (Linux). I can't say I didn't see it coming, but surprised it's happening so soon.

https://www.msn.com/en-us/news/techn...on/ar-BB1k5Bv1

The good news for me is they have a tool to migrate an existing system. I'll give them time to work out the bugs, of course, but maybe I'll finally get that SMB performance like you are seeing on your system.

Yeah it's pretty much an in-place cross-grade. Just select the other update train and it will migrate everything. Went pretty easy when I did it. Just make sure your hardware runs on linux before you do it.