First experience with NVMe M.2 in enclosure - bad.

[Darryl Ramm]

Yes OWC calling two products 4M2 confused me here. This thread was discussing external enclosures and I assumed had moved on to the OWC Express 4M2 external enclosure (and OK is Sonnet or OWC ripping off the other company with the "Express" naming). The OWC Express 4M2 does not use a PCIe switch and splits up the 4 x individual PCIe 3 lanes from one Thunderbolt connection to four separate M.2 cards.

The OWC Accelsior 4M2 card does use a switch. But OWC are a bit unclear with specs. Similar to Syba and other brands with similar cards this is a 4 x M.2 card with 16 lanes internally off a switch going to those 4 cards, but the input to the switch is only x8 (or x4 or x1 if plugged into a x4 or x1 slot). It's not x16 card like a Sonnet M4 4x4 card that has 16 physical PCIe lanes to the outside world and is capable of driving 4 x M.2 PCIe 3 cards at maximum bandwidth if plugged into a PCIe3 x16 slot.

The Accelsior 4M2 card is nice with the switch flexibility, and I do recommend switch based solution anytime folks want something with multiple M.2 cards (as opposed to the OWC 4M2 express chassis that has no switch).

But these things can be a little complex, especially with claims from OWC that are mixing up use with legacy PCIe 2 based Mac Pro Cheese graters with other Macs, so maybe worth considering some usage scenarios:

The Accelsior 4M2 card is in its natural place in PCIe 3 x8 electrical slot you get PCIe 3 x 8 performance into the cards, so half of the actual M.2 card bandwidth of PCIe 3 x16 lanes that are actually present on those 4 M.2 cards. Great if you are using M.2 drives as JBOD, each of up to two of the four M.2 drive can get it's maximum PCIe 3 bandwidth as long as other drives are not bottlenecking the overall card. Or maybe just install two M.2 cards if really hammering the drives... but folks here won't be so the expansion even at potentially degraded performance often make sense. If you are using RAID 0 then you won't get any overall performance benefit from using the switch, except the flexibility benefit on where/how the card can be used. Including because it's switch based and Macs don't support PCIe slot bifurcation then you *do* need a switch based card to do something as simple as hanging 2 or more M.2 cards off a single PCIe slot (at any performance).

If you plug the Accelsior 4M2 card into a PCIe 3 x16 electrical slot you still get overall PCIe 3 x 8 performance. It's electrically a PCIe 3 x 8 card, regardless of there being 16 lanes on the other side of it's PCIe switch.

If you plug the Accelsior 4M2 card into a PCIe 2 x8 PCIe electrical slot in a Mac Pro classic cheese grater then the switch performs PCIe 2 to PCIe 3 lane aggregation, but since it's only a x8 PCIe electrical card that gives you 4 x PCIe 3 bandwidth on the other side of the switch, so just the bandwidth of a single M2. card, you can maybe use one or two PCIe 3 or PCIe 4 M.2 drives on that card but you're not going to get scaling to 4 x M.2 cards. It's just out of bandwidth, and that will be painfully obvious if hoping to use RAID 0 to increase aggregate performance.

And if used in a PCIe 3 Thunderbolt PCIe expansion chassis there is only 4 x PCIe 3 lanes on the other side of the Thunderbolt chip, and the switch in the expansion chassis can share those across multiple slots if the chassis has those. But the entire card can only see 4 x PCIe 3 lanes of bandwidth at most.

If the card is used in a Thunderbolt PCIe expansion chassis that is running internally in PCIe 2 mode (like Sonnet make) for compatibility with Avid's non-PCIe-spec-compliant HD native and HDX cards. Then the same thing as a legacy Mac Pro Cheese Grater applies. If that chassis has a x8 electrical slot internally then the 4M2 card can be used in it, and it will perform link aggregation to get up to 4 x PCIe 3 bandwidth, but at least that's better than if being used in a 4 lane slot where it would only get 4 x PCIe 2 bandwidth... regardless of if anything else in the chassis is using PCIe bandwidth or not.

These switch based cards give maximum flexibility, but they don't guarantee maximum performance in all cases. And there will be a reason here that the card with 4 x Samsung 980 Pro only delivers 2,500 Mbit/sec ... likely it's in an Thunderbolt expansion chassis so limited to 4 x PCIe 3 bandwidth, so around the performance of a single 980 Pro drive in a PCIe 3 slot. Striping the drives here provides ease of use/larger size (at reduced reliability/increased risk of bugs and mistakes vs JBOD), but will have limited increase in performance.

And this one can't when trying to RAID 0 up to four M.2 cards. And it's all kinda sad that at best we are talking PCIe 3 performance here, where all high performance cards are PCIe 4 today. Here is hoping the new Mac Pro has lots of PCIe 4 or PCIe 5 slots.

And as I always keep reminding, the best SSD storage on a modern Mac without PCIe slots is the internal SSD, expand that as much as you can afford, and then anything over Thunderbolt is a bonus.

[Ben Jenssen]

Originally Posted by Ben Jenssen

I got the OWC Envoy express today, and had exactly the same experience as with the other enclosure I tried. Everything works fine copying, reading, speed testing, until I get to about 25-30% occupied space; the disk and its volumes ejects, gives the eject error alert, and is difficult to reformat because it will self-eject very quickly after showing up in Finder.
So, I assume it's my Kingston 2TB NVMe M2 SSD that is at fault. I'll try to get them to replace it or give me my money back.

Just info for anyone who might experience something similar. I'm assuming the NVMe M2 is broken, and I'm returning it.

I saw some slightly worrying things with my internal disk that started just as the NVMe failed for the first time. Longer boot times, a 'pop' system sound everytime desktop/Finder shows up after reboot (w/no alert message), and it wouldn't show in startup control panel, but it would boot. Otherwise it worked just fine for weeks.

So I decided to erase it and do a fresh (clean) install of Big Sur, and get it from 11.6.3 to 11.6.8. That was easier said than done! Lots of troubles; Disk utilities would become unresponsive at times, failure upon failure to erase or remove volumes (starting from a bootable external backup, or a Big Sur bootable installer.

Long story short, I found that the only way to get my internal to cooperate again was using the Apple menu item "Erase this Mac" or whatever it's called on an english system. Finally I got it back to normal behaviour.

All is good now. In fact the mac feels tiny bit snappier, like they often do after a clean install. But this faulty SSD definitely broke something on my disk/mac.

(Side note: I've said here many times that I don't agree with folks who warn against using Migration Assistant. Many years ago, it might have been good advice, but now it's an essential part of dealing with macs and installations. I installed 11.6.3 from the usb installer, upgraded to 11.6.8, installed PT, iLok and CCC, verified that PT was running fine and proceded with Migration Assistant. After bootup my system looked and felt exactly like I'm used too, and all settings, prefs, licenses etc. This is SO useful. I do alot of customizations on my macs, and would have taken days or weeks to restore stuff. PT didn't like the migration and would launch but would not open the sessions I tried. Trashed prefs - and all is good. No other apps are complaining, this is a PT thing.)

[Righty27]

... seeing as we've recently been on the topic, some hot-off-the-press info on the latest PCIe 4.0 Samsung 990 Pro SSDs ...

https://semiconductor.samsung.com/co...with-heatsink/

Once Sonnet, etc. start to deliver NVMe boxes (chassis, combined docks/drive slots, etc.) with PCIe 4.0/5.0, would it affect anything regarding reaching the speed potential of these latest drives e.g. might more plug-and-play users start to achieve faster speeds, without requiring more expensive chassis/card-based solutions or resorting to RAID?

Or does Thunderbolt 4's restricted PCIe bandwidth mean that nothing changes and one would still require a Thunderbolt 3 connection using all 4x lanes to get close to the speed potential of these latest drives (which at least appear to be more power efficient, so presumably run cooler at the slower speeds)?

[Darryl Ramm]

Originally Posted by Righty27

Once Sonnet, etc. start to deliver NVMe boxes (chassis, combined docks/drive slots, etc.) with PCIe 4.0/5.0, would it affect anything regarding reaching the speed potential of these latest drives e.g. might more plug-and-play users start to achieve faster speeds, without requiring more expensive chassis/card-based solutions or resorting to RAID?
Or does Thunderbolt 4's restricted PCIe bandwidth mean that nothing changes and one would still require a Thunderbolt 3 connection using all 4x lanes to get close to the speed potential of these latest drives (which at least appear to be more power efficient, so presumably run cooler at the slower speeds)?

There won't be any Sonnet Thunderbolt 3 (or 4) expansion chassis with PCIe 4/5. Thunderbolt 3 inherently is tied to PCIe 3. (* see note)

But first... Samsung leapfrogs WD Black SN 850 performance with 990 Pro. Cost is ~$300 for 2TB, so around 980 Pro prices. Stunning performance at low cost. Nice they have a 4TB version coming. Expect the 980 Pro to be discounted... might be time to buy some cheap 980s. Samsung and WD are two clear market leaders. Things might stabilize for a while as vendors hit PCIe 4 performance limits. And also for a while there is not a huge audience who can adopt PCIe 4 as most installed PCs are still PCIe 3, thanks in large part to Intel dragging its feet there. And noteworthy that M.2 technology is already sampling/testing with PCIe 5 and Samsung is shipping their U.2 PM1743 PCIe5 SSD that does ~13 GB/s sequential I/O. It's great to see vendors like Samsung developing PCIe 5 technology for the enterprise market that will trickle down to the consumer market.

There is likely strong demand for this in some desktop niche markets including gamers and data science and machine learning workstations, some software development .. but the main use is servers, servers, servers... the Samsung 990 Pro is a retail packaged product, cloud providers, server manufacturers etc. will be using different variants, and a heck of a lot of them.

Also worth noting that current top M.2 drives like the Samsung 980 Pro and WD Black SN850 are all PCIe 4, have been for a while and run with degraded performance over a Thunderbolt 3/4 connection. The Samsung 990 Pro with performance closer to PCIe 4 limits will just make that difference more glaring.

It's "amusing" that a single few hundred dollar M.2 drive like this 990 Pro can consume around 2X the bandwidth of a single Thunderbolt 3/4 link. Apple has tied themselves to Thunderbolt, and for many uses, including audio interfaces and/or DSP processing etc., there are not pressing needs for PCIe bandwidth, but for high end storage this is already and will increasingly be an area of weakness. I hope Apple addresses this with good PCIe 4 or 5 expansion in a Mac Pro. And no, there is no Thunderbolt 5 story that will arrive anytime soon.

note *: There theoretically could be advanced solutions that take multiple Thunderbolt 3/4 links and say aggregates two Thunderbolt links to get one PCIe 4 M.2 card performance. The PCIe switch chips, development, packaging costs all for a small market likely means this is a non-starter. Current vendors don't even want to package PCIe3 switches into M.2 storage boxes, doing that might add $100 to few $100 price increases to make better/more flexible storage. I doubt there is a market for ~$1k to few $k M.2 PCIe 4 aggregated switch based storage boxes vs, just running the M.2 drives at degraded PCIe performance. The PCIe switch chip market (as well as SSD market), new technology areas are driven by cloud and enterprise uses and those vendors, especially Broadcom, know where their toast is buttered. And I expect that will help keep leading edge switch chipsets quite expensive.

So in summary, likely roadmap for as far as it matters to anybody here is to just run increasingly faster M.2 drives in current performance Thunderbolt 3 enclosures or chassis. If leading edge storage performance is important for you (and it won't be for most Pro Tools users) then do what I'm doing and hold out for the Mac Pro.

[Righty27]

Thanks Darryl - no point hanging around waiting for new enclosures, etc. for most of us then ... and probably a good time to look out for deals on the Samsung 980 Pros, as you suggest :)

In reality, for those who decide not to wait for a CTO build of a new Mac just to get extra internal storage (even though this is clearly advantageous and not that bad value), these speeds are more than enough for external 'data' drives, whether for audio or even 4K video.

It remains to be seen how quickly 8K is adopted considering all the camera/acquisition challenges involved, quite apart from this typically being cropped/output as 4K, so I imagine that the most appropriate use-case for many here would be to overcome the e.g. 512GB internal SSD in the base Mac Studio and boot off a very fast and larger external drive ... if just to avoid the lead times!

[sw rec]

I just checked the “big on-line retailer” (no free plugs here but you can guess). 1TB Samsung Evo 980 Pro is only $139.99…that’s with or without heatsink, your choice. Will the heatsink fit in the Sonnet case?
Funny, my first recording drive was a monstrous 1 gig SCSI drive. It was $1200!!!

[Darryl Ramm]

Yes heatsinks fit in the Sonnet (and any other) PCIe chassis.

[Guy McDude]

FWIW, heres my NVMe experiences:

After much "drive counseling" from Darryl [thanks again].
I weighed the pros and cons, and went with:

Sonnet Echo III expansion Chassis.
Sonnet SSD Fusion M.2 PCI card.
3 Samsung 980 Pro NVMe drives. [2tb record, 2TB samples, 1TB Temp storage]

and this part of my M1/ PT move has been very solid.

Im still wrangling other weirdo PT issues like the white cursor in classic mode.
[ why am i the only shocked and screaming about this??]

But the speed and reliability of the NVMe part of my changeover has been very solid.

thanks again Darryl.

[spinsong]

PCI vs pcie could mean something very pricey as some of you older pci card adapters might know. What would you rather have more bark or more bite. I prefer good sound over a bite in my pocket. Look at it from a price perspective. Look up nvme archival and look at the price tags. 17,000.00 for some hard drives. It’s viable from a performance standpoint, but not practical if you decide to look further down the road. Heating issues, amongst countless failures. Don’t expect to get Maserati speeds without paying the price. My land yacht gets me where I need to go, and it’s reliable. Flick the switch, and it just works, just as in years past. Speeds are also in par with the rest. I don’t need blazing speeds and overheating to run audio. Whatever floats your boat.
What’s the point of the whole rant. If you want to go cutting edge expect to pay the price over longevity. Pcie are a thing of the past. Go ssd and hard drives. Even computer manufacturers have cut the cord. To add fuel to the mix, pcie is intel based technology if I’m not mistaken, it’s possible it’s not compatible with the new chips without using Rosetta 2.

Sent from my iPad using Tapatalk

[Darryl Ramm]

Originally Posted by spinsong

PCI vs pcie could mean something very pricey as some of you older pci card adapters might know. What would you rather have more bark or more bite. I prefer good sound over a bite in my pocket. Look at it from a price perspective. Look up nvme archival and look at the price tags. 17,000.00 for some hard drives. It’s viable from a performance standpoint, but not practical if you decide to look further down the road. Heating issues, amongst countless failures. Don’t expect to get Maserati speeds without paying the price. My land yacht gets me where I need to go, and it’s reliable. Flick the switch, and it just works, just as in years past. Speeds are also in par with the rest. I don’t need blazing speeds and overheating to run audio. Whatever floats your boat.
What’s the point of the whole rant. If you want to go cutting edge expect to pay the price over longevity. Pcie are a thing of the past. Go ssd and hard drives. Even computer manufacturers have cut the cord. To add fuel to the mix, pcie is intel based technology if I’m not mistaken, it’s possible it’s not compatible with the new chips without using Rosetta 2.

Sent from my iPad using Tapatalk

What a pile of confused incoherent and technically wrong waffle. Why you need to keep posting stuff like this when you clearly don't have any clue what you are talking about is beyond me.

No PCIe is not "a thing of the past" just what are you talking about? No PCIe is not Intel technology. Not one computer manufacturer has "cut the cord". All modern computers are built around PCIe, including the new Apple Silicon Macs, they rely heavily on PCIe internally, and most SSDs used in modern computers... PCIe based.

PCIe has no relation to Rosetta 2. Every Apple silicon Mac uses PCIe extensively... and uh they work fine without Rosetta 2.

Oh and you want to claim HDD are more reliable than SSDs. Just no. Yes HDD are good for archival, arguably better than SSD, but better as an operational drive hell no. More reliable, much faster, more compact, and becoming effectively cheaper than HDD for many uses.

Your lack of any technical understandings in your "technical" posts is just staggering.