Options for expanding Mac storage via external drive(s)

[Righty27]

I'm looking to build a reliable setup that allows me to utilise my existing MacBook Pro (16-inch M1 Max with 1TB SSD) with all my personal/media data on external drives (Pro Tools, FCPX and increasingly DaVinci Resolve). I wish to access these routinely across three locations, with the MacBook generally at my workplace or remote whenever the planned second Mac (probably a Studio, semi-permanent at one location) isn't available, typically on evenings/weekends at the 'other' location. Although not specifically Pro Tools-related, there's also a Thunderbolt-equipped M1 12.9-inch iPad Pro with DaVinci Resolve that would also benefit from access to media on these drives.

My idea/hope is that it should be possible to share the same external drives/data across primarily the two Macs, in order to avoid synchronisation via the likes of CCC or cloud services. The two approaches I've been considering involve either (a) booting off a portable external drive or dock with built-in NVMe slot(s), or (b) relocating the Home folder to an external drive (in order to maintain compatibility with standard iCloud features for syncing Documents and Desktop folders, which have to remain in the Home folder). I would then only have to be concerned with travelling with the external drive(s) for the media files, as routine documents would be available on all my iCloud devices - backups would be easy to manage via each Mac.

I've been experimenting today using a fresh 2TB Samsung T7 Shield for testing before committing to purchasing 2 x 2TB NVMe SSDs and, most likely, a Sonnet Echo Dual NVMe Thunderbolt Dock. Darryl has previously explained the pros/cons of this Sonnet hardware i.e. based on the dock's use of TB3 and lack of a PCIe switch, each of the two built-in NVMe slots gets 2x lanes of PCIe bandwidth equating to approximately 1,500MBps. Darryl's preferred, higher-performance approach (Sonnet chassis with PCIe switch) or even faster recommendation (larger internal Mac SSDs), do not ideally suit my proposed workflow of sharing data across devices, and isn't helped by local issues in procuring CTO Mac hardware with larger drives, let alone costs.

Approach (a):

• Create a bootable Ventura external drive containing user Home folder, thus should work with standard macOS features, such as iCloud Documents, etc. + containing routine Mac folders, Photos, ~/Library (settings, e-mail, etc.), etc.
• Work in conjunction with separate external drive(s) for Pro Tools, big plugins/samples, video media, etc.
• Pros - easily transferable between Macs vs. cons - not ideal to be running OS/apps from an external, as well as associated data ... but cannot get it to work regardless of multiple walkthroughs, trying different cables, etc. - see https://eclecticlight.co/2022/11/10/...e-silicon-mac/

Approach (b):

• Create user on MacBook, then move Home folder to external drive
• Pros - no messing about with external boot drives, as each Mac would boot from internal system drive, yet should still be possible to share data across Macs (?)
• Cons - online claims of poor reliability and OS updates sometimes breaking a setup that Apple don't allegedly support

Has anyone got either of these to work or have any experience of performance, reliability, etc.?

I'm not sure if either of these are actually quite common/basic models that members use - just out of my comfort zone after many years using a straightforward 'stock' setup with external drives and a single Mac?!

[Darryl Ramm]

Uh I'm going to step away and look at this from a different angle.

I would do this in quite different ways. And you are making this quite complex with wanting to move boot drives etc.

You sure don’t want to be booting off external drives and moving those boot drives between computers. I just don’t see any reason for this. You should be trying to make stuff as simple and as foolproof as possible. Hopefully it’s not something silly like application licensing reasons. If so, fix that in some other way.

The boot drives should stay with each computer, you want to be able to update macOS on one system independent of others and test stiff, or do update that fix things for a specific computer/hardware setup. You also likely want the other comptuer to be able to boot/run backups etc. be used by an assistant or temp worker, etc. while you are working on a different one.

Moving the boot drive between computers can cause problems with computer/hardware locked software licenses. Cloning boot drives on Apple Silicon based macs is more complex that in the past, and you really need to get up to speed on that. Well documented all over the Web, especially on the CCC/Bombich’s web site.

There is no need for the user's home directory being on an external drive, you likely want the home directory on the internal SSD, You want that user to be able to log in and have at least core tools available to them if they need to administer or repair stuff. There is absolutely zero need for the users’ home drive to be on the external drive if you just want to share stuff, just put the files on those external drives, change the ownership/permissions on the drives/folders as needed, maybe put a symbolic link to the drives from your documents folder…. But I’d prefer not to just label the drives with useful names and use them via /Volumes/whatever. And you should then always be sure where things are. And you manually set the accounts on all computer up so they all have the same user and group names and numerical UID and GID. You can use Migration Assistant to help with this, but I'd just do it manually, like I'd also manually reinstall clean apps on each system. KISS, simple. Just get it done.

User app preferences are stored under ~Library which means you might change settings on one computer and they won’t propagate to the other but worse case you just manually set preferences for your apps on each system,

I would not be planning on moving everything you need between computers all the time. What happens when you drop all the drives on the ground? When they all get stolen out of your car? Get each computer set up so it’s working and move just the documents/assets you need. I just would not unnecessarily be wanting to unplug, fondle, hit, drop, lick, … the drives that my home directory, operation system or any of this other stuff is on.

My preference with multi-system setups is to have local fast SSD storage on each workstation with Apps and samples etc. on them but to have a central NAS repository (on a good fast network) of sessions and to copy those sessions down to the computer local SSD when working on them. Spend the time to develop a process, good labeling, versioning etc, and safeguards to stop content benign deleted/overwritten. Ideally do stuff with simple tested shell scripts or droplets. If security considerations allowed and I had a fast enough home internet connection and a good firewalls setup at the studio then I’d consider working that remotely, or maybe staging via cloud storage. Or I’d grab a Samsung T7 drive and use it as a transfer drive to bring sessions home on and copy to the local faster storage (or for most stuff you could just run them on the T7, but then if you make a mistake you don’t have a local backup with you).

And if you are not big enough to justify a NAS (and yes it adds complexity) then I'd pick say the Mac Studio system as the main system, have all online/nearline sessions mounted there, have most backups done from that system. And have that Mac Studio running as an NFS (or shudder, SMB) server so you can copy sessions to the other Macs.

Do not try to share the home directory on iCloud or share on a NAS etc. It’s asking for problems, again KISS. As simple to use as possible.

[Righty27]

Thanks Darryl.

Whilst I intentionally omitted my reasons for exploring if such an arrangement/workflow might be possible, I feel I need to rule out any strange desire to over-complicate matters and explain why I have ended up asking these questions.

My day job includes a team who create TV/radio/social media ads/videos (sometimes in conjunction with agencies, outsourced specialised/overflow tasks via Fiverr, etc.) mainly on FCPX, however also some DaVinci Resolve, Cinema4D, etc. I'm also a ‘hobbyist’ musician/producer whose collection of applications, plugins, VIs, etc. occupies lots of space on my system drive (larger samples, etc. already on external drives) and I occasionally use Pro Tools for work too.

I have a holiday home on the other side of an international frontier, where I spend some weekends and holiday periods. Apart from the flexibility to be able to work on media projects ad-hoc, in either home or my office, I'd ideally also like to routinely access my personal Mac data wherever I find myself. My current KISS method involves a single, company MacBook, which I don't plan on using as the only location for my personal data. However, I was looking to add a second, semi-fixed personal Mac Studio to try to reduce the necessity to carry the MacBook to my holiday home, as I need to access personal and work data at all three locations.

By keeping all my audio/video media on external drives, my hope is to start using standard iCloud syncing features for Documents and Desktop, however these must be located in the normal Home folder. Additionally, I have a large "On my Mac" e-mail archive which would live in ~/Library. These, and not the audio/video media workflows, are the primary drivers for a 'portable' Home folder or the idea of an external boot drive.

The external boot drive consideration (optimistically?!) looked to minimise the admin overhead of app/plugin updates x2. Finally, the lack of free space on a 512GB Mac Studio system drive (CTO not an easy option for me here) with all my apps/plugins (large e.g. samples already on an external drive) leave little working space and might create an SSD wear issue, thus I was looking to use the system drive for mainly static data.

Occasionally transporting drives (obviously with care), for whenever I plan to work on media projects, is not my biggest concern as these will tend to stay in one place or another e.g. over the summer holidays. I have sufficient licences (mostly on an iLok for audio) and my backup strategy is pretty solid - local/travelling TM for each computer in my family + central (family home) TM for each + central CCC clone + automated Arq backups to BackBlaze B2 (in addition to any iCloud/OneDrive sync'd files).

Originally Posted by Darryl Ramm

There is no need for the user's home directory being on an external drive, you likely want the home directory on the internal SSD, You want that user to be able to log in and have at least core tools available to them if they need to administer or repair stuff. There is absolutely zero need for the users’ home drive to be on the external drive if you just want to share stuff, just put the files on those external drives, change the ownership/permissions on the drives/folders as needed, maybe put a symbolic link to the drives from your documents folder…. But I’d prefer not to just label the drives with useful names and use them via /Volumes/whatever. And you should then always be sure where things are. And you manually set the accounts on all computer up so they all have the same user and group names and numerical UID and GID. You can use Migration Assistant to help with this, but I'd just do it manually, like I'd also manually reinstall clean apps on each system. KISS, simple. Just get it done.

I appreciate that one could use symlinks for basic folders, such as Documents, however do you believe that symlinks from the User/Library on each Mac to a folder on an external drive would be reliable?

Would identical naming of user accounts on each Mac suffice - I’ve never changed UID/GID however are you advising that these also need to be changed to match on each computer? I’m taking this opportunity for clean app installs anyway and was planning to use Migration Assistant to transfer the user account (only) to ensure everything matches, however would welcome your confirmation that careful matched naming would work.

Apologies if anything remains unclear as I try to limit this post to the most pertinent details. It may well transpire that the only solution is to keep my personal data on the Mac Studio - however, based on a stock 512GB model, this would use up 80% with my apps/plugins/local e-mails in the User/Library + Documents and Desktop (for iCloud syncing and thus access from the MacBook also). The alternative would be to place a CTO order for a a 1TB system drive, however that could add months which I was hoping to avoid, as I was working on this now and attempting to finish over the summer holidays.

Thanks again for all your help!

[JFreak]

If you need to physically transfer your home folder with you from one computer to another, all you need is big enough external drive and one symbolic link on each system.

[Righty27]

Many thanks Janne.

Originally Posted by Righty27

I appreciate that one could use symlinks for basic folders, such as Documents, however do you believe that symlinks from the User/Library on each Mac to a folder on an external drive would be reliable?

I'm aware that macOS actually allows for an advanced setting to locate one's Home folder on an external drive. Any reason why using symbolic links would be better? For example, presumably the user/folder names (+ Darryl also mentioned some IDs) have to match what is set up on each Mac.

Do you have experience suggesting this would be reliable e.g. including the user Library folder, especially considering Darryl's points regarding switching computers (licensing, settings, etc. if an app is installed/updated/removed on one computer but not the other) + my concern regarding e-mails "On my Mac"?

It may be this simple, however most of the online feedback regarding these methods appears to be negative, so it's hard to tell if that's the minority with issues or typical.

[JFreak]

Symbolic link makes it irrelevant where your directory is physically located. It is just a pointer.

Ideally you would install your system once, then clone the system drive to other computers (this will retain user/group id's) and as a last step move home directory to external drive and create symbolic links.

It is a possibility that mismatching user/group id's cause trouble, but try first. I have not faced anything like that.

[Righty27]

Thanks, Janne.

I've spent enough time on this to appreciate that, even if I can get it working, problems could quite possibly arise somewhere down the line e.g. macOS 13.3 looks to have broken the 'official' Home folder on an external drive functionality (not using symlinks), only to be fixed in 13.3.1. Instead, I've spent my time today sourcing a CTO with 2TB that I can receive a bit quicker. This will defer my build somewhat whilst avoiding wasting more time. However, it should allow me to keep one standard Home folder on the Mac Studio + use iCloud for Documents and Desktop access on the MacBook ... and having decided to wait, I've gone for the extra storage headroom.

Audio/video media can still go on external drives (e.g. Sonnet NVMe Dock with the Mac Studio + something more portable to carry around with anything current). Apps/plugins will have to be installed and kept updated on each Mac - there aren't many nowadays that allow only a single installation (Waves maybe?) and only the (rarely accessed) "On this Mac" e-mail wouldn't be available on both Macs.

Thanks again! :)

[Darryl Ramm]

I would not ever consider using iCloud desktop syncing. That's been warned about multiple times on DUC. You do not want a cloud service doing stuff to stored documents or slowing down the computer performance etc as you try to use near-real-time critical apps. I would not use any cloud storage for anything except dropping backups onto or potentially moving sessions there to download once I get home.. (but that may have issues if you can't tell when the upload completes).

The home directories should physically live on each computer, on the internal drives. All computers are usable independently and redundantly if there are problems on others. By trying to move them around or connect via iCloud you are making all this non-redundant, non-tolerant of mistakes or other problems.

Just have content stored on disks, wether you have symlinks inside the home directories is a convenience choice, I mostly would not, just reply on nice labeling of the volumes to know what is where.

To answer one question: you have to have the same user on different computers have the same UID and the same groups on different users need to have the same GID, and you want all those user and group names to be exactly the same for each UID and GID on each computer. Don't add unnecessary accounts, you can disable login on accounts that don't need to be there... but even on a system that "jack" can't log into it's nice to know that files you see might be owned by jack because he created them on another system and you moved them here. The UID and GID actually determine who on the local computer owns/has permissions in the filesystem. The name is just symbolic for the wetware user. For basic stuff you need to know is how to get to the well hidden "Advanced Options" in User&Groups settings. See https://support.apple.com/en-bw/HT201548 as an example. You can guess why Apple hides this stuff so well. Be careful here. You may also need to use chown of chgrp commands to change owners of files and folders after doing this. You can use the System Migration tool to move user accounts to different Macs and it will keep the UIDs and GIDs consistent. Do not use it to move drivers or apps. Don't use this if you have lots of garbage in the home directories, e.g. old email you don't want to move, or clean that out later... but I really prefer nice new clean setups not accounts encrusted with ancient garbage like prefs files, logs, core dumps, etc.

And again some of the stuff you are talking about is youve taken separate redundant of failure systems and risk combination them in a way that becomes highly sensitive to problems. Just in principle don't to this. Especially not in a business environment. Just do things like software installs manually, test they work on each system, do a controlled rollout to other systems, yes it takes time, but it is safer and you are in control.

If part of the issue here is persistent storage of Emails... uh just use a web mail client and don't store stuff on the local computer. I ran BSD UNIX Sendmail servers for years, used every different client side software you could imagine. Just finished helping people with documents trapped in Exchange clients on ancient windows systems. And there is just no way I would want to use local client Email software. Just get it into the cloud and leave it there. Lots of options out there. Simplify where possible. KISS.

[JFreak]

Originally Posted by Darryl Ramm

The home directories should physically live on each computer, on the internal drives.

This is just not true on UNIX systems, which macOS also is. It is a common practice to have two servers handling users and third for handling files, and both (user handling) servers point to same file server.

[Darryl Ramm]

Originally Posted by JFreak

This is just not true on UNIX systems, which macOS also is. It is a common practice to have two servers handling users and third for handling files, and both (user handling) servers point to same file server.

The word should here... this is my recommendation, it's not saying something can't be done. Lets try some others...

"Janne should not molest the reindeer" (a recommendation)
"Janne cannot molest the reindeer" (a statement of fact that something cannot be done)

Lets see I've been doing UNIX for 40 years. I worked for multiple UNIX systems companies, including Silicon Graphics (most teams I can think of had home directories on our own desktop Workstations). I started off coding, hacking on systems, managing UNIX, messing with internals, running benchmarks, managed engineering teams, worked with some of the largest UNIX users in the world.

Not sure why you are picking on 2+1 servers in the example, but oh well.

And just because Macs have some UNIX DNA be careful about over extrapolating stuff. Some Mac specific things can make some of this messier. Like ~/Library.

So thanks, I know how this can work. I am providing advice here on a specific setup to a user listening to what they are trying to do, and also factoring in their level of understanding and where I know they have already had problems trying to do stuff. And for that and all the reasons I stated I would not do anything except have the users home directories installed on the internal SSD on the local Mac. But again, that does not dictate where you store media assets/sessions etc.