[reichman]

You're asking all the right questions, but it's a long adventure ahead. Encoded Dolby Digital over an optical connection is a consumer format. Don't do this and don't worry about it. In a pro studio, you need (6) dedicated paths for 5.1. That can be done over optical using the ADAT optical format, but that's less common these days. Analog, AES, Dante, etc. are more common.

Yes, you need a 5.1 monitor controller. You don't need Dolby-branded hardware or software to work in 5.1. You do need bass management across all (5) satellite channels. This is typically done in the subwoofer in smaller studios (see Genelec subs), and more often done in the monitor controller or audio interface in bigger studios. Not everyone in post uses bass management. If your satellites are really good, you may not need it. Nearly all consumer systems do use bass management. I'm a big fan of Metric Halo interfaces and the MTRX interfaces because they provide Eucon control, bass management and easy pathways to Atmos should you ever go there. Good luck!