The first thing you should do is make sure that you only have compatible plug-ins installed. You can find the compatibility grid here:
www.digidesign.com/plugingrid
If the plug-in you're trying to use is not on the grid, you should question it's compatibility and check with the manufacturer to find out if it's compatible with version of Pro Tools you're running.
Keep in mind that most plug-ins are constantly in development and many manufacturers are still adjusting to changes/improvements we've made to the RTAS engine. While they've been tested and are considered 'compatible', they may not be taking full advantage of our latest SDK (see note below about CPU usage..). We are working closely with manufacturers to help them update their plug-ins, so expect improvements as time goes on.
The next optimization is making sure you're giving VI's that use large sample libraries their own drive to stream or load samples from. This is a drive completely independent of your OS and audio drives. Plug-ins like Ivory, Structure, Kontakt and others benefit from having a separate drive - Ivory practically requires it. Making sure that you've also followed the manufacturers instructions for optimizing the plug-in settings is also critical to getting the best performance.
Another critical aspect - make sure you have enough RAM. 2GB is about the
minimum recommended for solid VI performance. The more you have, typically the better VI's will perform.
Because RTAS plug-ins all all real-time - meaning that, as soon as you instanciate them, they take up the necessary CPU cycles - they can be sensitive to anything that can interrupt real-time data streams, so following the steps in the General Troubleshooting link below my signature and doing all the optimizations there will likely help. It's also good to not have any more VI plug-ins instanciated than absolutely necessary. If you're finished with one or more VI's, I'd recommend doing a bus based bounce to a new audio track and then disable the plug-in, so that those CPU cycles can be used by other plug-ins.
Lastly, due to the fact that many VI's are still not
fully RTAS compatible (see my note above), sometimes it may be necessary to reduce the number of CPU's in the Playback Engine by 1 - so, if you have an 8 core processor, set the number of CPU's to 7. What this does is opens up a CPU for any non real-time processes to run - you still take full advantage of both CPU's
and increase performance in Pro Tools. Typically 9128 and 6086 errors are indicators that you have some non real-time code happening in plug-ins.
Update 04.07.2011: With the growing size of software instruments libraries and how much data their loading, more and more often we're seeing issues due to RAM shortage. Pro Tools is currently a 32 bit application, meaning it works within a 4GB address space. Practically speaking, with the overhead of Pro Tools, plug-ins and associated files, you will have 2-2.7GB of RAM available for software instrument samples. Once you hit this limit, you may experience 'access violations' and/or crashes of Pro Tools. Pro Tools cannot currently 'poll' plug-ins to survey how much data they're loading (or going to load) to circumvent the RAM overage. When using software instruments that load lots of samples, it's best to keep an eye on RAM usage to be sure you're staying within it's limits. Saving the session and quitting Pro Tools when nearing the limits will allow the RAM to be flushed and you likely will be able to reopen the session and see additional RAM headroom. In situations where no additional RAM headroom is available, reducing the number of files and automation, committing EA tracks, removing region groups, etc. should reduce RAM usage for that session and allow you to continue working.
There are also solutions for situations where you need to be able to use larger sample libraries in your software instruments. The first is to use another DAW in ReWire mode and load your plug-ins and samples via the other host app, using Pro Tools to do your sequencing, piping the MIDI data from Pro Tools to the slave DAW and returning the audio data. A second solution would be to use an application like Vienna Ensemble Pro, which works in a 'server' mode - the plug-in is instanciated in Pro Tools and VEP runs as it's own application, with the plug-in creating virtual nodes to receive MIDI data from Pro Tools. It can also be run on a separate machine via Gigabit ethernet, which handles all the timing, MIDI and audio information between the applications. In any of these cases, if you use a 32 bit DAW or the 32 bit version of VEP, you'll be able to address and additional 4GB of RAM in those apps. If you run a 64 bit version, you'll be able to address virtually unlimited RAM.
Hopefully this answers some common questions. I'm going to keep this open for relevant follow-up posts, but will remove any that are not specific to this topic or that do not add something to the discussion.