Yes any SRC algo will require filtering from any speed to any other. Quality of SRC does vary, and for anything important that I must digitally resample I use the izotope resampler in batch via audiofile sample manager. This site plots the quality:
http://src.infinitewave.ca
(Should be noted that izotope assisted the creation of that website, but I think it's fair.)
Mostly for 2 track I use the Tascam DV-RA1000HD as my mastering deck which I can run at any sample rate vs. the DAW's rate, and just work analog.
As for the effects of sample rate, the better your converters are the less it will matter at the conversion stage. What does matter however is the sample rate when doing nonlinear processing in the digital domain. Virtually no nonlinear plugins have antialiasing filters; nearly all of them will alias harmonics back into the audible band at ugly nonharmonic intervals. The higher your sample rate, the lower in level these aliases are likely to be, and eventually, they are down within the noise floor and irrelevant. 48KHz is much better than 44.1 in this regard, and 96KHz is about all you would ever need with today's mic preamp dynamic ranges.
An example of this problem affecting product design is given in the Lavry AD10 which will only do its tube/tape sims at double speeds because Mr. Lavry doesn't like inharmonic aliasing distortion and I don't blame him.
Note however that your available DSP power goes down _more than half_ if you double the samplerate. This is because DSP gets more costly as you handle frequencies further away from half nyquist. If the programmer doesn't take that into consideration, the quality of the DSP may actually suffer as you go higher in rate. The scaling of the Liquid Mix demonstrates this tradeoff. For all these reasons quad speeds are unlikely to be justifiable in practice at least in the near future.