IME they all behave erratically. There was a very lengthy discussion at AVS about this and it came down to nothing concrete. People were getting different results which seemed to be related to the displays or AVRs used. Me, I'm getting perfect Pixel Format output, whichever I choose. Full RGB and Limited RGB they both output as expected. IIRC someone who also had a Pioneer display also confirmed it. Many people could not get Pixel Format Full RGB to work at all.
As for 10.5 vs 10.4, I haven't noticed any changes besides they screwed with the workaround for disabling Powerplay (which gives the sound issues) by editing the profiles.xml file, which worked fine with 10.4 and before.
I gave my 5770 another try. Here are my findings:
1.) Power saving on the card is broken. Sound will play correctly, if the GPU clock is not allowed to drop. I use ATI tray tool to force power saving off, and LPCM sound via HDMI works. This is really pathetic. I have to run the card with full power (or at least a constant clock) all the time, otherwise there will be "glitches".
Even games suffer from this problem, e.g. the start menu of Tomb Raider Underworld, as it doesn't "stess" the GPU enough to force high clock rates.
2.) Black level behaviour seems to be related to the refresh rate. I consider PC levels basically broken, they only work "a little bit" when using 1080p60, but not with 1080p59, 1080p50 or 1080p24. Only the HDMI output has this problem, via DVI output the levels are correct (but sound won't work, of course).
3.) Studio levels seem to work fine. However, for TMT 2&3 "Dynamic Range" in advanced video settings must be set to Limited. For PowerDVD, Zoom Player & Co. it must be set to Full. Very odd, the 4xxx cards didn't require this (and CCC doesn't offer this setting with 4xxx cards). Same is true for YCBR.
This is very annoying, if you have a high end projector with DVI input (requiring RGB PC-Level). Or a PC monitor. Isn't this a card for ... PCs?
4.) The annoying "AMD external events utility" service must be running, otherwise sound won't work. The 4xxxx cards work without it.
Conclusion: With some tweaking (forcing high GPU clock, using studio levels, creating a special video profile for TMT) the ATI 5xxx card seems to be usable. Still, the 4xxxx cards don't exhibit these problem. They don't bitstream HD audio (which IMHO is a software limitation), but for ReClock PCM audio is better anyway. TMT3 .180 doesn't bitstream HD audio anymore, if ReClock is loaded.
Speaking of bitstreaming, it doesn't seem to work all the time, even without ReClock. I'll test a bit more....