If you have iTunes on another computer, and you network the Apple TV (built-in, I believe), then you don't even need to store the music on the Apple TV itself. It will stream it THROUGH the Apple TV, much like I do with my Airport Express, albeit without the pretty interface of Apple TV.
That said, to answer your question, nope...don't long for anything in the quality department. The Airport Express streams over Toslink, so I have a very clean and digital connection to my setup. It really sounds pretty amazing.
BTW...where are you located? On the off chance you're near the Chicago area, I'm happy to demo it.