USB Audio ASIO Driver is the solution. It enables buffersizes down to 32 samples (0.73 ms) and creates an ultra highspeed USB audio connection, bypassing the operating system's audio.
This does not only give you low latencies, but better sound quality also.
SoundManager (Mac) and Multimedia Extensions (MME / Win) use non-highend sample rate conversion in order to sync the different audio signals from applications to the sample rates used on the external soundcard. And sometimes there's not even a way to control the rate to be the prefered one. (E.g. 48kHz instead of 44.1kHz.)
Using ASIO it's possible to do direct "one to one" hardware connections and even use higher bitrates (e.g. 20bit or 24bit) if supported by the device. Plus this driver can further increase quality by adding dither.
But of course, you need an ASIO compatible application like Cubase, Logic Audio, Reason, Live, Digital Performer, SONAR, BPM Studio, etc. If the app supports ASIO2 you'll be able to benefit from more features.