With Leopard, OS X finally has a bundled virtual desktop implementation. Pre-leopard I used a (now unmaintained) 3rd-party app called VirtueDesktops to get this feature, and while not perfect, it worked fairly well. It certainly didn't annoy me. Spaces, on the other hand, does annoy me, to the extent that I find it unusable.
Spaces SWITCHES MY DAMN DESKTOP WITHOUT ME ASKING IT TO. When I command-tab between applications, Spaces changes to the desktop with the active window. Now, at first when I encountered this feature, I thought "oh, that's pretty nice," and indeed it would be ok if it worked consistently. It doesn't, though. Usually Spaces switches correctly, but occasionally (say every 5 or 10 application-switches) it switches to the wrong desktop!
If you don't believe me, try this out: Open two terminals, and put them on different desktops. Now open a browser on one of these desktops. Command-tab between the browser and the terminal a few times, and see what happens.
This makes Spaces unusable for me. When I'm working on code, command-tabbing between the browser, the terminal and textmate, having my concentration broken by this frequent and frustrating bug is something I just can't put up with. So I've turned it off, and I'm either going to install virtuedesktops again or just learn to live without the luxury of virtual desktops.
Also, it doesn't play nicely with firefox (disappearing windows).