The answer is a semi-useless "It depends." Tank size, oil vs. oil-free, CFM, etc. Most compressors will come with a little kit (usually sells for ~$20) that has a length of hose, an air chuck, and some air tool fittings.
I think the most important thing to pay attention to is CFM. This tells you how much total flow the compressor can provide on a constant basis, and also how fast it will fill the tank. Horsepower is often loosely related to CFM. Remember that 1HP = 746 watts, and thus you can't run more than 2 HP on a 15 amp circuit (1800 watts - line loss).
Larger tanks allow a larger reserve and sometimes running tools that take higher flow for short periods than the compressor can provide. With smaller tools, they allow the compressor to spend less time running. Most consumer compressors have a limited duty cycle (50% is common) and aren't meant to run constantly. Don't expect full duty cycles until you get something with greater than 5 CFM. Tank size usually corresponds somewhat to compressor power.
Oil or Oil Free
Oil free is usually regarded as louder. Belt driven compressors run at lower motor speeds and are usually quieter than direct drive. Don't worry unless you're painting or some other specialist application.
Who knows the PSI / CFM? I keep one with my motorcycle for dealing with low tire pressure. These things take a while, but they work. I find myself using a real air compressor is worth it if I'm doing anything more than just topping up tires once a month. Noisy and slow, but portable and cheap. ~$20
DO NOT BUY THIS
It has a 3 gallon tank like next air compressor, costs about the same, and has 1/5 of the flow. That means it takes five times as long to come to pressure and 5 times as long to do anything useful. Further, the cutoff pressure is very close to the operating pressure. $75 - $130.
The smallest I would buy is this:
It's great for inflating tires and has been good enough for me to use when seating the bead on motorcycle tires (I change my own off the rim). It's damn near useless for tools. Anything that runs for more than a moment is going to make you want something bigger. Air ratchets and die grinders just won't work on this machine. Tank pressure is only 125 psi.
About $80 on sale, $140 normally.
Might be acceptable for a 50% duty cycle on many air tools. Tank pressure is 150 psi, and it's a big reserve. Useful for more minor tools and probably the deep end of the "cheap" spectrum. $250 - $300.
Starting to get really useful. Will run most things, and possibly even allow painting with an air sprayer. This is the most you can put on a 120 volt 15 amp circuit. That circuit should be dedicated to just running the compressor. This is a belt drive instead of a direct drive. $420 - $530
To infinity and beyond
After this, you start looking at dedicated 220 volt installed devices or engine-driven, $500+ (quickly several thousand) price tags, and a lot of things that only the most dedicated garage junkies would consider installing. Useful if you're going to use an air-driven paint gun or are running a shop.