I can't answer from a building perspective, but I from what I recall (working from memories over a decade old) from my geotechnical engineering classes:
- over 2 cm : rock
- 2cm to 2mm : gravel
- 2mm to 0.2 mm : sand
Then you get to silt and clay at smaller sizes. I might be wrong on the boundary between rock & gravel, as the wikipedia page for gravel lists the upper limit o gravel at 64mm / 2.5 inches, and the lower limit of sand lower as well.
You also see terms like 'coarse aggregate' and 'fine aggregate' when dealing with concrete mixing (fine tends to be sand or rock dust, coarse is larger).
You've already seen the 'smooth' (aka 'river rock', aka 'pea gravel' when small) vs. 'crushed' (aka 'sharp') issue. Smooth will always leave voids and won't compact well, so I'd assume it'd be better in drainage situations
There are also products available that are of mixed grade for better compaction (I've beard it called 'crusher base' or 'crusher run'), which are intended for compacting before your sand course for patios and the like. In some areas, you might also be able to find recycled crushed concrete.
I'm not going to get into larger rock products, as there's just too many (flagstone, slabs, blocks (dimensional stone), tumbled blocks, cobbles and pavers, riprap, decorative boulders, etc.)