The full list of reasons (partially based on some other good answers here) seems to be:
- design improvements (a "good thing" on the road to the world having the best screws)
When people come up with improved designs, those start to get used.
- network effects (a "bad thing")
Some people will stick with older, inferior designs that have many users, since those will be easier to find and they will already have the screwdrivers and bits for them. Also...
- economy of scale (a "bad thing")
...gives an advantage to more established designs (even if they are terrible), as production of those will be large-scale and efficient, so new designs (better or worse) will tend to be more expensive.
- patenting (and other business reasons that have nothing to do with design) (a "bad thing")
As Chris Cudmore points out in a comment to a different answer here, this is the reason Phillips screws are more widespread than Robertson (square) screws, despite Robertson ones being generally better (far more difficult to strip the heads. ...and what was Ford's first choice again? ...exactly.).
- desire to have some things be tamper-proof (mostly a "bad thing")
Guarding against vandalism (special washroom stall screws) and preventing theft (special car wheel lug nuts) are decent goals, but the motives are not always as good. (Is Apple just trying to protect users against themselves with pentalobe screws, or are there other, worse reasons for them, such as pushing more people back Apple for any repairs?) The tamper-proof requirement is pretty clearly at odds with what most users would want, though, as each variation means another set of screwdrivers or bits someone would need to own to be prepared. Of course the fact that the designers of these screw heads often make money off each screwdriver or bit sold points to that being one reason the screw is out there in the first place.
- different applications (mostly a "good thing")
To end on a positive note after all of that, the different applications of screws really do leave room for multiple "best" designs. The points about wet applications and applications with small countersink-type heads (which can only fit much smaller bit sizes of the grippier designs) are great.
(Still... We would be much better off if we just started using Robertson screws 99% of the time!)
Actually, triangle heads should be best. They should have the best torque (as long as the sides are vertical), while still being simple to cut (low manufacturing costs). I just looked up whether they exist, though, and apparently they are being used as yet another tamper-proof design.