I'm replacing a one piece fiberglass tub with a two piece (tub + surround) unit. In the bathroom right now, there's just drywall, no special material. Should I replace the drywall with greenboard, or with a cement board material (i.e. Durock)? How many feet adjacent to the but should be replaced with one of those two materials? Should I also replace the material behind the vanity with one of those two?
Greenboard (or any paper faced drywall) is an unacceptable substrate for any wet application (shower or tub stalls). Regardless of tub stall surface (tile, fiberglass, etc) you will get some moisture penetration through the shower liner and this will eventually create bigger problems.
You need to install Hardibacker or Durock (or any cementitious board) as the substrate for your tub/shower stall. Matter of fact, in many municipalities it is required by building code. It only needs to be installed in the area of the tub/shower, though no need to extend further out.
The rest of the bathroom can be done in greenboard though (and that isn't a bad idea).
As long as you can ensure that the surface is sealed with a waterproof layer then you don't really need to make any special modifications. The main thing you need to ensure is that there's no way water can get onto unprotected wall or floor surfaces.
In the UK baths are fitted into rooms that have normal plaster and the walls tiled to provide the water proofing.
If you have to replace any of the existing drywall then it might be worth going for a special water resistant panel (e.g. Aquapanel), but they are more expensive.
For right around the shower/tub I would use cement board since mold will not grow on that. If you are worried about the other areas of the bathroom you could use the "green" board stuff, though if it cost a little more than regular drywall.