In my bathroom there is vinyl on top of plywood, which is nailed to the sub floor. Should I pry up the plywood before putting down cement backer board for tile, or can I go over the plywood?
The plywood is there to make the vinyl the same height as the rest of the flooring in the house. If you add backerboard + tile on top of it, you are going to add somewhere in the range of 5/8" to 3/4" of height. That is enough to be a tripping hazard, and it also wouldn't look very nice.
Pulling up the plywood and the putting down the backer board is the right thing to do.
Tile should be sitting on a minimum floor thickness of 1-1/4". Assuming the plywood sitting on your floor joists is 5/8", you need another 5/8" plywood which must be screwed down with floor screws every 6". Drive longer screws into the floor joist to secure the original plywood and the new securely.
Another option is to use Schutler Ditra which is a 1/8" "membrane" that provides the same strength as 5/8" plywood but without the added thickness of the plywood. The product is fairly expensive however. The also have a thicker membrane as well.
Bottom line: insufficient thickness will result in cracked tile and grout over time.