I've had lots of problems in rental properties with tile floors over "waterproof" wood-based subfloors (either "waterproof" plywood or "waterproof" particle board).
These subfloors always flex a little bit under load, which inevitably ends up cracking joints between tiles. Sometimes cracks are impossible to see. Then water seeps below the tiles and degrades thinset mortar and subfloor. Eventually tiles come loose. Even if the wooden subfloor is protected by a waterproof membrane, the subfloor might hold but the thinset will eventually fail if it stays soaked.
I've switched to vinyl flooring. It's easy to install, it comes in a roll, just put it down on the floor... and it is in one piece and watertight. Add silicone caulk on the edges and that's it. Silicone caulk must be checked for mold or degradation once in a while, but problems are easy to spot unlike hairline cracks in tile joints.
It is also more comfortable than tile, as it feels warm to naked feet. That's a nice perk.
If you don't want to use vinyl and you really want to use tiles, then I'd recommend using epoxy grout. It is awesome also for wet areas around showers and bathtubs. Unlike cement-based grout, it is non-porous, so if it gets moldy, it is very easy to clean. Also it's much harder to crack.
The edge by the tub is actually breaking in due to it being wet.
Looks like you have to really check the subfloor for rot. A full bathtub is very heavy, it needs a solid support. Also rotten wood will flex a lot and this will crack all the joints, and tear silicone caulk in the corners.
My question is, what type of wood should I use for a bathroom sub floor
If you use vinyl which is flexible and does not crack... then the floor is allowed to flex a bit. I'd go with 22mm marine plywood.
If you glue the vinyl to the subfloor, DO NOT put glue all the way to the edges! Leave about 100mm (4") unglued. So when you walk on the floor and it flexes, the subfloor will move down a bit, but it won't pull the vinyl down, so it will stay attached to the silicone corner joint and not tear it off.
If you take your current subfloor out, check under the tub for rot, also check the drain and all the fittings and joints while you're in there.
and should I coat it with waterproof sealer?
Not 100% necessary if you use vinyl floor, although it can't hurt. You should use plywood or OSB rated for "outdoors" or "marine" which means it's made with waterproof glue.