I'm renovating a bathroom. Is there some kind of paint or chemical that I can apply to the subfloor around the toilet flange to keep it from rotting?
As Tester said there shouldn't be any leaks and therefore no rot. Any leak from the wax ring should be repaired immediately.
However, pressure treated lumber will last longer in a wet environment if you are concerned about overflows, sweating, or other reasons the floor may get wet in that area.
Coating regular wood is not going to stop it from absorbing water on the edge or if it is soaked with water.
If you are laying tile in that area: They make a liner product called Kerdi and roll on products like HydroBan for use under ceramic tile in wet areas like showers. Ceramic tile is not waterproof by itself.
If you're putting in new wood, a patch of pressure treated would be the way to go. If you search for "wood preservative" you'll find a lot of products. Thompson's Waterseal, Waterlox, Rustoleum Woodlife, Cuprinol, many others. Something's available at just about any hardware store.
Some are only labelled for outside use, you might be OK with a small patch, but I'd check with the manufacturer. Most are toxic to some extent, you have to wear gloves, ventilate well, etc. - read the label. Many won't go on top of any existing finish - probably not an issue with a subfloor. There are lots of non-toxic alternative products available online if you prefer.
If you want to keep it simple, you could probably just do something simple - prime and paint it, seal it with polyurethane, oil it with tung oil - that would be quick, easy, and cheap, and probably better than nothing.
We assume people will not realize their wax ring seal has been compromised until it too late. We try to use plywood for an underlayment or better (concrete board for tile, etc.) We definitely stay away from particle board and the likes.
Then, we coat any wood in a thick layer of RedGuard just around the toilet. The trick is to get it coated on ANY exposed wood at the penetration; that includes edges. You can use something as simple as a chip brush.