Interesting. I can think of a couple scenarios.
First, the switch may be causing a small arc as the contacts separate, and that is tripping a GFCI/AFCI breaker in the panel. The switch may also be worn to the point where it can cause a short from hot to ground; this would be unlikely to trip a breaker, but it could happen. Lastly, there may be exposed wiring in the switch box, coupled with a loosened mounting of the switch int the box, causing a short between wires when the switch is flipped.
In all cases, the most likely culprit is the switch. Single-pole 120V15A switches are a buck-fifty a pop, so it can't hurt to just replace it. Make sure there's no bare wire that can short against a ground; I've found it helpful to give the switch or outlet I'm installing a wrap of electrical tape, which covers the terminals and other energized metal around the switch body and helps prevent shorts. Make sure no bare metal can touch any other bare metal when you put the new switch back in the box.
If that doesn't solve the problem then the next likely culprit is a GFCI or AFCI panel breaker, which is tripping in response to arcing at the point where the contacts in the switch disengage. This is normal; a very small arc will occur when the contacts are very close together right when the switch turns on and off. AFCI breakers are sometimes prone to "false positives" like this; you can trip an AFCI breaker plugging in an appliance that's already turned on, for instance. You do not need an AFCI breaker on bathroom circuits; only bedroom circuits built or last renovated after 1999. So, as long as your bedroom and bathroom are on separate panel circuits (and they should be), you can use a normal breaker. You DO need GFCI protection for outlets on bathroom circuits, meaning if you remove the AFCI breaker and replace it (or, more accurately, if you have a licensed electrician do it for you), you will need to replace the most "upstream" outlet on that circuit with a GFCI outlet, and you will need to make sure all outlets in the "wet" areas (within 4 feet of a sink, tub, shower or toilet) are protected by that GFCI.
It may also be caused by a GFCI breaker protecting the switch and fan, that is detecting current being driven through the neutral side (which remains closed when the switch is turned off) by the rotating fan motor as it winds down. A motor is basically just a generator in reverse, after all. With the big flywheel of the fan keeping it rotating after the power is cut, it can induce voltage, causing an imbalance between the hot and neutral that trips the breaker. Again, in this case you can solve the problem by replacing the panel breaker with a normal "slow-trip" breaker, and just protecting the outlets with a GFCI receptacle instead of the whole circuit.
I find it hard to believe the fan could be causing a tripped breaker if the breaker isn't a GFCI, but if you still haven't found the cause by this point the fan would be the last thing to check. Ensure the motor is securely mounted in its housing, and that all wiring is properly insulated with no possibility of a short. Make sure the motor itself is in good repair; there may be a worn bearing or bushing that is causing the armature to contact the shell during a sudden change in stress (like when the electomagnetic force that spins it goes away). I can't think of much that would cause the fan to trip a normal breaker when turned off, that wouldn't be causing far worse problems when turned on.