An intermittent problem like that is almost always a broken switch or a loose connection. Often, you can "feel" when a switch is broken, so if it feels normal to you, then a loose connection might be the issue. Either way, you need to pull the switch out and check it out. If you are lucky and it is a loose wire, you'll just need to tighten the screw holding the wire and you're done.
It's possible that whoever installed the switch didn't use the screw connectors and pushed the wires into the holes on the back instead. Ugh. If that is the case, just cut the wires at the back of the switch and try the screw connectors instead.
Switches are cheap, so don't spend a lot of time trying to fix this. If the problem isn't obvious, replace the switch.