Even if you only use it at on/off extremes, it's still a dimmer switch.
Early "smart devices" like dimmer switches, motion sensors, etc. have a problem. They need power to function themselves. Normally that's gotten from the hot and neutral wires. However these "smart devices" are meant to replace plain old switches, which are not powered (except by you) and do not need a neutral. So to power themselves, they leak a small amount of power through the light bulb, even when they're off. This works with old incandescent bulbs. It doesn't work with the power supply/driver circuits inside CFLs and LEDs, unless the drivers have some extra circuitry to allow for it.
The early dimmers work by pulsing the power to the lamp. Again this works great on incandescents. But the switching power supply/drivers in CFL/LEDs see this as a power defect and automatically "correct for it", defeating dimming. This is hard on the LED and hard on the dimmer. Dimming LEDs need special circuits to "reverse engineer" what the dimmer is doing and set an appropriate dim level on the LED.
One way to correct this is to stick one incandescent in among the LEDs. This will become the "preferred" path for that stray power the dimmer is leaking. You might get away with the same trick using one dimmable LED.
So you have a dimmer, whether you like it or not. It is leaking power through the bulbs and the non-dimmable LED bulbs do not like that. You need to figure out what you want to do about either changing the dimmer switch to an on/off switch, or fitting it with compatible bulb(s).