Yes, that's how older dimmers, lighted switches, motion sensors and smart switches work. They need power to run themselves. But rather than supply hot and neutral to properly power the switch, they just hook themselves up in series with the incandescent bulb. This takes advantage of a fun fact about incandescents: when they are off, they are essentially a dead short.
But this fun fact is not true for LEDs, which have complex electronic power supplies to convert from the 120 volts AC to the 3 volts DC that LEDs actually run on. These power supplies respond unpredictably to this particular kind of power leakage. Quite often, the LED lights up a little bit, just like this. Now, you can see the power that the dimmer has been wasting all along.
It helps to buy LEDs rated for dimming, which have more complex power supplies designed to cope with this situation.
You can also get a dimmer that has a hard "off" position. They have a smooth dimming range from "low" to "high", but if you push past the "low" position, you go "over a bump" (stiff resistance) into a hard off position. Who knows? You might already have that, for most people it never occurs to them to try pushing past the bump.