I have 5 puck lights under my kitchen cabinet, I replaced the very hot 20W halogen bulbs with 5 x 2 Watt LED's. They were a little bigger than the halogen bulbs and I had to remove the glass cover because of the extra clearance, I feel safe without the covers because there is no fear the LED's would explode like the halogens would if water splashed on them. Before I replaced all the halogens I had only 4 LED's installed and 1 halogen. When the power was switched off the lights would shut down completely. When all 5 LED's were installed together, they would emit light in the off position at a dim enough rate that I am not worried about waste or cost because of the cost savings with the LED's. The reason the LED lights remain on when the switch is off is because they are connected to an X-10 Home Automation module so I can dim them and turn them off remotely. Due to the very dim, (about 5% or less of the full light of these bulbs I am sure I am still saving money over the cost of the halogen bulbs which were to warm to be near and were dangerously hot. If you have an X-10 device connected to your LED's, then I would not worry if they were constantly on unless you had so many in sequence it was actually wasting your money. Five bulbs on 24/7 at 5% probably still use only 10% of what the lights would cost with the halogens. The convenience of X-10 remote controls and switches far outweighs the inconvenience of a single switch.
The X10 switch is leaking current through the lights. The same current was probably leaking through the halogens that you had previously. However, the current was so low that the halogens did not visibly light. The LEDs are so much more efficient that they do visibly light.
Bottom line: turned off, the wasted power is the same in both cases, even though the LEDs ads lit. Turned on, the LEDs are taking much less energy.