If everything else is hooked up correctly, then you have a bad switch. This is easy to diagnose if you have a meter that measures continuity. Most digital multimeters have this option. Kill the power to this circuit at the panel before testing for this or performing any work on this circuit. The connection between the two screws of the switch will be open in the "off" position, and will be closed in the "on" positions. If the meter reads "closed" in either position then you have a bad switch. If you don't have a multimeter, just put a new switch in and see if that fixes it.
The other possibility is incorrect wiring. If you replaced a fan/light combo then you probably had 3-wire going from the switch box to the device box (black, red, and white wires). When you wire the new light fixture, select one wire and cap it on both ends so you won't use it by accident (I usually cap off the red one, we usually use black in residential applications for the hot wire). This will only leave you with a black and white going to the lamp. After that, it's a simple act of wiring the black to one screw of the switch and putting your whites under a wire-nut in the switch box. The black from the panel goes on the other screw. Don't forget to bond your grounds to the boxes if they're metal.