If there is no ground wire (bare or green) coming into the box, grounding to a metal box is the next best thing. But it should be done by putting a grounding screw into one of the small threaded holes in the metal box, or using a grounding clip. The grounding wire from the switch is then put under the screw or into the clip.
This will ground the switch if there is a well connected metal sheathing on the cable coming into the box, or if there is some other connection of a grounding wire to the box.
If there is none, you have an ungrounded switch. Assuming it functions, this poses no more risk than we routinely put up with when switches were all ungrounded a few years (decades?) ago.
If the switch was fully ON and you tested across the hot and the switched hot, it should read as a short (or close to it). If it were open or dimmed, it should show more resistance than 0, but how much depends on the position and circuitry.