so I moved into a new apartment last month. The water wasn't very hot so we thought it was just because the weather has been getting cold. Turns out the water heater has only been on pilot, so the water wasn't getting hot. Does this mean that the water heater was leaking case because the water was only luke warm? We didn't smell any gas.
Many gas appliances have a "Pilot mode" where they will just burn a pilot light, and ignore the thermostat setting. Typically a 3-position control reading "Off-Pilot-On" and it sounds like yours is probably of that type and was in "pilot."
Turning this into a hunt for a gas leak is just plain illogical.
Almost certainly what the tech flipped from "Pilot" to "on" (and was internally laughing all the way to the bank about.)
If the water heater has a simple mode to just maintain the pilot flame it should be as simple as putting the water control into active heating control mode.
It is possible that the temperature control knob on the water heater was simply turned down to its very lowest setting.