I have a room directly above my garage and it's usually 10+ degrees difference in temperature compared to the rest of the house. I'm planning on insulating the door and the garage ceiling with rigid insulation boards. I have been reading prior suggestions and I see that most also suggest to close up the ventilation windows in the garage. However, I have my water heater in the garage so I'm thinking that's the reason for the ventilation windows. If I close them up, will that be a problem?
When you say ventilation windows, do you mean operable glazed windows or small permanently open screened openings? Either way, fuel burning appliances require adequate free openings in enclosures to ensure proper exhausting of combustion by-products. Electric water heaters do not require such openings.
Your local building authority probably has specific requirements for such openings which you need to comply with. An example may be something like the garage must have at least 50 cubic feet of volume for every 1000 BTU/h of appliance heat output. If this requirement is not met, provide two openings in an exterior wall with at least 1 square inch free area opening each for every 4000 BTU/h of appliance heat output.
If your locality has no such requirements, following the above recommendations would be adequate.