A 'handy man' did work on my plumbing and shut off the main water to the house. He did not unplug the well pump, and it got really hot to the touch. When we hooked everything back up again, the pump had started to leak.
Is the pump serviceable?
Sounds like a really odd setup you have there. The main water shutoff is between the pump and the pressure switch??
That is the only way that the pump would remain running like this. Water pressure in the house on the other side of the main water water shut off drops telling the pump to turn on. The pump stays running till the valve is opened again.
Since you can see it leaking, we have to assume that it is an above ground impeller pump. Several things here are of concern if this is the case.
Under completely blocked outflow, the pump develops high pressure, probably in the 150psi range. The only reason it doesn't go any higher is that the clearances between the impeller and the housing allow for leakage back to the inlet side. You have a basic hydraulic dynamometer brake which dissipates the input power as heat. The water increases in pressure and temperature leading to boiling and cavitation, the increased pressure on the shaft seal causes it to push its components together tighter causing greater heat, eventually melting the seal contact surface if it is the standard carbon ring seal and leading to leakage.
As the bearings are contained in the motor and are either sleeve or ball bearing and separated by an air gap between the shaft seal and the impeller side bearing, you may not have any damage there. The seal, though is definitely toast. You also have to check the impeller and housing for erosion, if the edges have guttering, that's toasted as well.
Repair consists of removing 4 or 6 bolts to remove the impeller housing. Inspect the impeller and impeller housing for erosion, determine if the impeller is pressed on or has a reverse thread onto the motor shaft and use appropriate removal technique. Once you have the impeller out of the way, you have to push the shaft seal out, check the motor shaft for damage, spin the motor over to feel for bearing roughness and push a new seal in place.
My experience is with largish irrigation pumps which have substantial cost and therefore repair is an option. A typical well pump is probably a replacement item and not worth the effort.
For that assessment, you would have to provide Manufacturer Make and Model so we could see if manuals are available. The more common above ground well pumps used to have repair parts available.