Wow, nice fuse panel. It's well preserved and probably still fully effective.
You can only hope the modern panel you replace it with holds up so well!
Although we use it all the time, the term "hot" can be a little ambiguous.
Keep in mind that voltage is only meaningful between two things. Sometimes when referring to voltage to ground / earth, we'll just assume that's the other thing. So if we say "the red wire is at 120V," or "the black wire is at 120V," we really mean there's 120V between those wires and ground.
(There are ungrounded systems where there is voltage present between parts of the system, but no voltage to any of the conductors and ground, but those are seen in big industrial systems, not your home.)
I suspect you're testing with a non-contact voltage tester. Remember that these testers generally measure voltage to ground.
In the US, most single family homes are supplied by a Edison three wire service. In this system, there are two lines (red and black in your fuse panel) and a neutral originating on the same transformer winding. The lines originate at the end of the winding, the neutral, which is grounded, originates at the middle. There's 240V from line to line, and 120V from either line to neutral.
So with a non-contact voltage detector, you'll only detect voltage on line1 and line2, not on the neutral; it's grounded. But if you put one lead of a voltage meter on white, and the other on red or black, you'd read 120V.