Original answer below based on maims voltage 110v or higher. If as Harper suggests this is low voltage like a doorbell then just cover the box and don't worry about capping the wires.
Capping any wires you find like this is the safe & prudent thing to do. It won't hurt anything and would prevent wires from shorting.
If you have wires already twisted together do not change them unless you have a specific reason to do so. Just cap them "as is". It may be, for example, that there was originally a switch in this box connecting the red & yellow, and now they are twisted together so that the light or receptacle is always on - disconnecting them would cause problems. Your question was not 100% clear - if you actually have hot wires that were NOT covered already (e.g., with a wire nut) then you have exposed hot wires which is NOT a good thing and I would be concerned about what else may be wrong.
Based on your description, the white wire is likely a neutral. Since that is bonded to ground at the panel, there is no direct danger from it sitting uncapped because if it shorted to the box and the box is grounded then it would not immediately cause a problem. However, that is not safe under certain circumstances and therefore it should be capped with a wire nut for safety.
As far as false positives with non-contact testers. That can be a real problem. Typically that has to do with wires next to each other where one is really hot and the other is not but the tester shows it as hot anyway. But if you didn't have anything hot in the box then the tester would not show any wires as hot. So that red == hot indication is likely quite real.