The tankless water heaters I am looking at require 8 AWG wire but the existing wiring is 12 AWG.
Can I do this without having to rerun new wire?
If a heater requires 8 AWG wire, it's because it requires more electricity than a smaller wire is capable of safely delivering. If you used the 12 AWG wire, first the 20A breaker would trip, but if that was upgraded (DO NOT) then the wire itself would turn into a heater and burn down the house.
What I'm saying here is that when too much electricity is run through a wire that is too small, it heats up. Wire sizes are based on keeping a wire cool enough to not cause a fire when it's used. Wire sizes are hard requirements, not suggestions.
The only option here is to run new wire or choose a smaller water heater.
You cannot use an electric tankless water heater in place of an electric tank without a significant upgrade in wiring and most probably adding new breakers to make a total of two or three breakers. Even new wiring of the proper size might not be enough because your electric service might not have enough capacity for a central electric tankless WH.
What is the make and model of the tankless water heater you are hoping to install? If you look at the installation reuirements, it will probably require two or three 40 A breakers and of course separate wiring for each breaker.
The simplest course of action is simply to replace your electric tank with another electric tank which has the same power requirements.
If you have a natural gas furnace, you might be able to install a natural gas fired tankless water heater, but this might require running new gas pipe to supply both the furnace and the water heater. And it would be very expensive to have installed.
To give this question some perspective, let's consider a chart. This shows how hot your wires will get for a given AWG size and current.
If we follow the 12AWG line, which is required to be protected by a 20A breaker, we see that the wire will heat to about 10C above ambient (ie: ~32C in normal room temperature, or 90F for those who refuse to join the modern world) when carrying its full rated load of 20A
If we follow the 8AWG line, which must be protected by a 40A breaker, we see that it too will heat to about 10C above ambient when carrying its full load current of 40A.
If, however, we follow the 12AWG line out to where it crosses 40A, we find that the wire will now heat to 60C above ambient, or 82C (180F) in our standard room when overloaded to 40A. This is not trivial.
The temperatures below, further, are for single insulated, "free air" cables. In a house you will certainly have double-insulated wire and it will be stuffed into walls, through insulation, and in all types of other circumstances that will only make this situation worse.
Use the right wire gauges - always. It's not an option and it's not a nervous-nellie fearmongering story about burning your house down - the difference is dramatic. Rules are there for a reason.
8AWG is typically used for 40 amp circuits. Assuming North America, that's 4800 watts.
12AWG can handle 20 amps (2400 watts at standard voltage), so in order to use the same wires all we need to do is reduce the current, and we do that by raising the voltage. 8AWG is used for 40 amp (4800 watt) circuits.
The extra-good news is we only need to double the voltage, and you can do that with a small rewiring at the panel, no transformers needed. We can now send 4800 watts at 240 volts over 12AWG wire - it's now only 20 amps.
Wether the tankless heater can handle 240v is a different question. It might be switchable - two 2400w elements connected in series or in parallel. If that is the case then your expenses are one 20 amp double-pole breaker and 5 minutes reconfiguring the heater.
If the heater is not configurable then you use a step-down transformer at the heater, which will probably cost at least as much as new wiring.
All the above assumes the heater uses 40 amps at 120v, single phase. Honestly, I think this is unlikely as it's very rare for high-power appliances to use plain 120v. If it is already a 240v heater, and you still want to use the same wiring, then you need a step-up transformer at the panel AND a matching one at the heater.
At this point new wire will definitely be cheaper.
In summary, no, you don't have to run new wire. But running new wire may end up costing a lot less.