Disclaimer: I am not a professional. I am an electronics hobbyist who has researched and tinkered with my own home heating.
Yes, a relay is almost certainly all you need.
In conventional thermostat systems the device being controlled supplies 24 VAC (if it has an electric supply line to it) or a small DC voltage generated from the heat of the pilot light, for gas appliances not otherwise powered (a so-called "millivolt" system). In either case, the circuit consists of the power source, the gas valve, and the thermostat contacts.
(Side note for completeness: electric heaters are sometimes controlled by thermostats which directly switch the power to the heater. It's still true that a relay can replace the thermostat, but in that case a relay capable of handling the full voltage and current must be used, and installed safely. I have not heard of line voltage control circuits being used for gas appliances.)
You can cautiously confirm this for your system step by step:
- With a multimeter, check the voltage across the thermostat terminals with the heat off or the thermostat disconnected. It should be 24 V AC or something like 1 V DC.
- With the thermostat disconnected and the meter in resistance/continuity mode, check that the thermostat is closing a circuit across R and W when it is on and not when it is off.
- Having established it won't do harm by step 2, touch together the R and W wires going to the stove and expect that it turns on.
If all of that is as expected then you can add your relay without fear.
By the way, it is also entirely safe to hook up two thermostats in parallel to the same control wires (two closed relays are the same as one). In particular, you could add your Alexa-controlled relay and also connect the stove to a thermostat so that the stove is on if either of those controllers calls for heat.
You could also consider controlling the stove with the "emergency heat" / "second stage heat" terminals of the same thermostat that controls your heat pump, but that's trickier to do right because you're connecting two separately powered electrical systems and to make sure there are no consequences of that you'd want to use another relay as interface, with a 24V coil powered by the thermostat-c-wire from the heat pump.