We have a large 3600 sq ft vacation rental that typically houses 10-14 people. We have a 50 gallon gas tank water heater but guests have complained that the hot water runs out after a while.
Only the maximum number of people is relevant so if you say it's good for 14 people in your ad, that's the number you worry about. You need about 10 gallons tank per person based on recommendations for regular household use, and about 7.5k Btu per person. If it's for holiday use there may be considerations like people showering every time they swim to take into account. It's theoretically possible to supplement your system with a tankless heater, but the control would be complicated and the tankless heater would still need to be sized to cover the maximum load you want.
We have three full bathrooms as well as three sets of washer/dryers.
With tankless heaters, you size the heater as close as you can afford to the maximum reasonable load. As soon as you draw more water than it can heat, output will cool down and be noticed, but your guests might tolerate not being able to shower, run the sink hot and do dishes and laundry at the same time as long as everyone gets a hot shower before they throw in some laundry or soak some dishes. Because a tank heater can store significant amounts of heat, much lower Btu ratings are acceptable. This means if you want to use a tankless heater you might need to upgrade your gas line.
We are contemplating installing an additional gas tankless water heater in addition to the current gas tank unit.
Another simpler/more reliable option would be to use the current unit to feed say one bathroom and one laundry and one kitchen sink and adding another unit of whatever type to feed the other fixtures. This will also help with flow limitations since if your water heater is missized so badly it may mean pipes are also not sized for so many fixtures.
Would we run this in series (i.e., both units would be connected) and if in series, which unit would you install first? (i.e., would you have the tankless unit preheat the tank or would you have the tankless unit only kick in when the tank unit was depleted)
You could have it in series or in parallel. In series, it would come before the tank heater and only start heating when the output dropped below a set temperature. In parallel, when the tank heater temperature got too low it would switch in and close off the tank heater. I personally would favor series as you wouldn't need the extra solenoid valve, just electrical control. You would want to have the temperature controls for the system carefully set. That said, Your existing tank system is sized for roughly 4 or 5 people, so the pipes into and out of it probably are too.
Also, a plumber suggested installing a thermostatic mixing valve to give us more capacity. If we were to install this, where would we install it in the system?
I'm not sure how price works out on those, but if you use one on the output of the tank, it allows the tank to store water safely at a higher temperature. (Make sure you get a fail safe one intended for domestic use. The total output volume of "Hot" water is a mixture of really hot water and cold water. If pipes were inadequately sized, it would theoretically give you more total hot water throughput to put smaller thermostatic mixing valves closer to the fixtures.
To verify your pipes are adequately sized, you can let the current heater heat up and then turn everything on at once to verify everything is hot until the hot water runs out. That would simplify things.