For common bathtubs the correct overflow gasket should be wedge shaped with a flat side and an angled side. The flat side mates with the overflow horn (the piece directly behind the hole) and the angled side mates with the backside of the tub, fat part down to match the slope of the tub wall.
I have seen them made of both solid rubber and foam rubber. I prefer the foam rubber because you can get a decent squish on them and they will conform to irregularities on the back side of cast iron tubs (sometimes the porcelain glaze is slopped a bit back there, sometimes the casting is rough).
Do not attempt to smear caulk around it, it won't work. They can be replaced without ripping out the tub. Remove the overflow plate and take a look, you should be able to pull out the old gasket with a combination of long tools and needle-nose pliers (I use a hemostat- see picture).
There is always a bit of "give" on the overflow pipe, which helps, but try not to move it too much or you risk compromising the connection to the waste tee below (if it is the type with a slip-joint connection, many are glued plastic).
Get a piece of slim stiff wire about 8" long and bent in a "j" shape on one end and keep it handy. Get a helper too, unless you have 3 arms and hands. My technique for this is to install a plate screw into the horn so you have something to grab to apply pressure to the new gasket as you slide it into place (the screw will also prevent the new gasket from falling behind the tub if you drop it). Manipulate the gasket with your free hand, using tools if you have to, until it is positioned properly (you may have to jam it in a bit). When the gasket is in the right position pull back on the screw to hold the gasket in position between the horn and tub wall. Now use the wire to grip the bottom edge of the shoe to maintain pressure, have your helper hold it. Remove the screw and install the overflow plate (the wire should lay where the overflow ports on the bottom of the plate are).
Unhook the wire from below and remove, and you are done. Be aware that the seal provided by the gasket is not really meant to be totally watertight like the drain is, it's not meant to be submerged, it's there for incidental overfill protection.