There are indeed ready made products (called battery eliminators) that perform the function that you envision.
The "active cell" has the wires that inject the 3VDC from an external source into the battery holder. The "dummy cell" is just a plastic block that has an end to end wire. (Note that when using the pictured type device it is necessary to make sure that your wall mounted device just has two wires from its battery holder off into the internal circuitry).
Installation will require some modification of the device's battery door unless it is possible to operate the device without the battery door in place.
The whole kit also includes the wall wart type AC mains adapter:
If you start the search for "AA Battery Eliminator" you will quickly find that these units are moderately expensive. Based upon that you may be encouraged to build your own. The fake cells can be constructed using suitable diameter wooden dowel material cut to length. Sometimes the end caps can be taken from typical dead AA carbon zinc batteries as these end caps are often removable from the cell and can then be glued in place on the ends of the dowel. A hole drilled from end to end of the dowel and one from one side into the center can accommodate the wires.
In some cases alligator clips can be used instead of dummy cells:
Rechargeable AA Cells
Keep in mind that it may be another option to invest in a couple of sets of rechargeable AA cells and a separate wall/USB powered charger unit. The cost of such solution may be similar to the battery eliminator.
Do be aware that many low cost re-chargers are garbage and will do more to destroy your rechargeable cells than the actual use of the cells themselves. There are some decent battery chargers worth considering such as this unit from Lacrosse. I have the the Lacrosse BC-700 model and can report that it is an excellent product.
Buy AA Cells In Bulk
The overall simplest solution would be to purchase the AA alkaline batteries in bulk. With a little bit of research you can find bulk purchase options that include free shipping and prices of just over a quarter of a dollar per cell.