The five gallon bucket of concrete will not keep a 12 foot tall 4x4 post secure. You would be very surprised what a small amount of effort it would take to simply topple it over. The bucket idea may work to keep a 4 foot post upright most of the time but even that could be pushed over with surprising low effort.
From experience with building fences I can say that digging the hole in the ground is going to be the best choice. For a fence post where the 4x4 posts rise about 5.5 to 6 feet above the ground the holes need to be 2 to 2.5 feet deep to secure the posts firmly. For a post to rise up to 12 feet you will need a hole that is at least 3.5 feet and preferably a bit more in depth to secure a free standing post. That means that if you want to have your bat house up at 12 feet you will need to start with a 16 foot (standard length) 4x4 post.
If your reason for not digging a hole is a really valid one (like there is solid rock 6 inches below the soil line) then one possibility that you could consider is to get four five gallon buckets. Embed the post in one as you proposed in your question and then embed a long sturdy eye bolt into the other three. Place these other three buckets out from the base of the 4x4 post a good distance (suggest about 10 feet away) at 120 degree spacing. Before hoisting the tall post into position install three eye bolts into the post at a height of eight to ten feet from its base and attach some small diameter cable to each of these eye bolts.
After the tall post is erected then string out each of the cables to one of the buckets with the eye bolts and secure the ends of the cables there. Note that you will almost for certain want to secure these cables in a temporary manner to begin with because some iterative fine tuning of the cables will be needed to get them all taut with the tall post properly vertical.
Note that these support cables can represent a hazard to people trying to navigate around them so do take heed of this and take the necessary precautions. Also be aware that with enough force against the tall post it would still be possible to pull one or more of the support buckets of concrete from its position on the ground. For instance it would probably not be a good idea to ever lean a ladder against the post in order to climb up there to install the bat house (do that before hoisting the post into position).