I think your plan is basically sound but I'd beef it up a bit.
Number one, I'd definitely use a spring when you hang your heavy bag, that's going to take a lot of the shock off the support.
Unlike a floor joist, where you want a big enough joist that it's stiff, with a heavy bag, some flex is a good thing. The more the timber flexes, the less the shock on your walls. I am concerned that the strain will mess up your walls - and it could mess up the drywall on the other side of the walls.
Your idea to use 2x4 ledgers is probably undersized, too hard to get a solid attachment to the 4x4 beam and to the studs. I'd use a 2x6 or 2x8 lag screwed into the studs. Try to hit three studs with a ledger 36" long.
My first choice to hang the beam would be to rest the beam in 3.5" x 3.5" notches cut in the top of the ledgers. My second choice would be a hanger type bracket like the Simpson U44:
If you want to be extra protection for your walls, I'd put 3/4" plywood under the ledger, all the way to the floor, screwed into the studs all the way down. That will stiffen the wall so the drywall doesn't get messed up on either side of the wall.