I tried hammering the screw a bit to loosen it. Any suggestion? I'm thinking of getting something to saw off the holder and hammer the screw into the wall.
If you have a small socket set with a Philips bit, you can use that in the 1/4" socket. This would allow you to put pressure on the screw while removing it.
The problem is most likely that the hollow-wall anchor the screw is threaded into has let go of the surrounding material. It's spinning freely. You want to either hold it while you turn the screw, or simply grind off the screw head.
Try reaching behind the bracket with a needle-nose pliers. Grasp the anchor and hold it securely while you turn the screw. Since the wall is already fairly damaged, don't be afraid to dig the pliers into the surface to get a better bite.
Otherwise, use a rotary tool or angle grinder to remove the screw head. Once it's off and the bracket is removed, simply drive the screw shank below the surface of the wall and make your repairs.