The attempt to glue made the situation worse.
Yes, but it will require a hammer, chisel and a grinder. Expect a lot of dust. The brush holder is still available, so you will be able to get a replacement. Once the old one is out, it is held in place with grout.
You will need to start with a grout saw to remove the grout in the joint between the holder and surrounding tile, it will lessen the chance of chipping the neighboring tile.