Subject to various caveats, either a heat gun (electric) or a torch (fuel/air) might help.
...If you have not already tested if the paint on there was lead-based, do that, and if it was, don't do this.
...Ventilate really, really well. Even without lead, likely to be a bunch of stuff you don't want to breathe coming off. A charcoal filter (organic fumes) respirator might be a very good idea as well.
...Don't burn the house down. The brick won't burn but working along the edges can be risky.
Painting brick is one of those things that is easy to do, and VERY hard to undo.
While I doubt it, being paint, you could experiment with an acid on a small patch - probably vinegar (acetic, 5 or 10%) as being relatively low hazard and probably already in your house. Muriatic (hydrochloric) is more common in masonry uses and stronger, but also more hazardous. In either case follow label instructions and neutralize with a baking soda and water solution or other base when done. But that's more for cleaning any "mortar haze" off the face of bricks from the mortar in the joints, and I doubt it would help much with paint residue.