I found a way to use a large chimney balloon and pretty much block all downdraft. Sadly, the fireplace and hearth are still very cold, and it sucks heat out of the surrounding floor too.
Perhaps the firebox has tiny cracks letting in air, or the whole thing (brick outside, stone facade inside, no idea what's underneath) is just too heat-conductive. I'm already planning to hire/rent a thermal camera (for the whole house) which should help figure it out. But after that, what can I do to insulate it?