Tore the walls off and found the chimney looking like this:
It's seen a lot of leaking as you can see from the streaks. The exterior portion was rebuilt last year. That horizontal crack running across the middle goes all the way around on all 4 sides.
The house is ~70 years old. The attic got finished some time later. Late 60s based on the wallpaper. You can see some kids used to play up there before the chimney got walled up. The chimney only sees natural gas exhaust now but I think there used to be an oil furnace.
Is the crack something to be worried about? Any reason they covered the whole thing with mortar other than to hide problems? The visible portion in the basement is exposed brick.