I have recently moved into a late Victorian UK terraced house. I've stripped some of the old wallpaper off and the party walls look like this underneath. It can't be more than 1-2mm thick and in general it's fairly solid but where it has cracked it quite flaky.
That is probably old wallpaper paste. Generally it is water soluble and will wipe off with a little work and a wet sponge. If that gives you problems rent or use a wallpaper steamer.
It seems that you are confusing the very thin outermost yellow layer with the layer of finishing material beneath that. The yellow material is dried glue/paste from the wallpaper installation. The layer underneath is the finishing layer of cement, plaster, or whatever they used to cover the bricks. The latter is what is crumbling.
