I have taken the wall paper off of a wall, to find there are several layers of paint under it. I have sanded till it's as smooth as I can get it, but there are still patches not even 1 mm deep that look like little indentations. I have bought polyfilla smooth over, but when putting this on, layers of the previous paint are coming off. The peeled paint is getting stuck to the roller, and then making the rest of the surface have lines across it. Should I sand more and risk damaging the plaster? Any help would be appreciated as I am a total novice.
You could use a fine surface filler to fill the small imperfections and sand that smooth.
Alternatively, if the old paint is loose enough to pull off on the roller, you should probably scrape it off before painting anyway.
Finally, if there is too much surface imperfection, you could paper the wall with lining paper and paint over that.
Using a coat of white pigmented shellac will harden and seal the distressed paper that is flagging. Once dry, the surface will be able to be sanded lightly, to remove fuzz. This product is sold as a primer and is a good water-based stain blocker.
Deeper imperfections will then need to be filled and sanded. The non-latex, non water base fillers will not soften the surface. Oil based ones like MH are also less shrinking.
I would avoid sanding the wall because it is likely that the paint has lead in it, and sanding will put fine particles of it in the air which can make you and your family sick. Before you go any further, I recommend strongly that you purchase a lead paint testing kit to determine if there is any lead paint to deal with. These kits are inexpensive and are sold at home improvement stores and some hardware stores.
To fix the holes/depressions in the wall, I recommend that you use a paint scraper like one of those "n in 1" tools that has a good scraper on it. They usually have an end on it shaped like an awl that you can use to tap the wall to see if the paint is loose. Scrape as much loose paint as you can. While doing this, make sure you use a dust mask with a respirator and eye protection. You should also have good ventilation.
After the loose paint has been removed, you should prime the wall with stain blocking primer. This will encapsulate the possibly lead containing paint. After that has dried, then you can use joint compound to fill the defects. Once the defects have been taken care of, prime the wall in those areas again and then put on a final coat of paint. This will give you a professional level result.
It could be that some of the compounds in the smoothover are reacting with the paint to loosen it. If not, then you are dealing with loose or insecure paint and you'll need to remove that before you have a surface you can smooth.
If the paint seems secure until you use the smoothover, stop using it and find another compound to fill in the dents and low lying areas.
If the paint isn't secure, remove all the loose paint, then sand, then fill and smooth.
As others indicate, given the history you should check for the presence of lead and take appropriate action if there are lead based paints in your home.
Another option is to use a primer to secure everything left on the wall, then use smoothover or a similar product on top of the primer.