I’m refinishing my hardwood floors and some of the old varnish isn’t coming off. Before we apply a water based polyurethane do we need to remove all the old varnish or can some of it stay? Our floors are oak.
You probably can leave it, but you might not want to, depending on what you're going for.
If water beads up on the finish, you simply can't leave it, for obvious reasons with a water-based poly. Same with stain, if you're doing that, you won't want to stain some parts and not others (where the stain didn't take because of old finish).
If the pre-finished spots can take water without beading, then you don't have to remove it, especially if abraded. The spots with old finish will likely appear darker than the bare wood. How much darker it will look depends on a lot of factors.
If you are staining or just want a smooth finish and don't mind a more rustic look, such darkness might not matter to you. If 90% of the floor is perfect (flat, clean, new-wood-like) and just a few spots are off, that's likely not going to look acceptable. If the spots/streaks are well distributed and largely confined to individual boards, then it can look quite striking, as long as you're going for something more rustic that doesn't look brand-new.
you might try renting a floor buffer instead of a sander, and using a green pad with an 80 or 100 grit screen under it. That should flex slightly to "scoop out" finish in any dips in the floor that your flat drum passed over. It also strips finish very quickly compared to sanders and is often cheaper to rent, and the screens do a lot more area than sandpaper, so you save on supplies. If the low-lying wood is in decent shape, and you remove the finish, it will look a lot like the parts you drum-sanded.