Three methods I can think of.
Enlarge by friction
Your idea of wrapping the bit in sandpaper isn't even bad. The only "right" way to do this is a different bit, and that's not even worht it.
You just probably used the wrong sandpaper. Try some 80 grit and make sure to go in a back and forth pattern, roughly every half inch you plunge.
There also exists a tool like this called a drum rasp. It might help your situation.
Enlarge by cutter
If you REALLY want to make this PERFECT, you can use a router. Take it off once side with a 3/8 guide bushing and a 1/2" bit, then put on a 1/2" bushing and do the other side.
You could even use a tracing router bit. They have bearings and blade that'a almost identical in diameter, except the blade almost is always JUST wider than the bearing. It'll take off about 1/128th of an inch at a time, so you'll have to trace around your hole about 20 times to take off such a small bit of material.
Make due by Force
Keep in mind the pipe isn't perfectly round either, and your hole can be off just a little bit as well. Even if you take off half of what you really need to, the wood will flex and look gapless against your pipe. The pipe could be bigger than you think, the hole can be smaller than you think. Tolerances on both of those are actually pretty out there.
You might even find it easier to narrow up the pipe instead. Metal doesn't gum up sandpaper as easily as wood does.
You can, after a little sanding if you please, just force it in. Put a piece of sacrificial wood on top of the hole, get a hammer, and bang the pipe in there. 1/16 is 0.0625... There's nails that are wider than that, and they can be banged in with a hammer. That's only 31 THOUSANDTHS of an inch on each side.