If you have threaded iron gas pipe coming into the house, expect a MUCH MUCH larger job that it looks like if you do that - you have to start at the loose end of that pipe and back every joint out, one at a time, until you get to the siding.
The typical way of handling that situation with solid siding (clapboards, shakes, etc) is to cut a notch in the siding below the pipe hole, fit that notch around the pipe, and backfill the notch with the cut-out piece and glue/silicone.
If vinyl siding, don't remove a whole notch, just slit the siding below the pipe hole, twist to fit around the pipe, then slip another smallish scrap of siding in from behind (silicone optional) to hold the slit closed. It helps a LOT if the slit isn't dead centered on the pipe hole, but rather at either horizontal edge of the hole.
By all means, leave that pipe right where it is.