My steel beams are 4-1/4” wide. If I run my framing under the beam with 2x4 framing, I’ll end up with a 3/4 overhang on one side of the wall. How do I make up this gap? Just use furring strips or is there another method?
Simply use 3/4" furring strips. There is typically a wood plate above the beam which may need furring as well, that will allow the top edge to be nailed or screwed securely.
User3757614 is correct that it was an intentional dimension, though I wouldn't use anything less than 1/2" drywall for anything but a strip here and there.
Alternatively and if you want the beam in the wall so you can run up to the ceiling, then you should use 2x8's or larger to stud your wall and notch-out for the beam wrapping. This, will still give you something to screw into and be solid.
A final option is to do a double stud wall or a wall on each side of the beam. But, it's twice the work and typically only needed if you're soundproofing a theater room or loud Man-Cave.