As @Ecnerwal noted, there's always going to be a noticeable transition between smooth and orange peel (or any other texture).
Trim or a strategic corner would hide the change.
But if you had to try without the above, then sand down the texture adjacent to the transition. (About 18" would give you a good runway; you might be able to get away with a little less if your sanding hand gets tired.) It wouldn't hurt, actually, to lightly sand the entire wall, as that'll take down high spots that'll wreak havoc with your smoothing efforts.
Smooth with mud, keeping in mind that you can take fresh mud off the textured part easily with a slightly damp sponge as you work on it.
Use topping mud for the top couple of coats so you can sand more easily than if you used all purpose or (heaven forefend!) setting compound (which sands extremely poorly). (Around here, topping mud only comes in big boxes and you'll probably cringe at the waste. Don't. Focus on how much easier the sanding will be.)
Top tip, get a variable speed random orbit sander with a dust port you can put a vacuum on. Run the sander on low. Use the right filter in the vacuum to prevent killing it with drywall dust. Your house will still be full of dust, but slightly less so. The next level up would be enclosing the work area entirely in a plastic bubble, which would really help. (But still do the vacuum.) Wear a good n95 or better mask.