A few notes first
As you probably know, North American power is split-phase - it is 240V with neutral in the middle, giving 2 opposite poles or legs of 120V. So if this has a 50A breaker, that means you have 120V @ 50A on one leg, and 120V @ 50A on the other leg.
So it's 50A worth of 240V... but 100A worth of 120V.
By the way, I assume you have one breaker which is double the normal width of a breaker. It looks a lot like 2 breakers whose handles are tied, but it is in fact one breaker, and it has an internal "common trip" mechanism which is required for this application. So it must be 1 breaker, not 2 sandwiched.
Voltage drop and wire selection
The max you're allowed to plan to use is 80% of breaker trip. So with a 50A breaker you could only plan to use 40A. Your needs are much lower than that, given that, after all, we're talking two legs of 120V @ 40A each. So we'll compute on 40A.
Calculating for 40A @ 240V, using #6 copper wire gives 2.21% voltage drop. #4 aluminum wire gives 2.29% voltage drop. Both are extremely acceptable; even the wire salesman considers these really good numbers.
Honestly, I don't normally even bother doing the calculations until it's farther than 180 feet.
Pro electricians wiring their own houses would use aluminum. There is some scary talk about aluminum out there, but it applied to a mistake in the 1960s trying to downscale power-line alloy aluminum for small circuit use (15-20A) relating to how the wire was landed on lugs (screw terminals and backstabs). Aluminum terminations have to be managed more carefully, but doing that safely is super easy with large feeder like this. Even 1960s-era aluminum feeder is considered safe; the new stuff is awesome.
Anyway, the panel lugs themselves are aluminum, because aluminum lugs are the universal donor - they play well with both copper and aluminum wire. (that gives a hint as to the source of the above problem).
#2 copper would be insane overkill. Do not buy #2 copper. If you already have a 450' coil of #2 copper sitting around, sell it on eBay and use some of the profits to buy all the wire you'd need. Pocket the rest.
The #4 aluminum I recommend is about 1/7 the cost of #2 copper, e.g. the aluminum is $150 US vs $1000+.
If you really want to spend that much on wire, #2 copper is a bad choice because you are splitting the switch. It's far too big for your "few outlets" and far too small to carry the full 200A service from garage to house. If you ever reversed flow so the service came into the garage, you'd have to replace all that #2.
If your aim is to put in "the ultimate" wire once, and never trench again, well we recommend conduit for the "never trench again" part... but that aside, I would lay 250 kcmil aluminum. It's good for 225A service, will handle 200A without worry for voltage drop issues, and it's still much less than #2 copper.