You have two separate questions to think about.
Practical load -> Voltage Drop
How much do you actually intend to load up this panel? (remembering that 120V loads only accrue to one leg or the other).
Calculate voltage drop based on that, and use 5%+ for your voltage drop number, because otherwise the calculator might find the standard wire is 3.06% and make you bump a size over 0.06% - that's ridiculous. If the number is pushing 5%, then change it to 4% and see what it says.
For instance let's say you have a "30A" 240V compressor that draws 23A, a "20A" 240V table saw that actually pulls 12A, and a 12A/120V dust collector. The two 240V loads put 23+12A=35A on both legs, and the 12A dust collector adds 12A to one leg so your two legs are 35A and 47A.
Take the average of that - 41A - and put that into the voltage drop calculator. I get 1.95% at 4 AWG Aluminum wire, as the minimum needed to carry practical current in this example. Hold that thought.
Breaker Trip Ampacity
Now there are two breakers involved. One is at the main service panel, and that has a very important job - protecting the wire run to the subpanel, and protecting the subpanel itself if the subpanel doesn't have its own main breaker.
Now, at the subpanel in your outbuilding, there is also a main breaker in the subpanel. Its size doesn't matter -- it's not even required! What is required is a shutoff switch, and the cheapest way to get a shutoff switch in a subpanel is to use a panel that features a main breaker. This main breaker size does not matter.
However, size matters. The size of your panel in terms of number of spaces is very important to not running out of spaces. That creates an expensive, high-labor problem when you add things in the future, like a 240V table saw, a better welder, a water heater, electric car charger etc. Whereas avoiding this problem (with a big panel) will set you back a few latté's.
Some panels specify x spaces and y circuits (a larger number). That's a marketing lie. It depends on you being in a commercial application where AFCI and GFCI breakers are not required. The only number to believe is spaces. A "plenty of spaces" subpanel will probably have a main breaker bigger than 100A. Remember, that breaker doesn't matter.
What matters is the size of the feeder breaker inside the main panel. You want to size that large enough that it won't cause nuisance tripping from peak loads. In fact, it should be at least 125% of the maximum routine load you ordinarily expect.
In the example above, the higher leg was 47A... 125% of that is 59A, so 60A would be bare minimum there, but even that overlooks lighting and other routine potential loads. Fire up a heater and those loads get big fast.
You said "100A" off the cuff, and that's a good, cautious, safe choice (although a bit more expensive in terms of wire). Ed Beal suggests 70A... maybe. Anyway, once you know the feed breaker you want, that dictates the wire size to use:
- 40A: #8 Cu or #6 Al
- 60A: #6 Cu or #4 Al
- 70A: #4 Cu (rather pricey wire, aluminum is 1/3 the cost at this point)
- 80A: #2 Al
- 100A: #1 Al
- 125A: 1/0 Al
- 150A: 2/0 Al
Again this is dictated by the breaker size.
But wait. Remember the voltage drop calc we did above, where we figured wire size based on voltage drop for our practical load, and we got a wire size out of that too? If that conflicts, use the size that is larger.
For instance in part 1, we arrived at #6Cu/#4Al. But we decide to breaker at 70 or 75 amps for some future expansion room. After pricing #4Cu and #2Al, we decide to go #2Al and 75A. However, we also price #1Al, and it turns out to be not much more expensive, so we decide to bump to #1Al and 100A breaker.