I am going to put up a compost bay system with 125 x 125mm (5"x5") square posts. The plan is to not use concrete, but backfill the post hole with gravel. What's the best size gravel for the job? 10mm or 20mm?
It does not really matter or make much difference, so whichever is less expensive or more convenient.
The main thing (regardless of size) is to compact it well. Put in 50mm, tamp (or ram) with a flat-ended rod or stick. Put in another 50mm and repeat. Most people get tempted to put in much more at a time, and the compaction suffers as a direct result, making the post less stable.
As was mentioned, compacting the back fill is important. And just as important is the depth of the opening the post will sit in.
You can use any material for back fill (even the soil that was excavated). The reason for using stone (or some other hard water-resistant material) is so water can drain away from the wooden post.
You can view videos of post secured in place with expanding foam!
The reason not to use soil is because it retains moisture that can over time allow microbes to decay the wood.
A good (but sometimes exhausting) rule is to dig a hole a 1/3rd of the length of the post. So an 8 foot post is most stable in a +2.5 foot pit. If you dig the bottom slightly wider than the hole (bell shape) the post is even more stable. The diameter of the shaft/opening should be about twice the diameter of the post. A flat rock at the bottom for the post to sit on will help prevent settling.