I agree with ThreePhaseEel and would add a few things:
Make sure your feeder is 4 wires since it is a new installation. Two hots, a neutral, and a ground. The conductors need to be rated for direct burial.
According to Table 310.15(B)(16) of the National Electrical Code #4 Aluminum at 75°C is rated for 65 amps of current. Section 240.4(B) allows you to use the next higher standard size breaker in this case. Since 65 amps does not correspond to a standard size you could then use a 70 amp breaker. Since a 70 amp breaker and a 50 amp will be the same price you could use the 70 amp as long as the sub-panel is rated for 70 amps or more. Usually this would be a 100 amp sub-panel since these only come in certain standard sizes.
Also, as Harper points out in the comments, your first sub-panel should not be fed by multiple breakers. This is a Code violation and safety issue. Whatever the sub-panel is rated for, you should feed it with larger wire rated for that amperage from a breaker sized for that panel. Your conductors need to be sized for the breaker size. Essentially, you should reduce this feed to a single breaker e.g. a single 100 amp breaker feeding a 100 amp sub-panel with 100 amp wire. This would have an added benefit of opening up 4 more slots in your main panel.
Good luck and stay safe!