Definitely separate out that neutral and ground wire.
Don't bother replacing the aluminum wires
Yeah I know there's some ooga-booga out there about aluminum, but that has nothing to do with your heavy feeder, which is fine stuff and you should keep it. The problem related to very small wires in 15/20A branch circuits, specifically use of outlets and switches whose lugs were rated copper-only, and under political pressure, UL hastily cross-rated them for aluminum without proper testing. Aluminum lugs are the universal donor, they cheerfully take Al or Cu wire. Hence many panels use Al lugs, correctly and without difficulty.
So all you need for the care and feeding of Al feeder is to keep it clean and use the anti-ox goop. If you're splicing that somewhere, use insulated Polaris lug splices just big enough for the larger wire; much nicer than using bare splices and wrapping them with globs of electrical tape. The Polaris bars are made of aluminum.
Your #4 Al wire is capable of being fed from a 60A supply breaker. That is also limited by other segments of cable/wire, as well as the subpanel's bus ampacity if it doesn't have a main breaker of its own.
An outbuilding requires a disconnect switch; 95% of the time you get that by choosing a panel with a main breaker. That breaker size does not matter for its role as a disconnect switch. You can never count on the convenience of the local "main" tripping first, however, with a 40A "local main" and a 60A feed, it might just work!
Anyway, I'd hold onto that nice ampacity of the aluminum. I know you've got your plans figured out, but honestly, the world is too full of surprises to ever say that for sure. Whoever guessed they'd work the bugs out of electric cars, for instance.
Retrofitting ground
So yeah. Just pull a #10 Cu ground wire through the conduit, add a separate ground bar to the panel, pull any N-G bonds, and you're all set.
It's a shame you didn't pull the ground wire in (or at least a pulling string) when you pulled the phone/intercom out. Of course the phone and intercom were completely illegal; you can't put signal wires in the same pipe as mains power.
I wouldn't grab the Ufer ground, but I'd grab the copper wire going right to it.
Generally you can take a retrofit ground back to any of these:
- The panel that serves the load
- Metal conduit that goes back to that panel, if the metal conduit is rated for carrying grounds (all non-flexible and some flexible conduit is).
- Any junction box visited by a branch circuit coming out of that panel, which has a sufficiently large ground wire (any >=30A circuit must have a #10 ground wire, so that would suffice).
- Anywhere along the Grounding Electrode System; i.e. the run from the service panel to the grounding rods/pipe/Ufer.
