Big problem: no ground
NEC 2014 gives you broad latitude to share grounds among circuits. The one thing it does not allow you to do is share grounds among panels. Circuits served from the main house cannot use the grounding system from the subpanel, or vice versa.
Sharing neutral is out of the question
What has never been allowed is sharing a neutral among different circuits. This causes a lot of problems. It breaks GFCIs and AFCIs. It causes potential overloads, since neutrals do not have overcurrent protection (breakers) and depend on being mated with only one single hot which does have appropriate overcurrent protection.
What's more, neutrals have to be white. No tagging a colored wire. So if your three 12AWG wires do not have a white among them, you'd need to pull one - and might as well pull a ground too.
Of course... If the circuit was 240V, none of the wires would be neutral, would they?
Relays?
Given 3 colored wires, no neutral or ground, here's what makes sense to me. Power both circuits (the 3-way and the simple switched circuit) from the subpanel at the barn. Use the 3 wires to control the coils of two relays. I think if you put the relays immediately off the subpanel, e.g. in knockouts on the subpanel itself, you would effectively be inside the steel shielding of the subpanel and dodge the the grounding issue. It's a stretch, but it's all you got short of pulling more wires.
At the house, you bring 120 or 240V to your two switches. Plain 1-way switches. You send two switched wires and a common to the barn. Each relay coil takes 1 switched wire and shares the common.
Then you feed power off a breaker in the barn subpanel, which goes to the relay contact marked common
on each relay. The relay to control the simple load is a plain SPST relay (any other kind would also do) and is wired like a plain switch. The common
contact takes always-hot from the barn subpanel, the NO
contact is switched-hot for that load.
The other relay is a SPDT type. Note that this is exactly the same layout as a 3-way switch, and you do the standard 3-way switch layout. You wire always-hot to the common
relay contact. Each of the NO
and NC
contacts go to the two messengers, which go to the barn 3-way switch. The common
of the barn 3-way switch goes to the light. Standard 3-way switch layout, except one switch is a relay.
That's how I would do it with the wires present.