You have a multi-wire branch circuit!
FWIW the red hot leg on a completely separate GCFI breaker acts exactly the same way.
This. This right here.
You have learned a trick that you can get two "circuits" out of a /3 cable if you share the neutral. The problem is, you only learned part of the story.
The full story is that the "circuits" are in fact one circuit called a Multi-Wire Branch Circuit (MWBC). This concept only works if the two sub-circuits are on opposite "poles" - i.e. 240V across them. If so, the neutral is safe; it will only handle differential current. If they're on the same pole, the neutral will be overloaded - and neutrals don't have breakers!
Further, they must have a common shut-off - one finger movement must shut off both halves of the circuit. The shut-off must use a 2-pole breaker or two breakers tied with a factory approved handle-tie - both methods also enforce the "opposite poles" rule (except on GE panels where it's possible to handle-tie breakers on the same pole - not good!)
MWBCs also have very special rules when dealing with GFCI. When either one is involved, you are forced into using the "2-pole breaker" option rather than two handle-tied breakers. No other possible breaker configuration can possibly work. No way, no how.
The only other possible way to protect MWBCs is to use a plain 2-pole breaker, then individual GFCI receptacles at each recep location. That will work, certainly.
GFCI breakers need the neutral wire. No exceptions.
10-3 cable from panel to the outside junction box. From the junction box it's solid 12 awg to the actual plug (non-GFCI). I removed the 30amp breaker in the panel and replaced with a 20amp GFCI.
White neutral is connected to the panel neutral bus (verified not ground).
The black wire from the 10-3 is connected to breaker.
Nope, will not work. GFCIs work by comparing a) current passing through on hot vs b) current passing through on neutral: They must be equal. If the neutral current doesn't go through the GFCI, then that just can't work!!! This is basic physics.
It has to wire like this. Or it can't work.
Turning on the breaker everything is fine, hit the test button on the breaker and it trips like it should. Reset the breaker and walk outside and plug in my tester, 2 lights showing correct. Press the test button and circuit does not trip but plugging in even a 60 watt lamp causes this circuit to trip.
Correct. That is exactly the symptom you will experience if you bypass neutral around a GFCI breaker. The GFCI tester draws too little current for the GFCI to notice the imbalance (except when you're pushing the button, obviously).
None of my other GFCI breakers are wired this way, and they work
That's literally impossible. Either they aren't wired like you think, or, they aren't GFCI breakers. AFCI is nothing like GFCI, and some AFCI breakers (notably GE's) don't need neutral to go through the breaker. So if the wiring is as you say, somebody gave you AFCI breakers. They won't cut it for GFCI protection on bathrooms or kitchens.
I pulled the 12 awg cable today through conduit that my landscaper left so I know it's good.
Remember if the cable is outside, it can't be NM-B cable (yellow). If the cable is UF-B, it is very wide and flat, and requires large conduit. (its inside diameter must be 138% of the cable's widest dimension). Generally cable in conduit is a weak choice, if you're going to do conduit you might as well use individual wires. At that point you just add another gray wire and you have 2 full circuits no MWBC.
