Some circuitry in my old house is just two wire w/o ground and as I'm renovating I replace these outlets with GFCI if I'm not pulling two wire w/ground. I've found such an outlet that shows HOT/NEU REVERSE either way I attach the hot and neutral wire to the GFCI outlet. What is going on here?
First, the lack of a ground wire does not imply lack of grounding. Older construction often brings ground via the armored cable or metal conduit to metal boxes, and receptacles and switches can pick up ground via their mounting screws. Receptacles do not do this reliably unless they are marked "self-grounding".
You haven't mentioned this, but you are using a 3-light tester both for GFCI testing and also to detect wiring problems. Those 3-light testers are extremely unreliable because the labeling is written for wiring mistakes in new construction. The labeling is completely wrong for troubleshooting wiring failures in something that was properly wired.
On a grounded receptacle: The tester should indicate 2 yellow lights. When the "TEST" button is pressed, it should immediately trip the GFCI and the tester should indicate all dark. (because the GFCI tripped).
On an ungrounded receptacle: The tester should indicate 1 yellow light (middle). While the "Test" button is pressed, the GFCI should not trip. The light indications become meaningless, you might expect center yellow and red while you're holding the button down, but that is a testing artifact.
Always hook things up correctly. When "wiring correctly" and "optimizing for favorable messages on a tester" conflict with each other, always wire correctly.
When installing GFCIs, put all wires on "Line" (read the instructions for how each screw takes 2 wires). Do not use the "Load" terminals unless you understand how downline protection works and want to use it, note instruction 8(C) re: installation of "GFCI Protected" stickers. Using "Load" and multiple GFCIs results in redundant protection and great difficulty resetting. Here's how to untangle that when you do it by mistake.
You MUST use "No Grounding Conductor" stickers on all GFCI-protected outlets without true grounds. If you don't like the look of the provided stickers, feel free to make them on a Brother or P-touch label maker: any method is allowed except hand-written.