Why doesn't the new bulb fit all the way into the socket?
Likely because the new bulb is just ever so slightly bigger than it's supposed to be and it has a slight taper at the base.
That would explain why, when you push the bulb down, you can force it all the way into the socket, the metal pins make contact and it lights up. It would also explain why, when you release pressure, it moves back out of the socket, breaking electrical contact. Just 0.5mm of movement on its own (0.1mm realistically) would be enough to break electrical contact, but you might not be able to detect this movement with your eye.
It's possible that the socket isn't properly shaped, but, since you have a bulb that will stay in it, that's not likely. However,
NOMA 40W G9 Halogen Bulb that has always fit snugly into my IKEA November. emphasis added
Maybe, just maybe, the socket on the lamp base is just a bit too small. The bulb should be a snug fit, but if you have had to really push the existing, functioning bulb in harder than you think you should, the socket might be the problem.
It could be a combination of a socket that's just too small and the new bulb is just too big and between them, they're just not going to fit together.
I do note that your original halogen bulb is a Sylvania Osram. Those are generally recognized as being pretty high quality, so it's very likely that the bulb was made within the proper specs for the G9 bulb base. It's still possible, though, that the bulb is toward the higher end of the allowable range of base thickness while the socket is at the lower end of the allowable range, thus making a somewhat tighter than expected fit.
Which side of the LED is correct?
I think the answer to this one it pretty obvious, don't you?
The question you probably meant to ask but forgot.
How do I fix this?
Buy a different bulb. It could simply be a different copy of this exact same brand/model of bulb. It's possible that there's just a flaw in this one that slipped past quality control. Or, you could try a different brand. If you do try another one of this brand and it still doesn't work, definitely go to a different brand - this one may may just be off enough on their manufacturing/design tolerance.