These gutter guards do go under the bottom layer of shingles. There should be no nails as low as you seem to be indicating with the plastic guard installation attempt.
It's possible that you've slipped the guard between the wood roof deck and the tar paper underlayment and hit a staple holding the tar paper down. If that's the case, then pull it back out and carefully get it above the thin black tar paper, but below the shingle. If you're 100% positive that you've got it between the tar paper and the shingle, then someone has incorrectly installed your roof.
At the bottom edge of the roof, there should be 2 rows of shingles. The first row is installed upside down (with the tabs pointed toward the ridge), then nailed down above the dotted line. This line is the exposure line that helps keep the shingles in line and tells the installer how much of the tab to expose. The next row is installed directly above the upside down row, but with the tabs pointed down toward the gutter. This is the row you see. If you're really hitting a nail at that location, then someone improperly nailed down that shingle row.
Fortunately, this is not a difficult situation to fix, it's just a royal pain to mess with.
- You'll want to work on a dry day to avoid getting water everywhere and because wet stuff (like a ladder) is slippery.
- You'll want to work on a sunny day when the shingles are warm/hot so they're flexible and won't break.
- If you don't see a warm day coming and really want to get this done, I'd suggest a heat gun to warm up the shingles. You don't need them scorching hot, just warm enough to bend easily instead of being stiff. If they're stiff, they'll break when you try to lift them.
- With a wide, flat pry bar, gently lift the bottom of the top course of shingles so you can see the upside down course below it. You should see a nail here.
- Now that you know exactly where the nail is, put the pry bar below the bottom shingle, push it up to the nail, then gently start prying upward while pushing inward.
- You want to be very gentle to get the pry bar under the nail head so you do NOT tear the shingle.
- Once you've got the nail head lifted a little, you should be able to push the shingle down the shank of the nail, leaving the head exposed.
- You can now put the pry bar directly under the nail head and pull it.
- It's not 100% necessary, but it won't hurt to squirt a little bit of roofing cement/tar into the nail hole in the bottom shingle. The kind that comes in a tube that you put in a caulk gun is probably the easiest and neatest to deal with a situation like this. I say "neatest", but let's be honest, there's nothing neat when it comes to using roofing tar. Wear old clothes. Expect you'll get that tar in unexpected places. Know that it will never come out.
- With the nail removed, you should be able to install the gutter guard
- There should not be any reason to attempt to reinstall a nail here. The nails that you cannot see (because they're covered by the tab in the row above) that hold down the visible row of shingles are also through the starter course of shingles and are holding it down as well.
One would hope that this is the only place where a random nail got put in the wrong spot. If you come across others, repeat the process.
If you do tear shingles, ideally they should be replaced. However, you can get away with sealing the tears and "gluing" them down with copious amounts of roofing tar if you really need to. You'll probably be OK for the winter (or spring, if you happen to be in the Southern hemisphere), but you'll want to replace them sooner rather than later. Hire the job out if you don't feel confident, but know that with a little practice, you'll get the hang of it. The more times you try and mess it up, the more practice you'll have!