I had similar situation trying to install curtain rod hooks on the wall with thick sheet rock, hollow space and concrete block.
For the screw length issue - The guy at home depot matched the screw from the toggle bolt package by thread count to the sheet metal screws and picked the right length. This gives more choices.
My case was bit worse. I didn't have any screw length for which the toggle will open. The wing length was simply longer the the hollow space. I am posting couple of things I tried just in case.
For half the hooks, I took a sharp chisel and took out bit of inside edge of the sheet rock (through the hole drilled for the toggle bolt) up and down. I took out enough so the toggle will open if inserted vertically. With much difficulty I twisted it around to be horizontal where the sheet rock edge was in fairly decent shape and tightened it.
Since this was getting messier, I tried to cut the toggle wings bit shorter with a snip so it would not get stuck in the sheet rock. Again, after sacrificing few bolts, I had my own custom toggle bolt. It worked - I was able to install rest of the hooks this way.
Not sure how much load capacity I compromised cutting the wings off. Hope it doesn't fall on my head. Not an ideal solution - nonetheless, something to think about if your shelf is not going to be holding all volumes of encyclopedia britannica.