I am installing some some floating shelves on drywall using a drill. What's the best way to ensure the shelves are a sturdy foundation that can support stacks of heavy books? Should I use a stud finder and attach the shelf bracket directly to a wall stud? Or will the shelves be sturdier if I use wall anchors? If so, to determine the appropriate wall anchors, should I take into account the weight of the shelves and the contents I intend to store on the shelves? Finally, are butterfly or toggle anchors preferred for this project?
Attach the shelf brackets directly into the studs, no ifs, ands, or buts. Wall anchors are OK for light loads, but I wouldn't use exclusively them for a floating shelf, even if you weren't putting books on it. I would use at least 2" #8 screws, so you have at least 1.25" going into the actual stud. If your shelf bracket is rather thick, go with 2.5" screws.
If your shelf is short, or you want it positioned such that you'll only hit one stud, you should use wall anchors on the ends in addition to screwing into that one stud. The wall anchors in that case would be more to keep the shelf from rotating. Toggle anchors are certainly appropriate for this case, but the plastic anchors would be OK as well, since they're not intending to support much weight in this scenario.
I like a magnetic stud finder such as the StudBuddy, as it's dead simple to use.