Screws for shelves are loaded differently depending on the mechanical arrangement of the bracketing.
If you have an L bracket (or shelf strip), the top screw of each bracket is loaded by pulling, and the bottom screws are loaded by a combination of press force and downward moment (a.k.a downward torque or moment force).
A floating shelf, that is without an L bracket but with cleats or rods, places a high moment load and pull force on the fastener, somewhat abetted by the thickness of the shelf. Fastening to a stud would be highly recommended to permit loading with anything more than the shelf itself and some light nick-nacks.
To mount such a shelf on lath & plaster you need to make sure that the shelf is thick enough and that the screws can withstand pull out from relatively brittle boards or underlying lath.
Toggle bolts are designed to withstand such pull out force.
If you hit lath, the toggle may not be necessary, and a simple screw might be fine. This would depend on the type and thickness of wood used for the lath.

If the shelf comes with its own custom screws, you could use expanding drywall anchors.

The more you load the shelf the more the plaster will be compressed at the bottom of the shelf. So the compression strength of your plaster will matter too.
If your sense of aesthetics and DIY skills allow, you could additionally support the back of the shelf with a strip of wood, perhaps 1/2in to 3/4in thick, as wide as the shelf but in height about 2x or 3x the thickness of the shelf, with the top of the strip aligned with the top of the shelf (and the rest thus below the shelf). This would reduce the gypsum or plaster compression and reduce the moment load.
In terms of static mechanical loads, glueing it to the shelf would be recommended to help reduce the moment load.