Sounds like your sheetrock is somehow damaged from moisture or the EZ Toggle would have certainly worked. You now need to install wood backing, or move your speakers so the brackets can be screwed into the studs.
To install wood backing, first you need to cut out a rectangle of your wall's sheetrock slightly larger than the intended size of the backing panel, with the vertical cuts centered on studs. Don't try to save the old piece, replace it with new. I would use plywood instead of sawn lumber which would make a cold spot. Cut the plywood to fit flush between the studs, preferably with face grain horizontal.
Cut 2 narrow plywood strips to act as cleats. Using construction adhesive, nail/glue the strips to the studs set back from the front edge equal to the plywood thickness. Once the glue has set, nail/glue the plywood backing to the cleats. You are not supposed to nail into the edge of plywood, but in this case it's just to hold the panel in place until the glue sets.
Install a new sheetrock piece to fill the hole you cut out. Tape and mud the joints. Prime paint your finished mud job. If you have textured walls, and don't have a mud gun, there are spray cans of wall texture available for repairs just like this. Either way, you'll need to practice application to match the existing texture. Once dry, apply paint top coat to match existing wall. Whew!
Sure you can't just move the speakers to the studs? ;)