Here is the issue. I can use the toggle bolts, but it looks like drywall has lost strength. It can be seen in the pictures that drywall around the place of the bracket holes is swelled and about to come off.
As you have found out, curtain rod supports do not do well hanging from plaster anchors. They really should be attached to studs, but where that is impossible, toggle anchors give you a chance at success. I prefer the solid bar type that pivots after insertion.
The drywall in the immediate vicinity is beyond saving. Even toggles will not have much solid to support them. You need to move the bracket well away from the damaged area (to a stud if you can), or you need to build up the area to support it.
One way to reinforce the area is to create a wooden backing behind the drywall to serve as a support. Find a piece of wood at least 1 1/2 inches wider than the space between the screws. Cut it to about 6 inches in length.
Cut away the damaged section of drywall making a hole a bit wider than the board and about 3 inches high. You are going to insert the board vertically into this hole and attach it to the back of the drywall.
An easy way to do this is to drill a small hole in the center of the board and tie a string through the hole. You can then insert the board and use the string to prevent it from falling and to manipulate it into place.
Screw through the drywall into the board using drywall screws. Use two screws top and two on the bottom. This locks your new framing piece in.
Make a patch of drywall and fill the hole. Tape and patch with joint compound. Lightly sand and prime. Paint now or later.
Now, when you reattach the bracket, you will be screwing into the wood which is supported by many square inches of drywall. I would predrill the holes with a bit slightly smaller than the screw body to reduce the chance of splitting the wooden backer.
Forget the drywall. Go to a lumber supply and buy a pretty looking 1x4 or 1x6 board of generous length considering the width of the window or whatever. It is best if it is long enough to cover at least two studs, so for narrow windows, it might extended well beyond the window (consider making it full room length).
Route the board's edges (optional), sand (optional), stain (optional), and finish it (for sure) to be water resistant (Tung oil, varathane, etc.).
Take down the rest of the bracketing. Find the studs: drill small (1/16th inch) holes every one inch to find the first stud, then confirm stud spacing in multiples of 16 inch spacing. Screw or nail the board to the studs. Attach brackets to board using standard hardware.