So the banister holders are in plaster, and it's pretty crumbly. We removed the wooden rail, but the anchors are in no way attached to the wall soundly. How can I make this secure again?
Move the anchors to a stud and attach them with long screws. Use a stud finder (preferably with a deep scan capability since you mentioned plaster) to locate your studs. Or you can drill some small test holes since you already need to patch the wall.
Don't use drywall screws, you need something with more shear strength. I wouldn't use anything less than a 2" screw, particularly in the top two holes. You need it to have a good bite into the stud.
Then you just have to patch and paint the old location.
Another option is to run a board up the wall in line where the mounts go. The board can be attached to studs and then the rail mounts to the board. This allows the mounts to be slightly off of the studs. This is not as ideal as hitting the stud. The board can be routered and varnished.