After some initial sanding of a taped joint, the tape/paper is showing. It's sticking out and frayed, and still needs some sanding. What are my options for dealing with this? should I apply more mud and feather it out further or try to remove it and do it again?
If you are sanding the paper tape, you definitely need more mud. So put on more mud.
Removing it would not really help, it would just make more of a mess you have to mud over.
I suggest 3-4", 6-8" and 12" for a progression in drywall "knives" (a coat with the small one, knock off only the high points, a coat with the medium one, knock off only the high points, a coat with the big one, if there are significant divots when sanding after that (there should not be), stop and recoat to fill the divots.