I didn't realize that these were intended for use with drywall (I only found out the name of these things after the fact) and just put them in where we had a stud. Now, it turns out that my holes are not level and I need to adjust one of them slightly. I am planning to fill in the hole with a wooden dowel/carpenter's glue after I get it out, but wasn't sure how to remove the anchor safely to begin with.

How can I remove this without causing damage that can't be repaired easily with the dowel/glue? Should I remove both of them and just redo it without the anchors?

Edit: picture (anchors bottom right)