I would make sure the population that knows about the hole is dead and then patch it. Mice are very smart and will teach their "friends" or offspring the maze to get to the peanut butter. You must kill them all.
I would start with poison and glue traps. Break down the pellets into small pieces and leave by the hole. I have found adding chocolate or peanut to them HUGELY increases consumption. I have only found this true for mice.
They will get caught in the glue traps going in and out. Most mice will not stay in your house but will only go in for food and water. If you do fear they are staying, then frequent the room next to the hole and turn on all of the lights all of the time. They will leave.
Once you go 4-5 days with nothing in a glue trap cover the hole. I would fill it with a rough steel wool (they hate it) and then cover it with whatever - that is another question.