Graphite is very soft and shouldn't cause any excessive metal on metal wear. It is a very effective dry-film lubricant, just a little messy to work with.
I've used graphite to lubricate all the lock cylinders in my cars, house, etc for years. I recently thought the entry door lock on my motorhome was failing (locking the door with a key from outside became very difficult/impossible), but a few squirts of powdered graphite and a few dozen cycles of locking and unlocking and the problem has gone away!
I've never had a problem with a lock failing due to graphite causing excessive wear and premature part failure. I've had plenty of locks go bad which were never lubricated though :)