You shouldn't be using a drill.
My own what should I useduse chart:
- Drilling into something that is delicate or could damage/scuff = Drill
Examples - Drywall, cabinets, furniture, toys
- Drilling into masonry directly = Hammer Drill
Examples - Screwing directly into concrete or brick
- Drilling into something tough that needs a lot of torque = Impact Driver
Examples - Hardiboard, brick, framing wood, sheet metal
You are simply using the wrong tool for the job. Let me tell you the Impact Driver is the bee's knees for any tough job. It wins out by far and require much less pressure by the user.