You shouldn't be using a drill. 

My own what should I used chart:

1. Drilling into something that is delicate or could damage/scuff = **Drill**

Examples - Drywall, cabinets, furniture, toys

2. Drilling into masonry directly = **Hammer Drill**

Examples - Screwing directly into concrete or brick

3. Drilling into something tough that needs a lot of torque = **Impact Driver**

Examples - Hardiboard, brick, framing wood, sheet metal


You are simply using the wrong tool for the job.  Let me tell you the Impact Driver is the bee's knees for any tough job.  It wins out by far and require much less pressure by the user.