Technically your only option is to replace an impact drill with a pneumatic hammer - the latter drills much faster (so you produce noise for shorter periods) and the noise is different - lower frequencies (IMO less annoying).
Also how you drill matters. If you need to make a series of holes you better make them in long series with short pauses instead of one-by-one with long pauses. The more you concentrate your activity in time the easier it is for others to tolerate it - they can for example go out for that period instead of sitting home listening to noise all day long.
And finally user ChrisF is sooo right about you having to talk to your neighbours - being considerate to other people is very important in such cases. Being neighbours is a mutual relationship after all.