Your assumption is incorrect. The impact is what does the drilling, like a micro-chisel, the. The rotation merely removes the dust, and ensures that the impacts create a circular hole. This is for concrete/brick.
Tile drill-bits are very different, and must not be used with any sort of hammer action (as it would crack the tile).
Drilling through any type of concrete without an impact drill will fail as soon as the bit hits a piece of aggregate. You may get away with it in soft brick/breezeblock, but I wouldn't recommend it.
Note
This was answered before the OP added "manual impacting" to the question