It's not a matter of safety, but of being able to still drill holes with it.
Normaly, a drill bit is (mostly) symmetric at the tip - it has a small number of "flutes" (the indentations at the sides, which allow material to exit the hole that is being drilled), and they converge to however many small edges or "blades" at the tip.
During drilling, each time one of the edges is in contact with material, a force acts on the edge and thus the whole drill bit. These forces individually try to push the drill bit sideways. Especially for drilling (and to a lesser degree for milling), it is imperative that these forces cancel out. Neither drill bits nor the actual (often hand-held) machines are designed to offer much resistance to a net sideways force.
You know this from the feeling of the drill bit wandering around when you first start the drilling operation: at this point in time, there is nothing opposing these forces as the hole has not been started yet. That's why one often drills a small pilot hole, or smashes the surface with a pointy object to have something for the actual drill bit to bite into.
Milling can and often does function in the presence of these sideways forces because the machines are much more heavy (often ranging in the metric tonnes instead of a few kg), and have very substantial quills which can stomach significant radial forces; also the milling bits are often much heftier. You still want to make the bits and the quill as short as possible to reduce the lever, but still, as opposed to drills, they are designed for this.
To answer your question: it is not a safety issue, but you will find that you probably cannot even start a successful hole with it anymore - it will wander around very badly, scratch up your wall or stone, and just be a total mess. If you use it in a partially drilled hole, which gives guidance to it, you will probably be able to drill with it - it will just chew through the material with brute force, and if it were not a masonry bit but one for wood or metal, the end result would look ugly; but in the case of stone this won't matter really. Also, since the bit will press against the sides of the hole, it will become very hot; take care not to burn yourself when taking the bit out of the machine. But as said, I'd expect that you simply will not be able to achieve much with it either way.
It can and will eventually break completely, but you should be wearing eye protection anyways, as there is always a possibility of little stone pieces flying anywhere, so it's no issue from a safety standpoint as far as I'm concerned.