What is the real threat?
Asbestos comes in several forms. The one we're concerned with is a puffy/wooly fine-fiber version similar to today's fiberglass matt or rockwool; it's job is to prevent air circulation so the air can be used as an insulator. The air insulates, not the puffy mat material.
Asbestos is also made in "hard" form as a pressed/formed/molded panel. This is of little concern as long as loose fibers are not being created. This is often seen on the sides of houses.
Really, asbestos was the "fiberglass" of that time; since it occurred naturally in fiber form, it didn't need to be manufactured.
Decades ago, doctors started noticing a pattern of a certain cancer among older workers in certain industrial crafts: working in asbestos manufacturing plants, fitting pipe (or to be more precise, pipe insulation), ship wrecking (lots of insulated pipes on a ship), boilermakers (ditto) etc. These people worked with asbestos every day, as their day job, for decades of their life.
I'm not an oncologist, but I'll tell you how the mechanism starts: it starts with fine mineral fibers causing inflammation. Inflammation is the body's normal way of rejecting threats, and it's quite good at healing that. However when a body is subject to constant, daily inflammation for an extended period, constantly healing and rebuilding, the immune system is under a great deal of strain. So it has trouble doing its job at peak performance.
The takeaway from this is that any mineral fiber is going to have a similar effect - asbestos, rockwool, fiberglass, etc. What made asbestos so toxic was the cavalier attitude toward PPE that people had in the early-mid 20th century, which allowed doctors to notice patterns of illness. That won't happen with rockwool or fiberglass because they are newer products, and large companies won't let people work without PPE, so the illnesses won't happen there. Patterns with independent contractors are much harder to see.
I say this to point out that fiberglass and rockwool are not "magically safer" than asbestos -- but conversely, and to your point, asbestos is not significantly more dangerous than rockwool or fiberglass. Again, a casual "drive by" encounter is simply not going to have a health issue. There probably weren't any airborne microfibers, and even if there were, the inflammation occurred once, your immune system dealt with it once, and your body went back to normal. There was not the repeated, daily, constant inflammation and daily stress on the immune system that asbestos workers faced.
Why there is such mad hysteria about asbestos?
Many of the companies that made asbestos things, set up trust funds to support people who got cancer from asbestos.
The word for that, mesothelioma, is the most expensive advertising keyword in the world.
Why? Because these trust funds are extremely well-funded, and "ripe for the picking". Unfortunately, this is not a single government program you can fill out a few forms and collect. Each fund only wants to pay for liability caused by its origin company, and if someone worked with a variety of companies' products, well, it gets complicated. As such, the lawyers are involved. But it's easy pickings for lawyers, since the only matter of dispute is each fund's percentage of liability.
Contingency-fee lawyers take 1/3 of a settlement, typically. You can see where this is a gold mine for them! It's crazy money for work that is "shooting fish in a barrel".
Thus, lawyers have run one of the most aggressive advertising campaigns in history, to make sure "no stone is unturned". That's why "mesothelioma" is the most bidded word on the Internet.
Obviously, average folks are highly influenced by advertising. I bet you think WD-40 is a pretty good product :) And so this barrage of "Asbestos is the worst thing ever! If you ever worked with asbestos, CALL US!" advertisements, has led the public to believe asbestos is plutonium or something.