It's asbestos, not kryptonite
It's not that toxic, or it would have never attained its status as a popular material in the first place.
Doctors started noticing that laborers who worked every day in asbestos plants or installing asbestos for 20 years... were having serious lung problems, as well as in some cases lung cancer. The process of identifying this was not fast. However once it was proven that these injuries and cancers were coming from asbestos, the companies who employed these laborers set up trust funds to pay legal claims.
Access to the trust funds requires some documentation, but it's easy work for lawyers. There aren't any great Perry Mason moments in these legal actions; the trust funds are already established. Yet, some lawyers will collect 1/3 "contingency fee". (they're paid if you win).
As such, they blast advertisements 24x7 in their desperate search for asbestos workers who might qualify for the trust fund.
The real carcinogen here is advertisements.
The only reason there's any hysteria at all about asbestos is this carpet bombing of ads frothing people up into thinking asbestos is demon dust. The government decided there was no need for the government to produce calm and reasonable Public Service Announcements; community education "can be left to the private sector".
Anyway, that's why you are gripped with fear. The problem isn't the asbestos, it's the level of inherent faith you place in commercial advertising, without even realizing it. It is simply impossible for reason to be heard against the onslaught of adverts.
The fact is, incidental exposure (a day or two in a lifetime) to material that probably doesn't even have a high concentration of asbestos, is such a low risk as to be effectively zero. Ordinary masks are sufficient to manage that risk.
Meanwhile you have the government trying to communicate to employers (read: cheap and old fashioned) and off-the-curb, never-see-em-again day-labor employees (read: non-English speakers) how to protect themselves from daily, career-long occupational exposure to asbestos, so we don't get any new cases. Their advice is to use tip-top gear, because that adds up when you do it everyday for years. In that context, true, you want better than N95, but the way they communicate that to cheap employers is "N95 masks don't work".
Which is not true at all, in your incidental case. You are much better off having used that mask, mainly due to other risks.
So if you want to be part of the mesothelioma moral panic, just keep in mind negative thoughts are far more carcinogenic than whatever got through that mask.
One other thing: Gypsum is fireproof. Drywall is gypsum and drywall is used for firestops.