It's asbestos, not kryptonite
It's not that toxic, or it would have never attained its status as a popular material in the first place.
Doctors started noticing that laborers who worked every day in asbestos plants or installing asbestos for 20 years... were having serious lung problems, as well as in some cases lung cancer. The process of identifying this was not fast. However once it was proven that these injuries and cancers were coming from asbestos, the companies who employed these laborers set up trust funds to pay legal claims.
Access to the trust funds requires some documentation, but it's easy work for lawyers. There aren't any great Perry Mason moments in these legal actions; the trust funds are already established. Yet, some lawyers will collect 1/3 "contingency fee". (they're paid if you win).
As such, they blast advertisements 24x7 in their desperate search for asbestos workers who might qualify for the trust fund.
The real carcinogen here is advertisements.
The only reason there's any hysteria at all about asbestos is this carpet bombing of ads frothing people up into thinking asbestos is demon dust. The government decided there was no need for the government to produce calm and reasonable Public Service Announcements; community education "can be left to the private sector".
Great.
Anyway, that's why you are gripped with fear. The problem isn't the asbestos, it's the level of inherent faith you place in commercial advertising, without even realizing it. It is simply impossible for reason to be heard against the onslaught of adverts.
The fact is, incidental exposure (a day or two in a lifetime) to material that probably doesn't even have a high concentration of asbestos, is such a low risk as to be effectively zero. Ordinary masks are sufficient to manage that risk.
Meanwhile you have the government trying to communicate to employers (read: cheap and old fashioned) and off-the-curb, never-see-em-again day-labor employees (read: non-English speakers) how to protect themselves from daily, career-long occupational exposure to asbestos, so we don't get any new cases. Their advice is to use tip-top gear, because that adds up when you do it everyday for years. In that context, true, you want better than N95, but the way they communicate that to cheap employers is "N95 masks don't work".
Which is not true at all, in your incidental case. You are much better off having used that mask, mainly due to other risks.
So if you want to be part of the mesothelioma moral panic, just keep in mind negative thoughts are far more carcinogenic than whatever got through that mask.
One other thing: Gypsum is fireproof. Drywall is gypsum and drywall is used for firestops.
I think you can read between the lines here. i had on a typical n95 mask while doing a little demo. I didn't have raised suspicions about there being asbestos in the walls, as i already researched my home's primary wall material prior and it's not prone to the stuff. But, after removing a few SF of board i noticed a different breed, and the label is "national gypsum company" "fireproof GYPSUM W..." label cuts off. It's the fireproof part that raised my red flag. I bagged what i took off, cleaned the area, and am sending a sample off to a lab for testing. 
No level of asbestos exposure is considered safe. However, most problems arise after years of repeated and long-term exposure to the carcinogen, so even if you were exposed its similar to eating a raw egg, you probably won't get sick but you might