I'd say this isn't likely. Your bathroom is an exhaust, and your dryer is an exhaust -- running one to the other wouldn't make sense.
Start by looking around the exterior of your townhouse, between 1-3 feet off the ground. Do you see a dryer exhaust like one of these?
Note that you'd only see the square louvered panel or the triangular-ish hood, not the cylindrical ducts behind them. The triangular-ish one will have a damper flap up inside it.
If you see one of these, then turn on your bathroom exhaust fan and go back outside. Are the louvers or damper flap open, and/or do you hear sound coming from it? If not, good. Now go start your dryer (with clothes in it, so it doesn't overheat) and go back outside. The louvers or damper flap should be open at least partially, and you should hear the characteristic dryer air sound.
Assuming you only noticed the vent in operation when either the bathroom fan or the dryer were running, then the two aren't interconnected and the lint you're seeing is normal bathroom lint (toilet paper, bath towels, bath mats, etc) collecting on the vent cover (and inside it). This happens because when you shower/bathe the room gets really humid, and some of that humidity condenses around the vent as it's getting sucked out... then as the lint is getting sucked out, it gets trapped in the condensed moisture and any other lint that has already collected there.