You mention "inline with exhaust pipe" which makes me want to ask if it is an inline fan. That type of fan is installed anywhere along the exhaust path route, rather than being mounted on the bathroom ceiling. Those are the quietest you can get, since they are attached to remote structure, and can be installed some distance from the bathroom.
If you don't have that type, and you have the flexibility to replace the existing fan (such as the existing one not occupying an enormous hole in your bathroom ceiling), switching to an inline bathroom is your quietest option. If you already have that, I would look to changing the structure it is mounted to. It could be attached to your ceiling structure unnecessary, and simply moving it to attach to a collar tie or other portion of the roof structure in the attic would help tremendously. If location is constrained, you'll want to explore isolation mounts.