There are actually two problems, and they are related:
Low Power Fan
First I found a customer review on the Lowes web site that said:
I gave this only three stars as no where on packaging or instructions is there any note indicating this will not work with high efficiency bathroom fans. I spent many hours troubleshooting my Broan XB110L only to find during internet searches that this timer causes erratic fan operation. Actually in my experience this timer results in a fan that never reaches full speed and labors erratically at very low speeds.
which matches your problem pretty closely. And then I found this comment from a Home Depot customer:
Note: This timer requires a load of at least 40 watts to operate properly.
and that matches your problem exactly. Your fan is rated at 30W. Not clear if that is Fan + Light or just Fan. But even if it is just the fan, the LEDs are 13W so that would be a total of 43W - and likely less because the fan motor is probably listed at the high end of its usage (e.g., startup).
which led me to the definitive answer in the installation instructions:
40 Watt Minimum
- Do not use where total load current is less than 0.3 A 40 W or greater than 5 A
which confirms the problem.
Not LED Compatible
I was at first a bit surprised at this, as the device seems quite up-to-date and even uses LEDs in its own design. But both Lowes and Home Depot list it as not LED compatible. Since your fan has LED lights, that's a no-go right there.
Why is there a minimum load? There are three ways to power a smart-switch/timer/motion-detector/etc.:
Neutral requires a neutral wire. Many older homes don't have neutrals in the switch box, so it is a marketing advantage to have a product that doesn't require neutral - fewer returns due to "can't install it here". This timer does not use neutral.
Ground is a general "no, you can't do that" but it is allowed when properly designed but has a limited amount of current allowed for the "smarts" of the device. This timer does not use ground.
- Leak through switched hot
This is the only option left! Works great with incandescent & halogen but does not work well with LEDs. And I suspect they have some tricks that just don't work well unless there is a large load so that the timer's own usage is "small" relative to the fan/light load.
Bottom line: SEND IT BACK
Recommendations
Look for a timer that says:
- LED Compatible - That indicates it doesn't "leak current"
- Requires Neutral - That tells you how it gets the necessary current