During heavy downpours, especially ones when windy, water would begin to trickle and drip from the outside surface of the exhaust flue in the basement. Not sure how long this has been going on since the house was just purchased recently. Should I call an HVAC or a roofing pro to take a look? Initial Googling suggests a bad rain cap, but I'd prefer someone with the correct experience to make sure it's nothing else. Also, would the white coating be an indication that this has happened before? The home inspector never mentioned anything.
As Alaska Man said, call a roofer. That b-vent likely exits the roof through a 4" flashing collar and resealing it or replacing it should not be an expensive roofing repair unless the roof is in bad shape.
This is a common problem in windy areas with rain. In many cases the leak is rain getting inside the pipe and running down. Based on your photo I would say this has been happening for a long time. I would want to call a general contractor that also dose roofing because the roofing only guys may not be the best at repairing the inside damage. The clean out looks like it is rusting through , water has probably been sitting in there a long time.