You are experiencing what I call the , "coke bottle effect". When you take a coke bottle out of the refrigerator, it meets that warm room air and starts sweating (poof...condensation). For your situation, it's when the cool air hits the warm metal duct.
To fix this, you can 1) insulate the outside of the duct (so it doesn't warm to room temperature), 2) insulate the inside of the duct so the cool air never meets the warm metal duct, or 3) both...
1) Insulating the exterior of the duct does not guarantee you'll keep the duct isolated from the warm room air...in fact, the duct will probably rise in temperature as the closet rises in temperature.
2) Insulating the interior of the duct (yep, they make ducts like that,) will "protect" the cool air from meeting the metal duct. This is better, but not perfect, depending on the thickness of the insulation, temperature difference, etc.
3) To do both, I'd just change the metal duct to a prefab insulated duct without metal. This will work best in your case.
Or, you could install a metal "condensation drip pan" on the floor of your closet. We use to provide these under air handlers before we learned to install a condensation drain to the exterior. The principle of the "condensation drip pan" is that the moisture will evaporate before the pan overflows. I don't recommend this (and you'll need to careful to fit it around the duct penetrating in your floor), but it is an alternative.