The key criteria for me is the size of the holes. If you used brads, then I have no problem with using caulk, especially on baseboards. The only possible downside is if you leave a fingerprint in it. You have to have eagle eyes to pick out a small hole like that.
With crown molding though, I'd plan on a lightweight spackle. Mostly because of the cost and effort you already put in installing it, might as well finish it nicely. Vinyl spackle works great, but again if the holes are small, just about anything made by DAP works fine. If you find yourself with a large hole, bondo (auto filler) does amazing things. Or you can use painter's putty, glazing, just about anything can be made to look really good. Sand and touch-up as needed.