When doing work like replacing drywall and trim, I'll pull all the old unused nails from the studs, joists, etc. I do this mainly for two reasons: a) so they don't get in the way of new fasteners when I'm putting up new materials (it's happened), and b) I'm a little OCD about this. But then I have a bunch of holes in everything. Seems most people would just hammer the nails flat to save time. Is there some benefit to have the nails stay in the wood instead of leaving the hole?
This will probably be more of an opinion-based question, but here's my take. I think you hit the nail on the head with your reasoning in the answer.
If you want to avoid running your new fasteners into the heads of the old ones, pull them. If you ever plan to re-use the material or think you will need to saw it, it will also be good to pull nails/screws as to not ruin your tools later on.
If you don't care about the above and only want to get the job done quickly, just pound them in. The caveat with that is that if you're trying to save time, chances are you're not pounding them all in perfectly flat and end up having issues with your finished material not wanting to sit properly, depending on how flexible that is.
The little holes left behind by nails will not have any structural impact, if that's what you're alluding to with that part of your question.