Logically, you're correct, but practically, you might run into problems.
For the purposes of this answer, I'm assuming you're only crossing lithium ion batteries from different manufacturers. (Crossing battery chemistries is even more fraught.)
Lithium ion batteries don't like being run down to an extremely low voltage. (This varies subtly by chemistry, so I'll suggest you look up your specifics. You're in the ballpark of 3.2v per cell, which will multiply out to whatever number of cells is in series in the battery.) If you run them "dry", they get damaged internally, and either won't recharge or might (in extreme cases) catch fire. You really don't want lithium ion fires in your life. (Google "Hydrogen fluoride".)
So power tool manufacturers have built a deliberate cutoff circuit in to prevent overdischarge and protect the battery. You'd have to know whether that cutoff circuitry is in the tool or the battery. Crossing manufacturers might leave you without a cutoff circuit. (And I frankly doubt that any cheap adapter has the smarts to have a cutoff, though I could be wrong.)
If you absolutely had to do this, you need to figure out what the cutoff voltage is for your particular battery and be sure that you don't overdischarge your battery. (You need a voltmeter of some description for this.) You also need to use the manufacturer's charger with their battery. (Even if the adapter works on the charger... just don't!)