My kitchen floor has old linoleum. I want to put down sticky tiles over the old linoleum. The linoleum has a few holes. Is there something that I can use to caulk the holes before putting down the sticky tiles?
For small holes I just use spackle. Mix it up and use a putty knife working any air bubbles out. You may need a second application if the holes are deep.
If the holes are larger, after I have a nice even fill and when dry I will use a paint primer so the tiles will stick better, this is a good idea with self leveling compounds also so the tiles will stick and not have the edges roll up.
You can use what's called "underlayment leveling compound". There are generally two varieties, regular, which you trowel on and level yourself, and "self leveling" that is more liquid, which you pour on and let set, it naturally seeks level. The problem with the self leveling in this case will be the edges of the holes in the old linoleum may end up as ridges that show through your new floor. I would use the old fashioned type and level it myself.
Side note: some old flooring contained asbestos and if you knowingly cover it up, in some places you can be prosecuted long after you have sold the house when/if it is discovered. You can send a piece of it off to be tested, but I have just opted for full removal of old linoleum. It's hard work, but worth it in the long run (to me anyway).
The most universal answer for how to fix holes before tiling would be to mix up and use a small batch of the thinset you will be laying the tiles into and scrape it level with your existing floor as you would joint compound / spackle on a wall. A subsequent skimcoat with your thinset mortar can help you level out any other imperfections.
That said, as Ed eluded to in his answer, you can pretty much fill the hole with anything (mortar, cement, sparkle, painters putty) as long as it will stay put when you go to 'key in' and 'comb' your thin-set mortar.
However, it's best to try to match the consistency of the material you are repairing to avoid cracks in your grout lines. Thus for linoleum an electrostatic compound like concrete patch repair or painters caulking / putty would be ideal.