As long as the floor is sound solid, in other words no sign of wetness or dry rot etc., yes you can.
From what I read your concern is not to get water in that space where the linoleum meets the tub.
Before caulking it, it won't hurt to put some sort of cement such as Liquid Nails™ or what have you first. Somehow get it under the lip of the linoleum to help secure it to the floor.
Let the cement sit for few a minutes so it get nice and tacky, it helps the linoleum adhere without rising up again.
This will also help a lot with caulking it. It will look better and work better and makes everything go smoothly for you.