Get the installation white papers for the kind of vinyl flooring you're using and follow the instructions to the letter. They should tell you exactly what kind of underlay to use on top of the subfloor if any. For instance vinyl planking without a sticky side (not what you're getting by the look of it) needs no underlay, sealant or subfloor to function correctly as per the installation instructions.
As for sealing the wall surrounding the flooring, I assume that since you had carpet there this is a dry area of the bathroom and not required to be in direct contact with water. You don't need sealant around the wall as the drywall in that area should be greenback/mold resistant anyway. You can just trim, paint and caulk.
Additionally, you're going to need expansion gaps (~3/16") on all sides where the flooring meets a hard vertical plane in order to prevent buckling due to material expansion. This gap will usually be covered by trim or quarter round which you will caulk the portion that meets the flooring to prevent the spread of any water you may get on the ground from getting out of the shower and whatnot.