I've read articles where either one is used to secure shower liner to studs when making a shower bed. Is one better then the other?
I have used nails, I have used screws. Make sure you keep any fastener well above the curb of the shower. I typically run my liner up over 6" above the preslope and nail or screw only at the top edge using a heat gun, not set to high, to help ease the folding in the corners.
One advantage screws have over nails is they are easy to remove to readjust if needed. The downside is the material really puckers bad around the screw head when you draw the screw in tight. As mentioned in a comment, a washer would help this, but then you have the extra thickness to deal with under the cement board.
Nails work well, but if you have to adjust, and you do not leave the nails a little proud to make it easy to pull and reset.... well there's the issue, it can be a real bear to remove without damaging the edge of the liner to a degree.