I have to use some backing in order to caulk between tile walls and a shower base where the gap to be filled is 5/8 inch wide and 1 inch deep. Is polyethylene backer rod appropriate for use in a shower application? What type backer rod works best with silicone calk (open cell or closed cell)?
A 5/8 inch gap is an awfully wide one to try to bridge with silicone caulk. Even with some type of filler I think you are unlikely to get a result that you would be happy with. At least I know I would not be happy.
Silicone caulk needs to be applied all at once to get a nice installation. First off it will not be reasonably possible to squirt out a 5/8 inch bead from a typical caulking tube spigot. Next consider that silicone material applied as a secondary application will not bond and adhere to previously applied bead that has cured.
If I had your issue I would be investigating filling up a majority of the gap area with a tile setting cement that is compatible with the backer materials behind your existing tile installation. I would then cut narrow pieces of tile (you could even band with a contrasting color) that you would set into additional tile setting cement such that it was flush with the existing tile surface. Once these strips were in place grout the spaces between the tiles and strips. Finally finish off with a neat narrow bead of the silicone material in the corner around the tub surface.
As Michael Karas said 5/8 is a large gap to be dealing with, and no matter what you do now it is going to look "hack-ish" unless you re-do something.
However as far as backer-rod is concerned you can use back-rod in a shower.
As for what type, I'd go with closed-cell for less possibility of water absorption.
For backer rod size you generally want 1/8" larger than the joint size, so 3/4" backer rod in this case.