First, make sure you get the right plank type. Trex (and some other brands) have planks designed specifically for use as decking. These have a fastening system that involves hidden grooves, which won't be hidden if you use it for a top rail. See: http://www.trex.com/products/accessory-hardware/.
There are special screws for composite decking, but they often contain multiple threads designed to have the right thread in the right material when you fasten down through the composite board. So verify the designed usage if you use these.
The planks are heavy and you aren't really trying to secure them against "falling up" or being pulled off. The screws in this application are mainly to keep the planks from sliding. You don't need massive holding strength on each screw, which is good because these composite boards have little structural strength. The material crumbles and breaks easily. Once screwed in, a small amount of prying force will pull the planks off, either stripping the screw hole or leaving hunks of material behind around the screws.
So to your question of what kind of screw to use. Several things are important:
- You need exterior grade screws, either stainless steel or specially coated.
- The composite material doesn't compress like wood; pressure just makes it fall apart. So it isn't good to use normal wood screws, which have a thick, wedge-shaped thread. You need either:
- a thread that is coarse and thin, more like a knife that causes minimal displacement of the material, like this style:
Or use one that taps a thread, like this style:
Both styles are often advertised as not requiring a pilot hole, but I wouldn't trust that. I would still pre-drill a pilot hole. Also, don't screw much more than about half way into the composite plank. The pressure of the displaced material can cause the top surface to protrude if the tip of the screw is too close, or even push out a plug of surface material.