A cap goes onto the exterior of a pipe, whereas a plug goes into an existing fitting (or, sometimes, into the pipe itself). There are situations where one or the other is more convenient for the person doing the work, and certainly situations where "I'll just use whatever I have on hand rather than run to the supply store."
There's no general rule that one style (cap or plug) is better than the other. There is a difference in attachment methods though. You've mentioned PVC primer and cement with the cap, and some kind of clamp with the plug.
When rigid PVC is used in pressurized service it is generally joined with solvent cement or threaded fittings, not clamps. This is because the pipe is hard -- it would require too much force for a clamp to securely hold against the pressurized water inside the pipe.
Clamps are used for pressurized service with softer tubing, though. For example indoor PEX and outdoor polyethylene commonly use barbed fittings pushed into the tube with a clamp or crimp band around the outside to secure the fitting.