I have a PVC P-trap under a sink and the trap nut is a very low profile, flush plastic nut that appears to be octagonal. The nut is only about 1/8" thick or less and 1" across so there is very little area for a wrench to grip onto. To make it even worse, the corners of the nut are not sharp, but are rounded off.
Is there some kind of special tool that is used for this kind of nut?
A channel-lock plier is the best tool for about any PVC nut. It's much quicker and more reliable than an adjustable wrench due to the clamping force you're able to apply and the gripping teeth.
I would opt for the channel locks for any kind of gland nut grasping as well, but if the ring nut is low profile and sunk, well first I'd just free up any other joints along the drain by loosening them or removing them where I could, as the P trap often comes apart in two pieces, and then I'd just try and GRASP the upper stem with one hand and the lower stem of the P trap with my other hand, and PULL the socket apart along the joint's longitudinal axis, and give that a go, adding a little twisting motion, you may be able to loosen the threads by twisting the joint a little bit, twisting back and forth, like wringing out a washcloth, and you can bend the pieces a little bit as you pull them apart and they may be springy or they may give a little, because you may just have a trap that slips apart like that, and it may be a decorative nut, or it may conceal an inner ring that seals it up.
In the event you have a threaded nut with worn surfaces, hammer blows concentrated by a center punch or similar chisel implement to the exposed head or faces of the nut, directed in a tangental fashion such as to impart loosening torque by hammering it tangentially, can sometimes loosen some of these class of low profile bolt heads, when channel locks, crescent wrenches, or a socket can not. You may find that you should attempt turning it left-hand as well as right-hand hey you never know.
While a channel-lock plier like the others said is definitely the tool for the job, in some cases like possibly yours, since the edges are rounded off, you might not be able to get a grip on it.
In this case, I would reach for my handy vise-grip pliers, my favorite tool. You can grip just about anything with vise-grips.
After you remove the nut with your vise-grips, I recommend replacing the nut for later convenience. It probably costs a few cents at your local hardware store.
