looking to replace a regular switch with a ceiling fan switch which has separate dimmers for the fan and light. Allegedly this was wired to be able to do this. The old switch has a red wire, a black and a ground. The new switch has a ground, black, red and yellow. The old switch’s black wire connects to a mass of wire in the box which also has another red wire. Question is how to connect new switch and where does the yellow go? The directions from the switch just says the yellow is for the light
It would be easier if you had a picture of the wiring at the fan. If this is wired correctly for what you want, there are two hot wires running from the current switch to the fan. One feeds the fan and the other feeds the light. Your new switch separates these two feeds - the wire running to the hot for the lights is fed from your new yellow wire, while the fan takes the other hot out.
Get a tester and/or a good chicken stick.
Right now, you have about 5 wires tied together with the wire nut. It seems likely that one of these is the constant hot. Isolate that, and half the work is done. That will feed the new switch. The other side of the old switch probably runs to the fan, and will connect to the new switch accordingly.
Here's the problem: That wire nut has about 5 wires. If I'm right, one feeds the switch and another one is incoming hot. Of the remaining 3, 1 probably runs to the light now. If this is the case, the lights will work with the old switch installed as it is now but turned off. If you can isolate that one, connect it to the yellow on the new switch. The others go elsewhere, and will probably need to be tied into the constant hot.
I'm not an electrician, so be careful and double check everything.