EDIT: Per the excellent answer below,this was (to my surprise) a three way switch. Thankfully I had a few extra three ways lying around! In addition, the other switch on the three way was wired incorrectly, I guess by the previous owner, which was making things even more confusing. They had apparently flipped one of the travelers and the line. Good times! All fixed now at least
I'm swapping out a dimmer switch for a smart switch, or was attempting to. I have tried a regular switch instead with still no luck.
Kitchen lights: A and B were wired up to the dimmer switch. The dimmer switch's ground was the left pink mark. The dimmer switch's neutral was connected to the C terminal for the second (pendant lights) switch.
Pendant lights: C and D were wired up to a regular switch. The ground was the right pink mark. As mentioned, the C terminal also supported the neutral for the first (dimmer) switch.
I swapped the dimmer switch out for a smart switch. I connected the ground to the same ground wire and the neutral to the bundle of white wires in the box. This was my first indicator that something was weird, because the smart switch did not have power. Moving the smart switch's neutral wire to the C terminal of the second switch provided power to the smart switch, but not to switch the lights on.
Here I did try the second neutral bundle in the box, but that also provided no power to the smart switch. Only the terminal on the second switch provided power.
Thinking something could be wrong with the smart switch, I used an extra switch I had lying around, a regular switch. This also did not work.
Next, I removed the switch and tried just touching the wires together, with power on. No luck. I tried the dimmer switch again in the original configuration, and it worked. I removed the dimmer switch again and attached wire A to the C terminal of the working second switch and wire B to the D terminal of the same second switch. Interestingly, this worked, but now I had one switch controlling both the pendant lights and the kitchen lights, which isn't exactly what I want lol.
I read online that in some cases, dimmer lights specifically need a dimmer switch. I swapped out one of the lights for a regular lightbulb, but this made no difference in any testing.
Removing both switches, I started testing different wires together.
- A+B - nothing
- A+C - kitchen lights on
- A+D - nothing
- B+C - nothing
- B+D - nothing
- C+D - pendant lights on
So, uh, any idea what's going on here??