I am trying to install a decora smart switch for my front home lights. The switch is in a box with the hall light switch and a switched outlet switch. I think the front lights switch is a single pole but not sure how to connect everything. Thanks in advance.
I think I see what is happening. Many devices (switches, outlets) allow you to connect via either the screw terminals or the "backstabs" as we call them (with some contempt because they tend to fail a lot).
You're supposed to use only screws or backstabs. But the screw is connected to the backstab internally, so this is a possible way to use both, which splices the two wires and also connects to the device.
It's just a lazy cheatey way to splice both those wires to the same screw without using a wire nut and pigtail.
Now, I think this is a 3-way switch, but you didn't mention it being controlled in a second location, and the wiring doesn't make sense for a 3-way, so I think this is being used as a 1-way switch simply by not using one of the brass screws.
The two yellow wires are definitely using the "use the switch as a splice" trick. Both those wires should go to a wirenut, with a third short pigtail going to that same switch's screw. Since your lights are in three locations, my guess is the two yellows are both switched-hot running in different directions out to the lamps.
I can't quite tell if the black wires are on the same "screw", but it looks that way. Note how the
It's a little harder to tell whether the blacks are on the same "screw"... But you see the stabbed black is a pigtail to a wire-nut. That is surely supply hot being distributed to several switches. Which would make the screw trick simply extending that hot further. If so, the wire on the screw now should go into that wire-nut, and the pigtailed wire should use the screw.
If it works out as I describe, now you have 2 wires going to that switch: the black is supply/always-hot/line, the yellow is switched-hot/load. If your smart switch needs neutral, you can add a white pigtail to that bundle of neutrals in the back of the box. (which you should not otherwise be messing with!)
If there were no switch you would have a hot wire bringing power from your service panel and a neutral wire returning to the panel. Connect these to a lamp and you get light. The hot wire is on the "line" side and the lamp is the "load". The neutral wire can be white or grey. The hot wire can be most any color but not white or green; black is the most common.
If we disconnect either wire, hot or neutral, the lamp goes out. However, for safety reasons we must never disconnect the neutral wire. This is an important rule and we will return to it later. A single-pole switch is connected inline with the hot wire on the way to the load. The line is connected to the darker screw terminal and the load is connected to the lighter screw. When the switch is on the hot line is connected to the load. When the switch is off the hot line is disconnected from the load. In either position the neutral wire remains connected to the load.
On your existing setup it looks like the black wire attached to the dark screw terminal on the switch is the hot line. The two yellow wires are connected to the line voltage when the switch is on and these presumably go to the front door and garage lights. The other black wire in the back of the switch is always connected to the hot line. I expect that the pigtailed wires attached to it are then connected to the two switches on the right hand side of this box but they could be used to expand this circuit elsewhere as well. Finally, one of the white wires in the back is the neutral wire returning to the circuit panel and the other white wires pigtailed to it are connecting various circuits to neutral.
This installation is OK but a bit lazy. It would be preferable to pigtail all those black wires together and connect a short extension wire to the dark screw on the switch. Same thing for the yellow wires.
To install the Leviton DZ15S light switch in a single-pole configuration (1) pigtail the other black wires to the black line wire and run an extension to the dark screw marked BK, (2) pigtail the two yellow load wires and connect an extension to the magenta screw marked RD, and (3) pigtail the white neutral wires and connect an extension to the light screw marked WH.
At this point the light should switch but to have a safe installation you need to connect a grounding wire to the green screw. If there is already a bare Cu or green grounding wire in the box then you may pigtail off it to connect to the green screw. Alternatively, the box may be grounded through a metal conduit in which case you need to connect a short jumper between the green screw on the switch and either a grounding screw in a spare hole in the box or a grounding clip attached to the edge of the box.
In the comments you reported installing an unrelated circuit by taking apart two white wires and connecting them to either end of a switch. These are quite possibly the neutral wires of that circuit in which case you have created a break in the neutral wiring whenever the switch is off. This is a serious electrocution hazard. You may want to open a separate question regarding how best to wire that circuit.