There's a couple of ways to play this. Neither is terribly aesthetic.
Deep metal junction box. You'll need clamps, but these are cheap and easy to find. Add a lid
A large PVC box. You can buy these in varying sizes (4x4, 6x6) and they come with no holes (you'll need to drill some). Make a hole and add a clamp to fit it
5 cubic inches per 6 Ga conductor terminating in the box (30)
3 cubic inches for up to 4 8 ga grounds. (3)
33 cubic inches.
If the NM clamp is external, no additional allowance. If the NM clamp is interior, add another 5 cubic inches (38).
A 4-11/16 x 2-1/8 steel box will have room to spare. You can dress it up however you like, paint it, etc. Or you ...
There are several problems with this installation already that need to be changed to met NEC, I would say, remove the square box and start fresh. [Make sure the power is off at the breaker before you start] Back in the day when I was in the trade, I had a 3 1/2" dia deep cut hole saw, Electrical supply companies make an "Old Work" ceiling box ...
If the nm is exposed in the back of the cabinet yes the NM would have to be covered where exposed.
I will use a water heater because it’s easier visualize.
Where the cable comes out of the wall I slide a piece of flex over it long enough to connect to the heater J box, I do use anti short bushings but the outer sheath of the NM is sufficient for most ...
These don't get "removed", they get cut off. Snip the copper solid wire at the end of those connectors, strip off the insulation and then use wire nuts or some other approved connector to connect the wires for your new light.