Long and boring, but the proper way...
You will have to see how the cabinet is fastened. If it is screwed on the sides to the "walls" of the housing then you need to remove it before removing the "walls"
If the cabinet is screwed on the back- to the actual wall (even id it has side screws) then empty it out completely and you should be able to safely detach the wooden "wall" you want to move.
In this case you will need pad out the extra space in-between the cupboard and wooden wall you are moving with another piece of wood that will fit. You need 3" inches.
A nice way would be to pad both sides of the cabinet with 2" thick pieces of wood to give you an extra 4". Painting it white will make it look like it should be there.
Then you need to screw from inside the right side of the cabinet through the padded wood into the wood of the existing cupboard -- but not so the screw comes out the other side. You need to measure precisely the length. screw in each corner and centre. Then put the left wall back in and screw from the left side inside the cupboard the same again.
You obviously have a problem with the light switch. IF you got enough space inside you could make a square opening in the side "wall" that you are moving and put the switch in there. The wires will be behind the fridge any way - just insulate them with some flex plastic pipe for wires.
The light switch will have to get moved to something similar like this. You can mount it flush /countersink it into the wall you are moving. The biggest problem is cutting out the hole to the correct size. You can go larger but you cant shrink. So be careful and try to fit it several times.
It is not an easy undertaking- but I would not do it any other way.
If you cant be bothered ...
If you cant be bothered doing it the right way. Then put several wall bolts in the back of the cupboard in the corners and centres. Then remove the left "wall" and put your fridge in.
But this will NOT work if you have a drywall behind the fridge.