You're probably not saving any meaningful amount of electricity by turning it off for just 2-3 days. Sure, it won't use any for those days, but the temperature inside will raise to room temperature. After you turn it on again, it will have to work for many hours to cool down again. This could cause abnormal wear on the compressor, because it's designed to be frequently turned on and off by the thermostat just to keep the temp level, and not to work for several hours in a row to cool the whole thing down from room temp. However, this effect is probably insignificant.
By not opening the door when you're gone, you're already saving electricity. The most work the fridge has to do is to cool down all the warm products you put in and all the warm air you let in. If you're not there to open it, it's a bit electricity saved already.
The most significant part had already been covered in other answers: turning a fridge off is a multi-hour effort that involves removing food, having the frost thaw and cleaning up resulting water. Turning it on again is also non-trivial, as it requires drying it thoroughly to prevent immediate frosting. Thawing and refreezing without all this effort will damage the fridge, through mold and ice damage. The whole cleaning is well worth it when you leave for few months, but not for few days.
If you want to optimize the fridge, make sure that the hot part (usually at the back or built into the sides) has room and unobstructed airflow. Small fridges are convenient to be stuffed in cramped spaces, like under desks. Simmering in a pocket of hot air greatly increases power consumption. Observe when the compressor is running and after a while (eg when it stops) use your hand to check if the air around the element feels warm (not the element itself, it's supposed to be very warm to touch). It should not: all the air warmed up by fridge should convect away freely. In a nice, cool environment, the fridge has much less work to do. Less work = less electricity.
You can observe how often and for how long does the compressor turn on when the fridge is not opened. It should barely work at all. If the compressor is on most of the time, then it means that the insulation or seals are damaged and the best course of action is to either repair the seals or get a new fridge.
The rule of thumb that one should follow in all areas of life is to first measure the actual costs before trying to cut them. In programming we call it "premature optimization". The problem with it is that it yields insignificant savings while detracting your attention from areas that actually need improvement. Get a kill-a-watt to measure how much running the fridge actually costs you. If the fridge is very old and leaky, you could find out that moving towards more expensive non-perishable food can be financed by ditching the worn-out electricity hog entirely.
I hope your fridge has an actual compressor. Peltier-baser solid-state portable fridges are ridiculously inefficient and IMHO not worth using at all.