That 1/4 inch drive may simply not be strong enough to do some work. For example, if I'm taking the lugs off of a car, a 1/4 inch drive may be overwhelmed. On the other hand, I like smaller sockets to get into a tight place.
So I like to keep a mix, up to impacts for the impact wrench, in case I need to do something on the tractor or to rotate the tires on the car. Those tend to be 1/2 inch drive, or at the very least, 3/8. I broke a 3/8 drive wrench last year, so sometimes the extra beef is nice to have.
If you will NEVER do anything heavy, you may appreciate the light weight and ease with which you can carry a 1/4 inch set. So is rotating the tires on your car major work for you, or is it something you might do?