I'd only replace all of the siding if ANY of the following is true:
At least 30 percent of the boards are visibly and significantly damaged. If you only have a few boards showing signs of rot, it's not a big deal.
There is evidence that the damage was caused by an insect infestation. That would mean that the damage results not from simple, unavoidable weathering, but by a more serious issue.
There is water penetration and resulting damage behind the siding (either inside the walls, or -- worse -- into the interior of your home). If the damage is limited to the siding itself, I wouldn't worry too much about it.
Absent any of these conditions, I wouldn't be too worried. Unless you are diligent about keeping it painted, all wood siding will rot a bit eventually, especially near the ground where it gets wet (and it doesn't take much paint loss to allow moisture into the boards and cause the damage you are seeing). Unless the problem has progressed to the point that it has become severe (affecting a large portion of boards, allowing water inside the building envelope or inviting insect infestation), then fixing it should be as simple as replacing any individual boards that are badly damaged, then putting a new coat of paint on the siding.
Contractors, Realtors and many home inspectors will likely encourage you to replace all of the siding, because they are incentivized in different ways to do so. (Contractors want to do the job; Realtors will be able to sell your house more quickly and easily if you do a full replacement, even if the cost of repairs won't be recouped in the sale; home inspectors are incentivized to find "problems" that their clients -- the home buyers -- use to negotiate a price reduction.) I'd resist their recommendations unless you have good reason to believe that full replacement is warranted. If they disagree, ask them to show you a house with wood siding that was built at least twenty years ago and does not have a bit of rot here and there.