My house has Water Softener (Ionics IQ Series) installed by previous owner. We really haven't replaced the filter diligently, is there any consequence for this? I mean, will it make my drink water bad?
That series was sold with and without a carbon filtering system which makes it a little harder to described. If you have a carbon filtering system (there's is called a "HYgene Bacteriostatic Filter Media" containing Silver Impregnated Activated Carbon) which is the more typically and suggested unless you have well water, the manufacturer suggests replacing the carbon filter every 3-4 years. The carbon is porous and is underneath a few levels of filter media and will hold onto the smaller particles. Also, that series appears to be fairly typical and has a salt tank filter, I highly recommend checking and more than likely filling that up. The salt is fairly cheap in comparison and the system is designed for the homeowner to maintain that themselves. The salt aids in the life of the filtration system, the filter system washes itself out and adds salt as a Regeneration cycle.
The consequences, the regeneration cycle is an attempt to clear the debris and if the media is too old or too much media has been flushed away the regeneration cycle will either stop and your water pressure will drop (would probably take months for you to notice) or the regeneration cycle will happen significantly more frequently wasting your electricity and remaining salt. The less filter media the system has, the more likely it is to get things (bacteria, hard minerals, debris) through which is no worse that not having the filter. The carbon will "bleed" and release the bacteria it was holding onto but that is just what it does and will not typically effect smell or taste for many years past it's prime if it is kept wet (which it is as long as the bypass hasn't been on since you moved in).
The systems themselves are expensive, the filter system look to be < $300 USD with installation. So you may want to look into getting it all fixed up. You'll have to monitor the salt about every 6 months but otherwise it should be maintenance free for at least 3 years.
If you are still concerned, most people install a bypass value system for maintenance of the system (an for heavy gardening, filling up the pool, going on vacation). This series appears to have the bypass installed as part of the unit. The bypass value does exactly what it sounds like, it bypasses the entire unit. Looking at the Manual this unit has a single valve to bypass (Page 5/6 manual using verbage "by-pass"). Sidenote: most home-made/plumber installs for a bypass system will have two valves, one for redirecting the input around the system and one for switching the output from the unit to the bypass.