You want to get an activated carbon filter (see this answer for an overview of how it works).
You'll probably want to look at a standard 10" filter housing and a matching carbon filter, both of which are readily available online and in retail everywhere, including the big box stores.
This can go under your sink or elsewhere upstream. There are clear models available, though these are really only useful for a sediment filter, and even then monitoring the pressure drop across the filter(s) is a better way.
Be sure to install an upstream shut-off nearby (if there's not one already) so you can turn it off to change the filter. And pro-tip: if you're installing under the sink or other confined location, dry fit it and ensure you'll have enough room to replace the cartridge.
For activated carbon, all you can really do is monitor for taste, and/or change it on a schedule (off the top of my head, at least every 6 months seems reasonable, but check what the manufacturer recommends). The filters are actually rated based on flow rate and the total volume they can handle.
You can also put the filter upstream of more than just the hot water dispenser (eg, include your kitchen sink cold water as well), and depending on your plumbing and space this might be easier or make more sense. Just be aware of the flow rate (gpm) rating, and total volume. I'd guess for most uses serving the kitchen cold tap only, a 10" filter would still be fine for this but you do have the option to go to 20" if you need more volume/flow and have the space.