Hard to know without doing some investigation or starting to take steps to address it and see where it pops out.
One thing to try, if you have not, is to flush the fixtures while the end of the faucet or showerhead is off - run 10-20 liters of water though, full-blast, both hot and cold. That should remove anything just sitting in the pipes. You might also want to turn off your water heater (depending what type it is - this is primarily for a tank-type heater) and flush it from the drain connection.
Most likely is that this is coming from the water heater, as various dissolved minerals are prone to come out of solution when water is heated. But it could possibly be coming from the street supply. A filter might address that adequately - one for the output of the hot water needs to be specifically made for handling hot water. I like the so-called "spin down" filters that do most of the filtering by swirling the water to throw particulates out, and then have a central screen - the nice thing is that they also have a valve on the bottom to remove accumulated debris without actually needing to change the filter, so they are economical and convenient to maintain .vs. filters with replaceable filters. The screens can be cleaned or replaced if needed, but they usually don't need to be replaced.