We recently moved back into our beach house after hurricane Sandy. During the storm repair the lower level was rebuilt including moving a bathroom and moving the house valve (only a few feet). We always now hear the sound of water running through pipes (a whooshing sound)even though no water is running. Here is the strange part. When I shut the house walve ( this is inside the house where the water line enters the house which is in the wall behind a panel)the pitch of the sound changes slightly but it does not go away. One would think if there was a leak after the valve the sound would stop. In addition when we run water in the house we hear the usual sound of water running in the pipes layered on top of the first sound. The pitch of the first sound does not change. The plumber is stumped as am I. When could possible cause this? It's pretty annoying.
I suspect the sound is from the drain pipes, and comes from a leaky toilet, beacause that's the most common thing that has a stored reservoir of water to keep leaking after you shut off the main valve.
You might try leaving the main water valve shut off for several hours. If the sound eventually stops, go around and look in your toilets to see if any are now empty in the top tank (without you flushing) - or speed the process up and flush all the toilets after shutting off the main water valve and see if the sound then goes away shortly after that.
If you'd like to check on @Tester101's theory, try shutting off the main electric breaker and see if the sound stops or not.