My guess, and it's only a guess, is that that pipe is where the downspout used to attach to the sewer system. Until very recently, downspouts used to connect directly to the sewage system. Many municipalities have passed laws mandating the removal of these connections because the treatment plant would often overflow during a storm, leading to the dumping of untreated sewage into natural waterways.
So what's happened, is there is a block downstream of the connection between this storm sewer and the household sewer. Typically, if the blockage occurs on the road side of the water supply cut off --
-- then the city is responsible. If it's inside, then the homeowner is responsible.
In my city, they'll run a camera down the pipe, and charge you $300 for the scoping if the blockage is on your side. I'm betting in the short term, you'd be better of to get a city licensed plumbing and drainage contractor to snake it out and absorb the cost. Not a DIY job, because the city will make you pay for anything that goes wrong.
The next issue is how to stop it from happening again. You'll probably have to dig down to the connection, remove the downspout and cap it off. Again, since you're playing with municipal systems, this is not a DIY job.
You might get lucky with calling the city, but then again, they might inspect certain systems and order you to fix them right away, at a cost you likely can't afford if you just bought the house.