If your perimeter drain (drain tile for houses built with vitrified clay) is lower than the sewer or storm drain service line from the city you need a sump to collect the water and you'd need to pump it or gravity drain to somewhere lower than your property. The pumped water is typically directed to a gravity sump well that allows any sediment to settle before the water gravity feeds to the city storm/sewer. Secondary pump is probably a sewer pump if they have washrooms/kitchen/laundry etc in the basement.
Illegal isn't a word I'd use in construction. Non code compliant, non bylaw compliant, non-zoning compliant would likely be more accurate. Even non compliant construction can still be accepted with inspector supervisor sign offs, grandfathering or board of variance approvals.
A lot of jurisdictions do not require engineers forget geotechs to sign off on anything. Geotechs in residential construction I'd expect to be quite rare and reserved for only the most challenging soils / geology (cliff faces, peat bogs, underwater, rivers, swamps). A lot of jurisdictions don't care about retaining walls that are not related to the house - these are considered landscaping features, much the same way many decks, fences and sheds do not require permits. Certainly a 3'-4' retaining wall seems pretty reasonable to not expect a permit.
Ideally you do store water before pumping. You want to minimize the cycles on the pump. The easiest way to avoid cycling is to have a massive reservoir and only activate the pump when the reservoir is full. Since the water from the powered sump dumps into the same gravity sump that the rain water would enter it makes sense that you hear the sump pump noises from the down spouts.
It is almost unimaginable that he wouldn't be able to get rid of his water through the city service line but there was that mud volcano in Indonesia that was putting out the equivalent of a football field of mud at 8' high everyday. I suppose if you hit something like that you'd have a hard time draining it out to the city.
Overflow from a gravity sump sounds odd as it would never be used unless the line to the city was plugged which you hope would be rare as the sewer system would also start backing up into the house and in a combined sewer/storm the storm water would sooner backup in a basement than up to an overflow unless there was a huge surge that made it both backup into a basement as well as out the overflow (but that should be almost unimaginable as well ).
If I had to guess it would be the reverse - your garage and/or retaining wall is taking advantage of this sump and draining to it. It would make sense to put drainage at the bottom of your retaining wall that goes into a gravity well instead of paying to pump the water up from a sump that is lower than the sewer line and it would also make sense that your neighbor would allow your garage to drain in the same way if your garage didn't have proper rain drains that were connected to the storm. Sometimes garages drain to rock pits which if your property is at a higher elevation would really just end up in your neighbors powered sump. Your drainage system is possibly non-conforming to current bylaws.
What is the concern? Is it the noise? If you are uphill seems like you shouldn't have any water problems?