Adding on to Simon B's answer..
If you'd rather not assume the main breaker is working correctly, you can test the voltage at both sides of the main breaker and visually inspect it to make sure nobody has connected some bonus wires to the wrong side of the main. Remember the entrance side is always energized and you should wear eye and hand protection.
Are my assumptions correct?
They look reasonable.
Is there a way to find whether it's the neutral wire or soil that's responsible for the issue?
Try a long lead and measure the voltage difference from several different places to the neutral. If the voltage is consistent, it's not the ground that's the problem.
Any way to mitigate this issue?
With the main breaker off the source has to be from the neutral line the power company is connecting you to. To find the source will be difficult, it could be anybody else connected to the same transformer with a fault to ground. Try installing ground rods, I used an array. I was only getting a few volts but was told that was normal. We eventually found it ...