Is it legal to have a handrail on only one side of the stairs if it's not continuous? (As shown in the picture below)
Or should I remove that small handrail and put a long one all the way down on the opposite side?
Legality would depend on the laws applicable to your local jurisdiction. I can tell you that discontinuous handrails are a safety hazard and are against all current model codes in the US. But at the time of installation, if it was allowable, you are not obligated to change it.
From a safety standpoint, you should remove the upper portion and install a long continuous rail on the opposite wall. The lower rail needs to remain as a guard rail to protect from falls off the stairs to the lower floor.
If or while you're at it, check up on all applicable details with your local building authority to ensure you're doing it right. Rail height, rail dimensions, standoff, extensions, etc. all have specific dimensions you should conform to if you're going through the the trouble to make a safer stairway.