It is your pool. It is your job to guard it.
I imagine your state law or city ordinance is clear on that point.
You are missing an important point here. You are trying to use your neighbor's fence to protect your pool. It's his fence and he has a right to remove it at any time. He could remove a panel right after you go to work, and put it back right before you get back. That's his right. If kids got into your pool and drowned because he took down his fence, that's all on you.
If you think that example is absurd, he could easily put a gate or several gates in his fence, and refuse to lock them. That is his right, and you don't get to say boo about it.
He does not owe you a fence
I suspect part of the tension is that you are trying to get effective use out of his fence, yet haven't paid a dime for it. His feelings are reasonable, and you should respect them. You need to pause here, and realize so far, your conduct has been that of an over-entitled jerk, and his negative reaction would be one you yourself would be having, were the roles reversed.
Normally when 2 neighbors want a fence, they split the cost of the fence. And most states have laws around how that is handled. I suggest you research that law.
Now, since this fence is already built, and you want to make use of it, I suggest pricing a comparable fence, so you know what genuine fence costs look like. You can do this on the pretense of doubling the fence, which is your ultimate option, after all.
Then once you have hard numbers, you can have an informed discussion with your neighbor about (instead of wastefully building two fences) "buying in" to the fence he's already built.
That's a bargain for you at $1 less than the cost of your own fence. However, if you are respectful, you can probably talk him into half the value of the fence. If he wants to turn down thousands of dollars, then clearly you've done something else to upset him pretty seriously.