I don't think so

First, you are liable for things under your control.  It's not clear if you have control over the fence wall.

Second, just give him your home owner insurance, let the insurance figure it out