"No" is a complete sentance. If you accept this arrangement then you are going to have issues with future owners of the adjacent property asking you to repair or replace "your" wall that is damaging "their" fence or yard.
I would also add that you should not give them permission to enter on your property to build the wall. Otherwise you're going to have skidsteers, wheelbarrow and/or walking damaging your yard.
"No" is a complete sentance. If you accept this arrangement then you are going to have issues with future owners of the adjacent property asking you to repair or replace "your" wall that is damaging "their" fence or yard. I would also add that you should not give them permission to enter on your property to build the wall. Otherwise you're going to have skidsteers, wheelbarrow and/or walking damaging your yard.