Technically they are half owners of the fence so as such have equal rights. I would have a friendly chat with them and suggest they sing out to you to let you know so you don't inadvertently do something that may be harmful to the rug. A political way to broach the subject that is non confrontational