You might want to look at a 2 step process of first smearing glue/epoxy/whatever onto the wall and working it in a much as you can, then let it dry/seal/cure,

Then rough up the nice new face and attaché the latch to that...