What is the maximum depth (distance from fence) you want to make a garden before it becomes problematic to do the gardening etc?

As well as smaller garden plants, we'll probably want a few trees to provide a bit of shelter from the wind and maybe some shade.

This is just in a residential suburb.

Thanks