UHD:True, this has been my point. I can see why someone might feel like filtered drinking and even shower water but I see no issue with chlorinated water in the toilet and definitely gardening water should not need filtering.
My place is still a new build in the planning stages; in your experience is it likely to be a huge pain in the ass/expense to have the plumber split off the outdoor and/or toilets before the filter? I feel like the pipes feeding the bathrooms would need duplicating for the toilet which is just wasteful.
It's not terribly different to having toilet/outside taps set up with rainwater - you have duplicate pipes from the point of entry to the fixtures. That's pretty common now.
Water pipe is pretty cheap - can't cost more than $1k if do it while the framing is open. Possibly a good return on your money if you consider the cost of cartridges