The only reason I modified the height of the door handles at my place was because I was completely replacing them and it was easier to drill completely new holes than to try enlarging the existing ones neatly. Then could fill and sand the old holes prior to painting the doors.
Given I'm partly through this process, the house has a collection of heights and handles throughout. :-)
How did you conceal the original holes? Was filler enough or are you still able to see the outlines?
It always seems to be difficult to successfully hide a repair like this, especially on a off-white door.
I was considering routing out a section and glueing a "patch" in.... But maybe I'm over thinking it..
Filler can work but it seems to shrink slightly over time so there are outlines visible now. Not sure exactly how long it would pay to leave it to reduce that issue though.
Patching could work but you'd still need to fill in the join unless you're really good at making an exactly fitting patch.