What kind of conflict of interest? I'd want concrete examples and to weigh up the options before I let that add complexity to an already complex project.
You have to keep an eye on any work. If you use one firm they give a warranty and manage everything. If you use a separate project manager if anything goes wrong you'll have to pay them to work out who caused the problem, or work it out yourself and be subject to the blame game.
No concrete examples, just the abstract concept of mistakes being made, and not being penalised or dealt with properly due to the building company also being the project management company.
To be fair, the person that advised this to us does have skin in the game (they're project management specialists). Hence my desire to get real world experiences from others who may have taken either route.