I would think enslaving the minds and will of people, would somehow have a negative effect on their souls / psyche.
Maybe damaging the soul if they’re ordered to do something against their nature.
I can only believe that observant individuals will catch on to others erratic behavior.