Because the Averia family force was so great that he couldn't help esther. Besides, he also loved averia very much.
About three years ago Esther met the queen mother. The Queen Mother loved Esther as much as her own child. The woman also knew Esther's true identity. Therefore, she always protects Esther because she himself knows that Lucius II, his biological son will never be able to protect Esther.