Perspective is everything. I’m sure with his mother gone things were extremely difficult for him. I doubt that the imperial consort or the other concubines were nice to him. Especially when he got frost poisoning. They most likely looked down upon him even more.
