Waking up with Sumire in his arms shouldn't be normal anymore, but at least a few days in the week. He ends up waking up with her in his arms. He was careful not to wake her as he stood up, got changed and headed out of the room.
Besides, Mamoru probably doesn't want Sumire seeing how weak he is. Last time they visited, Mamoru looked like he wanted to bolt out the window. If he had enough energy to spare, he definitely would have.