People don't understand that the MC here is putting up the act for us, the readers, as well. He's pretending he's normal for us too. That mask will slip eventually.
Everyone who is confused about why Ayanokouji isn't "cool and calculated" like in the anime, it's because in the LN, he tricks the reader into thinking he is "normal" as well. Later on you get glimpses of his true character.