I also agree, I personally don't really picture his outfits too much when reading. Remember when a author tries to cater to his critics too much the creativity of an author usually gets hampered and the story suffers.
Who cares what he’s wearing when people can just picture what they want. You do you, this is your story that I assume your writing for your self at the end of the day.
It don't matter this much. but when you put. 'he's wearing' *insert image* and it looks like something a loser or child would wear then all interest in that chapter is gone.