I think he's just worried that Rafael will fall for Audrey, or even worse, Audrey will fall for Rafael.
That being said, I initially felt pity toward the doctor but after he called Audrey a creature... nope... continue torturing him.
Frederick has been keeping Audrey away from Rafael. I just hope it's not because Rafael would go madly in love at her đ seriously, couldn't think of a reason. Good job until now, author!