no, because predators wouldn't avoid it if that were the case. The animals adapted to wear the colors of the mushrooms to ward off predators to that they could survive. If the Mushrooms were just mildly toxic then it would be useless to evolve in that direction