This is literal dogwater Kakashi Jiraiya and Fugaku would be able to take on the entire village to keep the brothers together and they all WOULD do it.
It's possible if Konoha isn't devastated, so Jiraiya needs to monitor any actions from the other villages, Kakashi got depressed after things started to get better. Fugaku is from Uchiha clan which has no support in the village because of suspicion.