Did they take a quest to exterminate the bandits. I think you should write in small things like this even if it's only mentioned to make it more believable. Vance as a former novel reader shouldn't have underestimated them and at least tried to get some benefits.
