If the bandits banded together they would win. First all supply routes would be cut off, due to the danger of traveling roads. Then the bandits would slowly pillage each town and demoralize the opposing forces. Then, with sufficient towns and reinforcements, they would kill off all the dragons and such with superior numbers.
They can also steal the horses.
Edit: Come to think of it, haven't bandits always won? Think of Rome, arguably the most poweful empire to ever exist. Bandits/Raiders were one of the biggest contributions to its decline.