The new Camry is significantly bigger (especially the interior room) than previous years, but it's still a sedan - not exactly the best solution for a growing family. I'd suggest a larger hatch and/or wagon.
If you want to buy new and keep it under $20k, then my suggestion would be the Mazda3 5-dr wagon. I own one and loooooove it, but besides my bias it consistently rates at the top of its segment and just seems like the best overall package. Styling, performance, interior layout and design - useable space - it's got it all. It's got above average interior room for its class and good cargo room (definitely noticeably more than a Prius), but it's best selling point is the performance - it has really great handling and a nice engine (not the best mileage, but at 26/32 it's no slouch either). It should be enough for a family of 4 though I admit it'd start getting cramped with 5. On trips it tends to start getting cramped with me and one other person + luggage, but that's only because my dog takes up the cargo space and the luggage goes in the back seat.
If you want to go used, then you could step up in size and get something like a Subaru Legacy or Outback wagon. Nice car - with a much bigger front row and much more cargo room than the Mazda3 - the bonus of AWD, and it's pretty sporty for a big wagon. Of course mileage goes down, but it's still not bad. You should be able to find an '05 model for around $15k, maybe even less.
Oh, and if you haven't yet, head over to
www.edmunds.com and get to know it well.
It helped me out immensely before I decided on the Mazda3.