Should I go to a top 20 university/college or settle for something else and then go to a top Grad school?

It does matter. That's how you get into a good grad school. Good grades at top universities. The school you go to makes a big impression on the grad school you are applying for.