Simply because our brains are like that, whether we like it or not. It's not that people who judge from appearance are dumb, it's just that it's the first thing we learn when we see a new person. Judging off appearance first isn't that wrong, it only becomes wrong if you don't change your opinion on the person after you actually get to know what they're like. Read about the Halo effect, it seems like it would interest you.