I guess because it's easier ! The human brain is lazy so by judging another person by his looks (The only public thing that this person is showing) he will have a general idea where to "put" that person in his virtual array so he will have an easier time making contact with him. Even if we try to avoid this it will always be happening in side our brains !