So the title is obvious. It matters because other people see us, unless we are hermits. And when people see us they judge us, whether right or wrong, and if we are trying to communicate something verbally, then the less time they take to judge us, the more likely they are to have the time and focus to listen to what we are communicating.
That's my simple summation of a great article in Forbes that it's worth taking the time to read. It's written from a female perspective but I think that the psychology behind it is equally valuable for men and women. So if you want the expanded version of my summary above, take a look, and see if it makes a difference in your life!