In the early church, when people were meeting in their homes, they came as they were. When Jesus preached, people came as they were. I'm not berating those who choose to dress up to one extent or another (I myself do, though I do not wear a coat and tie to church on most Sundays). But it does appear that the "come as you are" dress is no more or less nonchalant than the dress of the early church.