It started as a system, a genuine way to keep track of things. A way to ensure that you had a firm grasp on what you had worn the last time you had seen someone, and a way to ensure that you wouldn't repeat yourself too often. The fear was that, being someone who considered one's friends' preferences when choosing clothes, one would remember that Lucinda liked green, and so wear the green sweater when going to see Lucinda.
While this may not seem like something to fear, it's important to consider the possibility of repetition, or coming to that same conclusion over and over again, so that one wore the sweater every time on saw Lucinda. The resultant (feared!) tragedy, of course, being that Lucinda would not assume that the sweater indicated a shared interest in all things emerald, but rather that it indicated a limited wardrobe, possibly that you even only had the one sweater.