Mixing up your greys with a neutral color is definitely an acquired taste. Some people consider it a faux pas, but I think it's a fabulous look when pulled off correctly. Think of this type of styling technique similar to mixing up your jewelry box metals. Sometimes your outfit calls for a strict all white gold approach, but sometimes it's fun to accent a two-toned gold watch with a bold gem in a gold setting and a shiny pair of chunky silver earrings.
It all depends on the tone...literally...that you want to set for the outfit.
In the picture above, this fabulous brown bag is the accent this outfit needed. The brown lenses with gold frames act as a complementing piece and tie it all in together with balance...dismissing the fact that the entire outfit is created with cold tones, including the denim.
You don't just need to accent your greys with a color, neutrals work just as well when brought in with balance and thought...mix away.