Remember that you wouldn't have to make the entire change all at once. You could get one or two pieces that work with your current clothing, or you could just change up your accessories. Go for more later if/when you feel comfortable.
(Also remember that I, a stranger, love to see people just following their own styles.)