Yes, there's one blackface scene (in Swing Time), which is a little cringeworthy, but was apparently Fred Astaire's tribute to a black dancer who had influenced his style, so there was no mean-spiritedness behind it.

I agree with your opinion. I love Fred Astaire's movies so, so much. The man was a genius with choreography and Ginger Rogers was a fine partner for him.