I'm guessing you don't understand much about Japanese culture then? They aren't avoiding you because you're black (well, some might be), they're avoiding you because you're foreign and they don't want to be "confronted" by you with something they don't understand (e.g. a foreign language they don't understand, a question thy can't answer, etc).

I'm white and it's very common for people to avoid sitting next to me on public transport, or move to another seat when I sit next to them, or to "politely" turn away and pretend to be busy so that I don't speak to them.

It's not that they're trying to be rude. They just don't want to be shamed in front of someone who is ~different~ by not being able to understand you. It's a part of the whole 'Japanese pride' shtick.