No, your logic does take us there. If you can look at a person and can decide if they are attractive or not simply because you do or do not want to be attracted to them, you could change your sexuality at will by deciding "I will/wont like this gender!"

Of course, you also said attraction means you somehow can't control acting on it... so you are either a troll or a moron.