My thing is, if one character were the opposite gender and the relationship would easily be interpreted as potentially romantic

i seriously doubt it would. it's way too graphically abusive.

lecter as written in the book series was "gentler" to clarice starling (their relationship did take a sexual turn) than he's been to graham on the tv show.