It can be used as a sexual idiom, but it does not necessarily have to be. I can see why someone might have thought that OP meant it in a sexual way, definitely, but I think it's more likely that OP meant it in a non-sexual way, and I understood it in that way, and I don't think there was anything wrong or misleading in that usage.