Yup, one of the things that wasn't addressed is that there's a "neutral" in there too - doing good things, not doing anything, and doing bad things!

There are tons of reasons to NOT do things that harm others but help you in terms of logical consequences, and part of that is losing the trust and goodwill of people even if you're never formally charged with crimes or arrested or anything. All the people you screw over, provided they're not dead, will probably never take your side on anything again.