Also? Document everything! Write down the times and dates of every incident and as objectively as you can the details of what happened, specifically how it impacted the ability to do your work. Don't put in long rants, just a timestamp, what was said or done, what he said or did, and the negative impact it had.

The prof might shrug it off, even if you go as a group. If you go to him as a group, with documentation (leave copies with the prof, NOT the originals), it will be harder for him to brush it off.