To me this would 100% depend on the job and whether people returning late is affecting others' ability to LEAVE for break. Like, in my work all breaks are done so that there's always x amount of people around, so if you come back 5 or 10 minutes late, that next person is leaving late and may not get their full break if they need to be back at Y time.

In an office, as long as they're not missing meetings or something else time sensitive, who cares?