Why should we try to stop a speaker from coming to an event they were invited to? Let us imagine the speaker poses a threat to the welfare of the audience; or the speaker espouses views so egregiously offensive that they should not be given any sort of platform to express them. Something like safetyism is thus invoked as the reason for stopping the speaker from coming.
This is especially the case in institutions full of young people, like universities. People who bang on about free speech are just using that talking point as a figleaf to justify harmful speech — the kind of speech that makes life harder for oppressed minorities. This, at least, is one way to see it.
