Because your argument can be applied to anything. If i made the same argument that black people make me unsafe because of their statistical homicide rate
" There is nothing strange about saying your existence is putting me at risk. "
Your argument lacks the soundness property. It can be applied to thinks that are false. That means it's not a valid argument. If a line of reasoning proves false things aswell as true things it's not reasoning, it's a delusion and a contradiction.
If you accept this reasoning then you accept authoritarian decisions to make an example of people who disagree in all facets, you will be unable to oppose them when it's over something you dislike.