Lee Anderson has been accused of "parroting far-right misinformation" after saying he had sympathy for people protesting outside hotels used to house asylum seekers.
The Tory Deputy Chairman has sparked anger with his comments after an increased number of attacks on hotels.
MPs last week heard the government must clamp down on far-right groups Patriotic Alternative, Britain First and blogger Yorkshire Rose who have been blamed for driving the sickening violence.
Mr Anderson said the protesters, who threw missiles at police, were "just normal family people" with concerns about the safety of their community.
When asked if he had sympathy with people protesting outside the hotels Mr Anderson said: "Of course I do.
"These are not far-right extremists, they are just normal family people from some of these towns and villages that are upset that overnight 200 to 300 young men have arrived.
"And then they are saying things to young girls - and I know there have been a few attacks and some horrible incidents.
"So of course, people are going to be concerned."
Speaking to the BBC's Political Thinking with Nick Robinson, Mr Anderson said: "When you live in a community, you expect to be safe and you don't like sudden change.
"That is how humans behave."
Last week Home Secretary Suella Braverman said she understood people's "frustrations" about hotels housing asylum seekers and that this was "causing understandable tensions within communities".
However, she told GB News "violence is never acceptable".
The Hope Not Hate campaign group accused Mr Anderson of "parroting the misinformation used by the far-right to exploit hotels".
"This is precisely the same kind of myths that the far-right are peddling to whip up hate around hotels," a spokesperson said.
"We can't have a Conservative MP amplifying these messages."
Care4Calais founder Clare Moseley told the Mirror: “No one supports the policy of putting asylum seekers in hotels; their claims should be quickly processed so they can get on with their lives, work, contribute to society and support themselves.
“The solution is to lobby the Government by writing to your MP not targeting the hotels.
"The people being housing in the hotels are victims; of war, torture and human rights abuses; they have no control over the situation and don't want to be in hotels. It’s up to the government to fix this, it’s not the fault of refugees."
Responding to Mr Anderson's comments, a No10 official said: "That's not the view of government... the Government's position that there's never an excuse for violence or intimidation."
Follow Mirror Politics on Snapchat, Tiktok, Twitter and Facebook.