A broad-daylight brawl between commentator Andrew Bolt and apparent members of the anti-fascist "Antifa" movement have raised concerns about the safety of high-profile conservatives, particularly in Melbourne.

Present at a number of wild and violent clashes with right-wing extremists and police in recent years, the Antifa group claimed members of its "family" were responsible for Tuesday's attack on Bolt at a book launch.

Alarming footage posted to social media showed a hooded man throw a substance at Bolt's head on Carlton's Lygon Street, which then escalated into a violent scuffle, knocking over chairs of an outdoor restaurant.

The News Corp columnist and Sky News host fought back fiercely, landing kicks and punches on his assailants until they disbanded, while an unknown onlooker shouted: "What are you doing?" and "Go away".

Bolt and other commentators told Fairfax Media they feared conservatives faced an increasing threat to their safety from far-left activists, especially in the Victorian capital.

"It is ridiculous how dangerous it is for conservatives in this town to speak out," Bolt said. "The right to free speech has to be better protected - everywhere but particularly in Melbourne."

Troy Whitford, history and politics lecturer at Charles Sturt University, said political violence in Melbourne was "nothing new" but it had been a long time since violence from the far-left had been as prominent.

Antifa takes its inspiration from the German Antifaschistische Aktion network that arose prior to World War II. Dr Whitford described the Australian version as a loosely-assembled "reactionary group" that seeks to combat the alternative-right, with a particular focus on defending multiculturalism, anti-racism and feminism.

"They're basically taking on that very militant approach to fostering left-wing, progressive views," he said. "If you don't subscribe to them, they kick you in the head. Anti-fascism is what they go under but they're actually practising fascist techniques."

Unlike a political party, Antifa does not have formal members or an organised hierarchy, though it may occasionally have leaders or spokespeople. Activities are usually arranged by social media.

"That's increasingly the way that a lot of younger people are campaigning," Dr Whitford said.

Antifa is "overwhelmingly" concentrated in Melbourne and Sydney, he said, with pockets in Brisbane and other places. As with extreme right groups, attendees of anti-fascist protests may wear boots, balaclavas and other face-coverings, and many try to preserve anonymity when interacting online.

In recent years, the two sides have clashed violently at many events, including a Halal food festival in Ascot Vale, protests over mosques in Melton and Bendigo and several anti-racism rallies at Federation Square.

"Neither side necessarily is peaceful," Victoria Police Assistant Commissioner Stephen Leane said last year. "There's violent protest and violent confrontation every time that they come together."

Andy Fleming, a blogger broadly connected to the Antifa movement, said he did not advocate the assault of Andrew Bolt, though he believed that was not the protesters' original intention.

He agreed with Bolt on the principle of free speech but said it was reasonable for demonstrators to heckle and disrupt public meetings and events - particularly as fringe right-wing ideas encroached further on mainstream politics.

"Which doesn't necessarily mean glitter-bombing Andrew Bolt," Mr Fleming said.

But conservatives feel they are under unjust siege. Chris Berg, research fellow at the libertarian Institute of Public Affairs and now attached to RMIT, echoed Bolt's concerns about anti-fascists increasingly using violence against mainstream conservatives.

"It is definitely true, people are thinking about it more and more. And there are more and more concerns about risks that might be involved in events," he said.

"It's remarkable how rapidly it has changed. Just a couple of years ago you wouldn't have really given this another thought. Is it safe to hold controversial events in Victoria? I think it's very distressing."

Terry Barnes, a former health adviser to Tony Abbott, was more cautious. "Melbourne's basically a tolerant place but there are eruptions," he said. "The bottom line is: attacking anyone for holding a view that's different to yours is abhorrent."

Victoria Police on Thursday confirmed it was investigating Tuesday's incident and was on the hunt for two assailants and a third man who was wielding a camera. The substance sprayed was a mixture of shaving cream and glitter, police said.

In a now-deleted Facebook post, Melbourne Antifa said "some of our family in solidarity were attacked by Andrew Bolt while they were protesting today" and said Bolt should be jailed for his "violent, horrendous language".

Bolt, a long-time critic of Antifa on his News Corp blog, is auctioning his ruined suit for charity as "a souvenir of the Lygon Street Battle for Free Speech". The highest bid of $5500 on Thursday afternoon came from Victorian Liberal Party president Michael Kroger.

Speaking to Fairfax Media on Thursday, Bolt said he was "not a brawler" but would fight back when antagonised.

"I'm sick of the threats on my life and my reputation. I'm sick of being sued and bullied and I'm not going to take it," he said.

"We must be able to discuss ideas in public without being threatened, intimidated and attacked."

Follow us on Facebook