Big Jim Byrnes is so upset about his recent bad press he is talking about cashing up and becoming a stay-at-home dad.

"I just don't need the aggravation," he said. "We might sell down our $20 million in assets" and "I'll put my feet up", he said yesterday.

According to Mr Byrnes, his recent bad publicity is making him physically sick. It includes the Australian Securities and Investment Commission banning him from running corporations for five years, his auction house going down the gurgler and claims made in a Victorian court that he threatened to harm someone's children.

Before he pursues a quieter life, Mr Byrnes wanted to set the record straight.

A road contractor told a Victorian court recently that Mr Byrnes had produced photographs of his young children and threatened to harm them unless a business agreement was reached.

"Anybody who would threaten or do anything to anybody's child is the most vile creature on the planet," Mr Byrnes said.

While he admitted he did pay a visit to the Melbourne man, "our meeting started off - I will say, frosty", but he said it improved to the extent that the road contractor later gave him permission to negotiate the deal on his behalf.

Mr Byrnes said he would never threaten anyone. "I abhor violence," he said. Asked about his conviction in 2003 for assaulting a bikie, Mr Byrnes said that was different because "I'm an alpha male" who was defending his family.

"I was driving down Ocean Street, Woollahra - actually my wife was driving - and some maniac on a Jap bike who was doing about 150km/h then stands on the brakes."

According to Mr Byrnes, the lunatic bikie started banging on his car window and frightening his children.

"What can I say? I got out and he threatened to kick the side of the car and do all sorts of things. We had a scuffle. He lost.

"He wasn't very much of a bikie because he went to the police and laid a complaint for common assault. I didn't defend it. I had some other issue on at the time. I couldn't be bothered, to be honest."

He received a $400 fine.

His habitual offender record on his driver's licence is baloney as well, he said. He was sure that had been removed from his record at a local court hearing.

Anyway, having lost his licence several times, especially after being caught driving without a licence, Mr Byrnes has changed his ways.

"I haven't even had a parking ticket," said the man who once owed almost $3000 in parking fines. "Actually, I'll rephrase that: I get two parking tickets a week."

The good news is that Mr Byrnes has not had "a speeding ticket since I got my licence back three or four years ago". He has only lost points for chatting on his mobile while driving, he said.

As for his recent five-year ban by the Australian Securities and Investment Commission for incompetence and lack of corporate morality, an appeal has been lodged with the Administrative Appeals Tribunal.

The ban, he claims, was largely due to the commission's incompetence, not his own. What annoys Mr Byrnes most is reference to his stint in jail. "It was during the '80s," he said, "when a lot of people were in jail."

"And for the record, it was 3.9 grams [of heroin], which is about the equivalent of a level teaspoon. It doesn't excuse it, but it doesn't make me Pablo Escobar either."

"I'm actually a big teddy bear," Mr Byrnes told the Herald.

"You will find it very difficult to find people to say that I have spoken in any way other than in a polite and pleasant manner."

UPDATE: The original version of this article was edited after a decision to remove the name of the road contractor under Sydney Morning Herald editorial policies.