Wayne Swan lambasted National Australia Bank for being greedy, and accused them of kicking working families in the guts.
The bank became the only bank not to pass on the Reserve Bank's full 25 basis point rate cut.
But customers at NAB city branches said they were still getting a good deal.
Customer Michael Topalov said at their Hunter Street branch that he doubted people would switch to another bank.
"It is too hard... for such a small amount," he said.
"I think [NAB] had a pretty good rate in the first place. They are just adjusting what they've done before."
Darren Crisp, an ex-banker, said at the same branch that the move by NAB would have minimal impact.
"Initially they'll get bad publicity, but they'll bounce back," he said.
Chris Golis said he blamed the government for trying to cause controversy over 5 basis points.
"They have the cheapest rates going at the moment. It is typical of this government and the cheap tricks," he said.
Hugh Macrae at the bank's Pitt street branch said he thought customers would stay with the bank, but would pay extra attention to the next time the Reserve Bank cuts the interest rate cut.
"If they do it again – then people might start leaving," he said.
Otto Pirko on the other hand, said he was worried about how the bank's decision would affect his finances.
"I might search for a better rate," he said.
"I'm not happy."
Since the federal government banned mortgage exit fees on July 1, 167,500 people have taken out a home loan.
All these people are eligible to switch banks without exit fees, yet it is unclear how many of these are NAB customers.
An NAB spokeswoman said the campaign to disassociate with other banks - which included the "breaking up" ads - will continue indefinitely.