It is believed at least five two-month-old babies, their siblings and parents were given no notice as they were forced to leave Adelaide's Inverbrackie detention centre at 3am last week, without access to any legal advice.

The sudden relocations come as Immigration Minister Scott Morrison announced a $2.6 million educational package for school-aged children in the island's detention centre, run by Catholic Education Office of Western Australia, suggesting that the government is planning to increase the population of child detainees on the island.

Jacob Varghese, who is representing 26 Australian-born asylum seeker babies, said the families were living in daily fear that they would be "shipped off" to Christmas Island.

"There is really heavy-handed and unnecessarily cruel approach taken to removing people, which is knocking on their door in the middle of the night and shipping them off," Mr Varghese, a principal of Maurice Blackburn lawyers, said.

"Christmas Island is the worst place in Australia to put these people, because it is very remote and a long way from any first-class medical services," he said.

Mr Varghese said the families also had children under four years old.

The new educational arrangement on the island will provide full-time educational services from kindergarten to high school for the 56 children living in the centre, it was announced on Friday evening.

But the government is expecting the centre will enroll "to up 150" children - nearly three times the number of children currently on the island.

The increased number of children is likely to come from the nine onshore detention centres that the government will close over four years, including Inverbrackie in Adelaide and Darwin Airport Lodge, which both house family groups.

The Royal Australasian College of Physicians said that while it "cautiously welcomed" the plans to provide full-time education to the children, it slammed Australia's decision to detain children in such harsh environments.

"We are not fulfilling our international obligations by detaining children in any setting in terms of their human rights," the college's president, Professor Nick Talley, said.

A Human Rights Commission inquiry into the treatment of asylum seeker children on Christmas Island detention centre in March found children in a state of gross neglect, with little to no access to education.

Mr Morrison confirmed that 20 asylum seekers in family groups had been transferred from the Adelaide centre last Wednesday.

''The detainees had been transferred to Inverbrackie from Christmas Island for medical reasons,'' a spokeswoman for Mr Morrison said.

''The government does not disclose medical details of detainees for privacy reasons.

''The detainees were notified in advance of the transfer and were not prevented from contacting family, friends or others prior to their departure."

If a baby is currently born to asylum seeker parents in detention they are considered "unlawful maritime arrivals" by the government.

But this could change following the decision of a test court case in Brisbane where lawyers are fighting to release an Australian-born baby Ferouz from detention.

Ferouz was born in Brisbane on November 6 after his mother, Latifar, a 31-year-old Rohingyan asylum seeker, was flown from the Nauru detention centre to the Mater Hospital following pregnancy complications.

Until the decision is made whether babies can be considered UMAs, the Immigration Department have provided an undertaking not to send 26 Australian-born babies or their immediate family members to offshore detention.

The Ferouz case continues on Monday.