The future of Callan Park, in the city's inner west, has been thrown into doubt, with the University of Sydney confirming its intention to vacate a group of historic sandstone buildings inside the park.

The university aims to move its Sydney College of the Arts out of the historic Kirkbride complex inside the park "gradually over the next three years", according to a university spokeswoman.

The visual arts school will be shifted on to the university's main campus in Camperdown and absorbed by another faculty. It may even merge with another tertiary arts institution, the National Art School or UNSW Art and Design (formerly the College of Fine Arts).

"We think students' experience will be improved either by consolidating activities on Camperdown campus or through collaboration with other leading visual arts institutions in Sydney," the spokeswoman said.

The SCA has "serious financial sustainability issues", exacerbated by its distance from the main campus, according to Sydney University's Strategic Plan 2016-20.

"The Rozelle campus does run at a loss and the reasons for this are varied and complex," the spokeswoman said. "It is certainly the case that distance from the Camperdown campus inhibits students attending there which undermines the sustainability of the campus."

A discussion paper released by the university says "external opportunities" for the SCA are being discussed with the NSW government, the owner of the Rozelle campus: "If these do not provide a satisfactory solution, SCA must come onto the Camperdown campus and, given its small size, would be a school in a larger faculty. In relation to administrative efficiencies and savings, this is an obvious decision."

In contrast, the discussion paper suggests retaining the Sydney Conservatorium of Music at its present site and as a separate faculty.

Friends of Callan Park president Hall Greenland said the departure of Sydney University raises the prospect of "demolition by neglect" and the sale of part of the park.

"Another danger is that the present state government, like its predecessors, clings on to dreams of making money from this heritage site and the university's removal may well prompt plans to commercialise or sell off or privatise parts of Callan Park," he said.

The SCA has around 600 students and its alumni include Archibald Prize winner Ben Quilty, designer Marc Newson​ and film director Jane Campion.

It has been at Callan Park since 1996 following a $19 million refurbishment of the Kirkbride complex, a former asylum for the mentally ill. Mr Greenland said Sydney University's presence had been important in protecting the park and the historic buildings, which date back to 1885.

The Greens member for Balmain, Jamie Parker, said the withdrawal of the Sydney College of the Arts, with its kilns and art-making facilities, would adversely affect the provision of visual arts courses.

"This move is more about saving money than it is about creating an arts centre of excellence," he said.

Callan Park is protected by legislation but Mr Parker called on the NSW government to create a trust to safeguard the 60-hectare harbourside site.

"With the university abandoning the Rozelle campus, it reinforces the need for the state government to expedite a properly funded, independent Callan Park Trust to protect the whole site from subdivision or sale in the future," he said.