The digital privacy of Australians ends from Tuesday, October 13.

On that day this country's entire communications industry will be turned into a surveillance and monitoring arm of at least 21 agencies of executive government.

The electronically logged data of mobile, landline voice (including missed and failed) calls and text messages, all emails, download volumes and location information will be mandatorily retained by Australian telcos and ISPs.

Intelligence and law enforcement agencies will have immediate, warrantless and accumulating access to all telephone and internet metadata required by law, with a $2 million penalty for telcos and ISPs that don't comply.

There is no sunset clause in the Abbott government's legislation, which was waved through parliament by Bill Shorten's Labor with only minor tweaks. The service providers are to keep a secret register of the agency seeking access to metadata and the identity of the persons being targeted. There is nothing in the Act to prevent investigative "fishing expeditions" or systemic abuse of power except for retrospective oversight by the Commonwealth Ombudsman. That's if you somehow found out about an agency looking into your metadata - which is unlikely, as there's a two-year jail sentence for anyone caught revealing information about instances of metadata access.

Over time, your metadata will expose your private email, SMS and fixed-line caller traffic, consumer, work and professional activities and habits, showing the patterns of all your communications, your commercial transactions and monetised subscriptions or downloads, exactly who you communicate with, and how often.

People are being asked by the Federal Parliament to accept that this regime of agency access is vitally necessary for national security at a time of geo-political tension, jihadi recruitment and the war on terror. But in a country where the biggest terrorism threat comes from lone wolves and random acts of terror, it's a system that appears singularly ill-equipped to catch terrorists. What it does is render privacy a thing of Australia's past.

Security, intelligence and law enforcement access to metadata which overrides personal privacy is now in contention worldwide.

In the US the recently passed Freedom Act constrains security agencies' access to call records not considered essential for preventing terror attacks. In Israel, facing far more immediate security concerns than Australia, there is no mandatory metadata retention law as it is not seen as a proportionate response to the security threats the country faces.

Technology analyst Nick Abrahams of law firm Norton Rose Fulbright told Fairfax Media the European Union Court of Justice had declared an EU directive invalid last year causing member states including Britain into a review and reform scramble.

But in contrast, China is expanding its data retention laws.

"[In China] there are wide powers [coming] for relevant government agencies to request information, including the right to request any encryption software used by the telcos," Abrahams said.

What is metadata good for?

It is not surprising that investigative agencies are clamouring for access to this rich new source of information. Metadata can be devastatingly effective in exposing criminality from outlaw motorcycle gangs, paedophile networks, illicit drug dealers, fraud and corruption. In NSW the Independent Commission Against Corruption used metadata to telling effect in its latest round of Labor and Liberal party corruption investigations. But it had to justify its coercive targeting through demonstrable evidentiary leads.

In 2013-2014, there were more than 330,000 requests for access to metadata, which was not always available. A spokesperson for the Attorney General's Department told Fairfax Media metadata was a vital tool used in "virtually every counter-terrorism, organised crime, counter-espionage, cyber-security, child exploitation and serious crime investigation".

But not everyone is convinced that scooping up everyone's metadata is the way to catch terrorists. Former National Security Agency analyst Thomas Drake, who preceded Edward Snowden in blowing the whistle on unconstitutional surveillance in the US, last year told a Walkley Foundation seminar in Sydney that the NSA's massive data surveillance vacuum cleaner had not exposed or thwarted any terror plots.

In Australia, a policy launched in confusion – infamously catching out Attorney General George Brandis who was unable to explain exactly what metadata was – is still crammed with contradiction and obfuscation now it's written in law. Web browsing history – the record of actual sites visited – is excluded from the metadata to be stored: a strange omission, from a national security perspective.

Communications lawyer Patrick Fair from Baker & McKenzie told Fairfax Media: "If the government wants to catch terrorists surely it would be helpful to see what sites they have been viewing. In the context of national security excluding browsing history seems churlish".

Under the new law, the industry is not required to keep details of users' web browsing history, so – if you browse pornography on the internet, for example, you may believe you can do so in lawful privacy (unless agencies obtain a warrant).

But it's not that simple. In trying to nail down the frequently asked consumer question – will my browsing history be accessible? – Fairfax Media technology editor Ben Grubb has discovered private communication from the AG's department to telcos saying that carriers will not be required to store "destination" IP addresses. However, "it does say that if 'a carrier wishes to retain those additional elements (it) is a decision for the carrier'."

A destination IP address reveals which web servers a user has accessed and is a form of web browsing history, although it cannot always show specifically what website on that server you were accessing.

"For many telcos, they will likely start storing destination IP addresses from October 13 because it will be difficult for them to remove (this data) in many cases, especially for mobile carriers due to the way their systems are designed," Grubb said.

So, the fact that you visited a porn site or infidelity site Ashley Madison or "jihadi" content sites, may in effect be discoverable without the need for a warrant.

What counts as metadata?

The collected data must be retained for two years by this country's 395 registered carriers, 230 of which are considered operationally active and hundreds of so called "carriage service providers". Compliance will apply to anyone who provides access to the internet to third parties, the exact number, no one really knows.

Australian providers of email services will be required to keep records about each email sent and received by a subscriber, but popular overseas services like Gmail, Hotmail and Yahoo are exempt.

Call information, numbers dialled, rough location, dates and times of all SMS messages sent and received by a mobile phone subscriber must be retained.

Internet service providers supplying Wi-Fi to cafes, hotels, motels, restaurants, public and private transport will have an obligation to retain data emanating from those services.

Records of all unsuccessful or untariffed communications must be retained, including 1800 calls, missed or unanswered calls, emails or VoIP (voice over internet protocol like iiNet's Nodephone) sent to a non-existent or incomplete address.

Carriers must ensure through encryption and systems protection that none of your personal information is vulnerable to unauthorised access. But the new Act also says carriers may use the data collected for lawful commercial and "troubleshooting" purposes, something many of them already do.

Data retention obligations do not apply to internet and intranet services provided within corporate and university networks unless they provide internet connections to visitors "outside their immediate circle". This has the potential to create real issues for the university sector in particular.

What about privacy law?

Privacy advocates say a review of privacy issues associated with the new regime now is needed because the systematic storage of such a massive amount of identifying information leaves an individual's privacy exposed. Short of living without a phone or computer, you have no option but to leave a digital trail relating to the last two years of your life.

The Privacy Act allows a citizen to access and correct their metadata if he/she is interested so to do. However, you will not be informed if it has been viewed by ASIO or any other agency. There is a two-year prison sentence for disclosing any information about authorised access to your data. The information being kept may also be accessed for civil litigation but only if the Attorney General creates regulations to allow it. You are not given notice and consent options for the commercial use of your metadata as you are with personal information.

Baker & McKenzie's Fair told Fairfax Media a person's metadata can be reviewed at any time by agencies without that person's knowledge and it might be used for or against you in court.

"The issue here is not so much the weaknesses in the Privacy Act but the lack of real time supervision and accountability of law enforcement and national security agencies. Our supervisory regime is weak and unlikely to ensure proper use of the extensive data soon to be kept," Fair said.

The new law does not allow any agency at unauthorised will to tap your phone, read your texts or watch you in real time as you use email, do your online banking or browse the internet. While technology now being implemented by the industry will have this invasive real-time capability, it is not lawful in Australia without a court-ordered warrant. But if the digital footprint you are creating raises suspicion after an examination of your metadata, the retained evidence may be grounds for a digital surveillance or phone tap warrant to be issued. The Telecommunications Interception Act requires all communications providers to have a real time interception capability. So it is one small (lawful) step from metadata collection to interception and continuous surveillance. For obvious operational reasons surveillance warrants issued by the courts are top secret.

Look out for the 'surveillance tax'

The industry has been scrambling to comply with the new regime. There is a massive amount of preparatory and ongoing systems work to do, and the government has still to announce how many of the 230 telcos and ISPs affected will be compliant by October 13.

Communications Alliance CEO John Stanton told Fairfax Media the industry faces an uphill battle to meet the deadlines prescribed in the Act. "We are still debating with government the practical implications of some of the requirements. There is widespread concern, particularly among smaller providers, about exactly what is required of them and which elements of specific services constitute 'content' and therefore cannot be retained", he said.

And there is a confrontation coming between the government and the industry over the cost of compliance. While Treasurer Joe Hockey's May budget stumped up $131 million to be shared by the industry to cover compliance, industry leaders say this will not cover the enormity of the task and that consumers will have to pay more for services. Inevitably consumers will call this a "surveillance tax".

"The government put a small amount of money in the budget for this and the next financial year to assist service providers with their start-up costs, but – inexplicably – still haven't provided any guidance on how that money will be apportioned and when it will be available … So far as the compliance framework goes, it's not been a stellar performance from government," said Stanton.

The AG's spokesperson said a funding model was being developed "to ensure that a fair portion of the funding is made available to smaller providers that may not have sufficient capital budgets to build new systems".

The new Communications Access Co-ordinator, Ms Jamie Lowe, was not authorised to speak to Fairfax Media.

Because of the complexity of compliance, a new "industry" is being created with many registered service providers seeking to outsource their compliance obligations to specialists approved by the CAC. This necessarily will broaden the entities with access to your metadata. Again Australian subscribers will be left to trust in the integrity of not just the government agencies but the outsourcers who will have access to their metadata.

Quentin Dempster is a contributing editor.