US spy agencies have built an intelligence-gathering colossus since the attacks of September 11, 2001, but remain unable to provide critical information on a range of national security threats, according to the government's top secret budget.

The $US52.6 billion ($A59 billion) "black budget" for fiscal 2013, obtained by the Washington Post from former intelligence contractor Edward Snowden, maps a bureaucratic and operational landscape that has never been subject to public scrutiny. Although the US government has released its overall level of intelligence spending annually since 2007, it has not divulged how it uses those funds or how it performs against the goals set by the US President and Congress.

The 178-page budget summary for the National Intelligence Program details the successes, failures and objectives of the 16 spy agencies that make up the US intelligence community, which has 107,035 employees.

Former intelligence contractor Edward Snowden.

Former intelligence contractor Edward Snowden. Photo: Reuters

The summary describes cutting-edge technologies, agent recruiting and ongoing operations. The Washington Post is withholding some information after consultation with US officials who expressed concerns about the risk to intelligence sources and methods. Sensitive details are so pervasive in the documents that the Post is publishing only summary tables and charts online.

"The United States has made a considerable investment in the Intelligence Community since the terror attacks of 9/11, a time which includes wars in Iraq and Afghanistan, the Arab Spring, the proliferation of weapons of mass destruction technology, and asymmetric threats in such areas as cyber warfare," Director of National Intelligence James Clapper said.

"Our budgets are classified as they could provide insight for foreign intelligence services to discern our top national priorities, capabilities and sources and methods that allow us to obtain information to counter threats," he said.

The Capitol building, which serves as the seat of government for US Congress.

The Capitol building, which serves as the seat of government for US Congress. Photo: AFP

Among the notable revelations in the budget summary:

  • Spending by the CIA has surged past that of every other spy agency, with $US14.7 billion ($A16.5 billion) in requested funding for 2013. The figure vastly exceeds outside estimates and is nearly 50 per cent above that of the National Security Agency (NSA), which conducts eavesdropping operations and has long been considered the behemoth of the community.
  • The CIA and NSA have launched aggressive new efforts to hack into foreign computer networks to steal information or sabotage enemy systems, embracing what the budget refers to as "offensive cyber operations".
  • The NSA planned to investigate at least 4000 possible insider threats in 2013, cases in which the agency suspected sensitive information may have been compromised by one of its own. The budget documents show that the US intelligence community has sought to strengthen its ability to detect what it calls "anomalous behaviour" by personnel with access to highly classified material.
  • US intelligence officials take an active interest in foes as well as friends. Pakistan is described in detail as an "intractable target" and counter-intelligence operations "are strategically focused against [the] priority targets of China, Russia, Iran, Cuba and Israel".
  • In words, deeds and dollars, intelligence agencies remain fixed on terrorism as the gravest threat to US national security, which is listed first among five "mission objectives". Counterterrorism programs employ one in four members of the intelligence workforce and account for one-third of all spending.
  • The governments of Iran, China and Russia are difficult to penetrate, but North Korea's may be the most opaque. There are five "critical" gaps in US intelligence about Pyongyang's nuclear and missile programs, and analysts know virtually nothing about the intentions of North Korean leader Kim Jong Un.

Formally known as the Congressional Budget Justification for the National Intelligence Program, the "Top Secret" blueprint represents spending levels proposed to the House and Senate intelligence committees in February 2012. Congress may have made changes before the fiscal year began on Oct 1. Clapper is expected to release the actual total spending figure after the US fiscal year ends on September 30.

The document describes a constellation of spy agencies that track millions of individual surveillance targets and carry out operations that include hundreds of lethal strikes. They are organised around five priorities: combating terrorism, stopping the spread of nuclear and other unconventional weapons, warning US leaders about critical events overseas, defending against foreign espionage and conducting cyber operations.

The CIA headquarters in Langley, Virginia.

The CIA headquarters in Langley, Virginia. Photo: AFP

In an introduction to the summary, Clapper said the threats now facing the US "virtually defy rank-ordering". He warned of "hard choices" as the intelligence community – sometimes referred to as the "IC" – seeks to rein in spending after a decade of often double-digit budget increases.

This year's budget proposal envisions that spending will remain roughly level through 2017 and amounts to a case against substantial cuts.

"Never before has the IC been called upon to master such complexity and so many issues in such a resource-constrained environment," Clapper wrote.

The NSA campus in Fort Meade, Maryland.

The NSA campus in Fort Meade, Maryland. Photo: AP

The summary provides a detailed look at how the US intelligence community has been reconfigured by the massive infusion of resources that followed the September 11 attacks. The US has spent more than $US500 billion ($A560 billion) on intelligence during that period, an outlay that US officials say has succeeded in its main objective: preventing another catastrophic terrorist attack inside the US.

The result is an espionage empire with resources and reach beyond those of any adversary, sustained even now by spending that rivals or exceeds the levels reached at the height of the Cold War.

This year's total budget request was 2.4 per cent below that of fiscal 2012. In constant dollars, it was roughly twice the estimated size of the 2001 budget and 25 per cent above that of 2006, five years into what was then known as the "global war on terror".

Historical data on US intelligence spending is largely non-existent. Through extrapolation, experts have estimated that Cold War spending likely peaked in the late 1980s at an amount that would be the equivalent of $US71 billion ($A79.5 billion) today.

Spending in the most recent cycle surpassed that amount based on the $US52.6 billion ($A59 billion) detailed in documents obtained by the Post, plus a separate $US23 billion ($A25.7 billion) devoted to intelligence programs that more directly support the US military.

Lee Hamilton, an Indiana Democrat who was a former chairman of the US House Intelligence Committee and co-chairman of the commission that investigated the September 11 attacks, said access to budget figures has the potential to enable an informed public debate on intelligence spending for the first time, much as Snowden's disclosures of NSA surveillance programs brought attention to operations that had assembled data on nearly every US citizen as well as many abroad.

"Much of the work that the intelligence community does has a profound impact on the life of ordinary Americans, and they ought not to be excluded from the process," he said.

"Nobody is arguing that we should be so transparent as to create dangers for the country," he said. But, he said, "there is a mindset in the national security community – leave it to us, we can handle it, the American people have to trust us. They carry it to quite an extraordinary length so that they have resisted over a period of decades transparency ... The burden of persuasion as to keeping something secret should be on the intelligence community, the burden should not be on the American public."

Experts said access to such details on US spy programs is without precedent.

"It was a titanic struggle just to get the top-line budget number disclosed, and that has only been done consistently since 2007," said Steven Aftergood, an expert at the Federation of American Scientists, a Washington organisation that provides analyses of national security issues. "But a real grasp of the structure and operations of the intelligence bureaucracy has been totally beyond public reach. This kind of material, even on a historical basis, has simply not been available."

The only meaningful frame of reference came in 1994, when a US congressional subcommittee inadvertently published a partial breakdown of the National Intelligence Program. At the time, the CIA accounted for just $US4.8 billion ($A5.4 billion) of a budget that totalled $US43.4 billion ($A48.6 billion) in 2012 dollars. The NSA and the National Reconnaissance Office, which operates satellites and other sensors, commanded far larger shares of US intelligence budgets until years after the end of the Cold War.

During the past decade, they have taken a back seat to the CIA.

NSA was in line to receive $US10.5 billion ($A11.8 billion) in 2013, and the NRO was to get $US10.3 billion ($A11.5 billion) – both far below the CIA, whose share had surged to 28 per cent of the total budget.

Overall, the US government spends 10 times as much on Department of Defence as it does on spy agencies.

"Today's world is as fluid and unstable as it has been in the past half century," Clapper said. "Even with stepped up spending on the IC over the past decade, the United States currently spends less than 1 per cent of GDP on the Intelligence Community."

The CIA's dominant position will likely stun outside experts. It represents a remarkable recovery for an agency that seemed poised to lose power and prestige after acknowledging intelligence failures leading up to the September 11, 2001, attacks and the 2003 US-led invasion of Iraq.

The surge in resources for the agency funded secret prisons, a controversial interrogation program, the deployment of lethal drones and a huge expansion of its counterterrorism centre. The agency was transformed from a spy service struggling to emerge from the Cold War into a paramilitary force.

The CIA has devoted billions of dollars to recruiting and training a new generation of case officers, with the workforce growing from about 17,000 a decade ago to 21,575 this year.

The agency's budget allocates $US2.3 billion ($A2.6 billion) for human intelligence operations, and another $US2.5 billion ($A2.8 billion) to cover the cost of supporting the security, logistics and other needs of those missions around the world. A relatively small amount of that total, $US68.6 million ($A76.8 million), was earmarked for creating and maintaining "cover", the false identities employed by operatives overseas.

There is no specific entry for the CIA's fleet of armed drones in the budget summary, but a broad line item hints at the dimensions of the agency's expanded paramilitary role, providing more than $US2.6 billion ($A2.9 billion) for "covert action programs" that would include drone operations in Pakistan and Yemen, payments to militias in Afghanistan and Africa, and attempts to sabotage Iran's nuclear program.

The black budget illuminates for the first time the intelligence burden of the wars in Afghanistan and Iraq. For 2013, US spy agencies were projected to spend $US4.9 billion ($A5.5 billion) on what are labelled "overseas contingency operations". The CIA accounted for roughly half of that figure, a sum factored into its overall $US14.7 billion ($A16.5 billion) budget.

Those war expenditures are projected to shrink as the US withdraws forces from Afghanistan. The budget also indicates the intelligence community has cut the number of contractors it hires over the past five years by roughly 30 per cent.

Despite the vast outlays, the budget blueprint catalogues persistent and in some cases critical blind spots.

Throughout the document, US spy agencies attempt to rate their efforts in tables akin to report cards, generally citing progress but often acknowledging that only a fraction of their questions could be answered non-traditional even on the community's foremost priority, counterterrorism.

In 2011, the budget assessment says intelligence agencies made at least "moderate progress" on 38 of their 50 top counterterrorism gaps, the term used to describe blind spots. Several concern Lebanon's Hezbollah movement, an enemy of Israel that has not attacked US interests directly since the 1990s.

Other blank spots include questions about the security of Pakistan's nuclear components when they are being transported, the capabilities of China's next generation fighter aircraft, and how Russia's government leaders are likely to respond "to potentially destabilising events in Moscow, such as large protests and terrorist attacks".

A chart outlining efforts to address key questions on biological and chemical weapons is particularly bleak. US agencies set themselves annual goals of making progress in at least five categories of intelligence collection related to these weapons. In 2011, the agencies made headway on just two gaps; a year earlier the mark was zero.

The documents describe expanded efforts to "collect on Russian chemical warfare countermeasures" and assess the security of biological and chemical laboratories in Pakistan.

A table of "critical" gaps listed five for North Korea, more than for any other country that has or is pursuing a nuclear bomb.

The intelligence community seems particularly daunted by the emergence of "home grown" terrorists who plan attacks in the US without direct support or instruction from abroad, a threat realised this year, after the budget was submitted, in twin bombings at the Boston Marathon.

The National Counterterrorism Centre has convened dozens of analysts from other agencies in attempts to identify "indicators" that could help law enforcement understand the path from religious extremism to violence. The FBI was in line for funding to increase the number of agents surreptitiously tracking activity on jihadist websites.

But a year before the bombings in Boston the search for meaningful insight into the stages of radicalisation was described as one of "the more challenging intelligence gaps".

The documents make clear that US spy agencies' long-standing reliance on technology remains intact. If anything, their dependence on high-tech surveillance systems to fill gaps in human intelligence has only intensified.

A section on North Korea indicates the US has all but surrounded the nuclear-armed country with surveillance platforms. There are distant ground sensors to monitor seismic activity and platforms to scan the country for signs that might point to construction of new nuclear sites. US agencies seek to capture photos, air samples and infrared imagery "around the clock".

In Iran, new surveillance techniques and technologies have enabled analysts to identify suspected nuclear sites that had not been detected in satellite images, according to the document.

In Syria, NSA listening posts were able to monitor unencrypted communications among senior military officials at the outset of the civil war there, a vulnerability that President Bashar al-Assad's forces apparently later recognised. One of the NRO's functions is to extract data from sensors placed on the ground near suspected illicit weapons sites in Syria and other countries.

Across this catalogue of technical prowess, one category is depicted as particularly indispensable: signals intelligence, or SIGINT.

The NSA's ability to monitor emails, phone calls and internet traffic has come under new scrutiny in recent months as a result of disclosures by Snowden, who worked as a contract computer specialist for the agency before stockpiling secret document and then fleeing, first to Hong Kong and then Moscow.

The NSA was projected to spend $US48.6 million ($A54.4 million) on research projects to assist "coping with information overload", an occupational hazard as the volumes of intake have increased sharply from fibre optic cables and Silicon Valley internet providers.

The agency's ability to monitor the communications of al-Qaeda operatives is described in the documents as "often the best and only means to compromise seemingly intractable targets".

Signals intercepts have also been used to direct the flight paths of drones, gather clues to the composition of North Korea's leadership and evaluate the response plans of Russia's government in the event of a terrorist attack in Moscow.

The resources devoted to signals stealing are staggering.

Nearly 35,000 employees are listed under a sweeping category called the Consolidated Cryptologic Program, which includes the NSA as well as the surveillance and code-breaking components of the US Air Force, Army, Navy and Marines.

Even the CIA devotes $US1.7 billion ($A1.9 billion), or nearly 12 per cent of its budget, to technical collection efforts including a program called "CLANSIG" that former officials said is the agency's more targeted version of the massive data collection operations of the NSA.

The CIA is pursuing tracking systems "that minimise or eliminate the need for physical access and enable deep concealment operations against hard targets".

The agency has deployed new biometric sensors to confirm the identities and locations of al-Qaeda operatives. The system has been used in the CIA's drone campaign.

The NSA is also planning high-risk covert missions, a lesser-known part of its work, to plant what it calls "tailored radio frequency solutions" in hostile territory non-traditional close-in sensors to intercept communications that do not pass through global networks.

Spending on satellite systems and almost every other category of collection are projected to remain stagnant or shrink in coming years, as Washington grapples with budget cuts across the government. But the 2013 intelligence budget called for increased investment in SIGINT.

The budget includes a lengthy section on funding for counter-intelligence programs designed to protect against the danger posed by foreign intelligence services as well as betrayals from within the US spy ranks.

The document describes programs to "mitigate insider threats by trusted insiders who seek to exploit their authorised access to sensitive information to harm US interests".

The agencies had budgeted for a major counter-intelligence initiative in fiscal 2012, but most of those resources were diverted to an all-hands, emergency response to successive floods of classified data released by the anti-secrecy group WikiLeaks.

For this year, the budget promised a renewed "focus ... on safeguarding classified networks" and a strict "review of high-risk, high-gain applicants and contractors" – the young, non-traditional computer coders with the skills the NSA needed.

Among them was Snowden, then a 29-year-old contract computer specialist who had been trained by the NSA to circumvent computer network security. He was copying thousands of highly classified documents at an NSA facility in Hawaii, and preparing to leak them, as the agency embarked on a security sweep.

"NSA will initiate a minimum of 4000 periodic reinvestigations of potential insider compromise of sensitive information," according to the budget, scanning its systems for "anomalies and alerts".

Washington Post