Engineering:Signals intelligence in modern history

Before the development of radar and other electronics techniques, signals intelligence (SIGINT) and communications intelligence (COMINT) were essentially synonymous. Sir Francis Walsingham ran a postal interception bureau with some cryptanalytic capability during the reign of Elizabeth I, but the technology was only slightly less advanced than men with shotguns, during World War I, who jammed pigeon post communications and intercepted the messages carried.

Flag signals were sometimes intercepted, and efforts to impede them made the occupation of the signaller one of the most dangerous on the battlefield. The middle 19th century rise of the telegraph allowed more scope for interception and spoofing of signals, as shown at Chancellorsville.

Signals intelligence became far more central to military (and to some extent diplomatic) intelligence generally with the mechanization of armies, development of blitzkrieg tactics, use of submarine and commerce raiders warfare, and the development of practicable radio communications. Even measurement and signature intelligence (MASINT) preceded electronic intelligence (ELINT), with sound ranging techniques for artillery location. SIGINT is the analysis of intentional signals for both communications and non-communications (e.g., radar) systems, while MASINT is the analysis of unintentional information, including, but not limited to, the electromagnetic signals that are the main interest in SIGINT.

Origins

HMS Diana carried out one of the first signals interceptions in 1904, when it picked up the order for the mobilization of the Russian fleet at the beginning of the Russo-Japanese War.

Electronic interception appeared as early as 1900, during the Boer Wars. The Royal Navy had installed wireless sets produced by Marconi on board their ships in the late 1890s and some limited wireless signalling was used by the British Army. Some wireless sets were captured by the Boers, and were used to make vital transmissions. Since the British were the only people transmitting at the time, no special interpretation of the signals was necessary.

The Imperial Russian Navy also experimented with wireless communications under the guidance of Alexander Popov, who first installed a wireless set on a grounded battleship in 1900. The birth of signals intelligence in a modern sense dates to the Russo-Japanese War.

As the Russian fleet prepared for conflict with Japan in 1904, the British ship HMS Diana stationed in the Suez canal was able to intercept Russian naval wireless signals being sent out for the mobilization of the fleet, for the first time in history. "An intelligence report on signals intercepted by HMS Diana at Suez shows that the rate of working was extremely slow by British standards, while the Royal Navy interpreters were particularly critical of the poor standard of grammar and spelling among the Russian operators".

The Japanese also developed a wireless interception capability and succeeded in listening in to the then primitive Russian communications. Their successes emphasized the importance of this new source of military intelligence, and facilities for the exploitation of this information resource were established by all the major powers in the following years.

The Austro-Hungarian Evidenzbureau was able to comprehensively monitor the progress of the Italian army during the Italo-Turkish War of 1911 by monitoring the signals that were sent by a series of relay stations from Tripoli to Rome. In France, Deuxième Bureau of the Military General Staff was tasked with radio interception.

World War I

It was over the course of the War that the new method of intelligence collection - signals intelligence - reached maturity. The British in particular built up great expertise in the newly emerging field of signals intelligence and codebreaking.

Failure to properly protect its communications fatally compromised the Russian Army in its advance early in World War I and led to their disastrous defeat by the Germans under Ludendorff and Hindenburg at the Battle of Tannenberg.

France had significant signals intelligence in World War I. Commandant Cartier developed a system of wireless masts, including one on the Eiffel Tower to intercept German communications. The first such station was built as early as 1908, although was destroyed by flooding a few years afterward. In the early stages of the war, French intercepts were invaluable for military planning and provided the crucial intelligence to commander-in-chief Joseph Joffre that enabled him to carry out the successful counterattack against the Germans at the Marne in September 1914.

In 1918, French intercept personnel captured a message written in the new ADFGVX cipher, which was cryptanalyzed by Georges Painvin. This gave the Allies advance warning of the German 1918 Spring Offensive.

US communications monitoring of naval signals started in 1918, but was used first as an aid to naval and merchant navigation. In October 1918, just before the end of the war, the US Navy installed its first DF installation at its station at Bar Harbor, Maine, soon joined by five other Atlantic coast stations, and then a second group of 14 installations. These stations, after the end of World War I, were not used immediately for intelligence. While there were 52 Navy medium wave (MF) DF stations in 1924, most of them had deteriorated.

Cracking the German naval codes

The Admiralty Ripley building. Room 40 succeeded in intercepting and decrypting German naval signals.

By the start of the First World War, a worldwide commercial undersea communication cable network had been built up over the previous half-century, allowing nations to transmit information and instructions around the world. Techniques for intercepting these messages through ground returns were developed, so all cables running through hostile territory could in theory be intercepted.

On the declaration of war, one of Britain's first acts was to cut all German undersea cables. On the night of 3 August 1914, the cable ship Alert located and cut Germany's five trans-Atlantic cables, which ran down the English Channel. Soon after, the six cables running between Britain and Germany were cut. This forced the Germans to use either a telegraph line that connected through the British network and could be tapped, or through radio which the British could then intercept. The destruction of more secure wired communications, to improve the intelligence take, has been a regular practice since then. While one side may be able to jam the other's radio communications, the intelligence value of poorly secured radio may be so high that there is a deliberate decision not to interfere with enemy transmissions.

Although Britain could now intercept German communications, codes and ciphers were used to hide the meaning of the messages. Neither Britain nor Germany had any established organisations to decode and interpret the messages at the start of the war - the Royal Navy had only one wireless station for intercepting messages, at Stockton-on-Tees.

However, installations belonging to the Post Office and the Marconi Company, as well as private individuals who had access to radio equipment, began recording messages from Germany. Realizing that the strange signals they were receiving were German naval communications, they brought them to the Admiralty. Rear-Admiral Henry Oliver appointed Sir Alfred Ewing to establish an interception and decryption service. Among its early recruits were Alastair Denniston, Frank Adcock, John Beazley, Francis Birch, Walter Horace Bruford, William Nobby Clarke, Frank Cyril Tiarks and Dilly Knox. In early November 1914 Captain William Hall was appointed as the new Director of the Intelligence division to replace Oliver.

SMS Magdeburg ran aground off Odensholm and provided Room 40 with the German SKM codebook.

A similar organisation had begun in the Military Intelligence department of the War Office, which become known as MI1b, and Colonel Macdonagh proposed that the two organisations should work together. Little success was achieved except to organise a system for collecting and filing messages until the French obtained copies of German military ciphers. The two organisations operated in parallel, decoding messages concerning the Western Front. A friend of Ewing's, a barrister by the name of Russell Clarke, plus a friend of his, Colonel Hippisley, approached Ewing to explain that they had been intercepting German messages. Ewing arranged for them to operate from the coastguard station at Hunstanton in Norfolk. They formed the core of the interception service known as 'Y' service, together with the post office and Marconi stations, which grew rapidly to the point it could intercept almost all official German messages.

In a stroke of luck, the SKM codebook was obtained from the German Light cruiser Magdeburg, which ran aground on the island of Odensholm off the coast of Russian-controlled Estonia. The books were formally handed over to the First Lord, Winston Churchill, on 13 October. The SKM by itself was incomplete as a means of decoding messages since they were normally enciphered as well as coded, and those that could be understood were mostly weather reports. An entry into solving the problem was found from a series of messages transmitted from the German Norddeich transmitter, which were all numbered sequentially and then re-enciphered. The cipher was broken, in fact broken twice as it was changed a few days after it was first solved, and a general procedure for interpreting the messages determined.

A second important code - the Handelsverkehrsbuch (HVB) codebook used by the German navy - was captured at the very start of the war from the German-Australian steamer Hobart, seized off Port Philip Heads near Melbourne on 11 August 1914. The code was used particularly by light forces such as patrol boats, and for routine matters such as leaving and entering harbour. The code was used by U-boats, but with a more complex key. A third codebook was recovered following the sinking of German destroyer SMS S119 in a battle off Texel island. It contained a copy of the Verkehrsbuch (VB) codebook, intended for use in cables sent overseas to warships and naval attachés, embassies and consulates. Its greatest importance during the war was that it allowed access to communications between naval attachés in Berlin, Madrid, Washington, Buenos Aires, Peking, and Constantinople.

The German fleet was in the habit each day of wirelessing the exact position of each ship and giving regular position reports when at sea. It was possible to build up a precise picture of the normal operation of the High Seas Fleet, indeed to infer from the routes they chose where defensive minefields had been place and where it was safe for ships to operate. Whenever a change to the normal pattern was seen, it immediately signalled that some operation was about to take place and a warning could be given. Detailed information about submarine movements was also available.

Direction finding

Captain H.J. Round devised and oversaw the construction of an array of direction finding stations that were able to track the movements of enemy ships.

The use of radio receiving equipment to pinpoint the location of the transmitter was also developed during the war. Captain H.J. Round working for Marconi, began carrying out experiments with direction finding radio equipment for the army in France in 1915. Hall instructed him to build a direction finding system for the navy. This was sited at Lowestoft and other stations were built at Lerwick, Aberdeen, York, Flamborough Head and Birchington and by May 1915 the Admiralty was able to track German submarines crossing the North Sea. Some of these stations also acted as 'Y' stations to collect German messages, but a new section was created within Room 40 to plot the positions of ships from the directional reports.

Room 40 had very accurate information on the positions of German ships, but the Admiralty priority remained to keep the existence of this knowledge secret. From June 1915 the regular intelligence reports of ship positions ceased to be passed to all flag officers, but only to Admiral Jellicoe himself. Similarly, he was the only person to receive accurate charts of German minefields prepared from Room 40 information. No attempts were made by the German fleet to restrict its use of wireless until 1917, and then only in response to perceived British use of direction finding, not because it believed messages were being decoded.

It became increasingly clear, that as important as the decrypts were, it was of equal importance to accurately analyse the information provided. An illustration of this was provided by someone at the Admiralty who knew a little too much detail about SIGINT without fully understanding it. He asked the analysts where call sign "DK" was located, which was that used by the German commander when in harbour. The analysts answered his question precisely, telling him that it was "in the Jade River". Unfortunately the High Seas Fleet commander used a different identifier when at sea, going so far as to transfer the same wireless operator ashore so the messages from the harbour would sound the same. The misinformation was passed to Jellicoe commanding the British fleet, who acted accordingly and proceeded at a slower speed to preserve fuel. The battle of Jutland was eventually fought but its lateness in the day allowed the enemy to escape.

Jellicoe's faith in cryptographic intelligence was also shaken by a decrypted report that placed the German cruiser SMS Regensburg near him, during the Battle of Jutland. It turned out that the navigator on the Ravensburg was off by 10 miles (16 km) in his position calculation. During Jutland, there was limited use of direction finding on fleet vessels, but most information came from shore stations. A whole string of messages were intercepted during the night indicating with high reliability how the German fleet intended to make good its escape, but the brief summary which was passed to Jellicoe failed to convince him of its accuracy in light of the other failures during the day.

Zimmermann Telegram & Other Successes

Zimmermann telegram as decoded by Room 40 in 1917.

Room 40 played an important role in several naval engagements during the war, notably in detecting major German sorties into the North Sea. The battle of Dogger Bank was won in no small part due to the intercepts that allowed the Navy to position its ships in the right place. "Warned of a new German raid [on England] on the night of 23–24 January, by radio intercepts, [Admiral Sir David] Beatty’s force made a rendezvous off the Dogger Bank... The outnumbered Germans turned in flight. ... the Kaiser, fearful of losing capital ships, ordered his navy to avoid all further risks."

It played a vital role in subsequent naval clashes, including at the Battle of Jutland as the British fleet was sent out to intercept them. The direction-finding capability allowed for the tracking and location of German ships, submarines and Zeppelins. Intercepts were also able to prove beyond doubt that the German high command had authorized the sinking of the Lusitania in May 1915, despite the vociferous German denials at the time. The system was so successful, that by the end of the war over 80 million words, comprising the totality of German wireless transmission over the course of the war had been intercepted by the operators of the Y-stations and decrypted. However its most astonishing success was in decrypting the Zimmermann Telegram, a telegram from the German Foreign Office sent via Washington to its ambassador Heinrich von Eckardt in Mexico.

In the telegram's plaintext, Nigel de Grey and William Montgomery learned of the German Foreign Minister Arthur Zimmermann's offer to Mexico of United States ' territories of Arizona, New Mexico, and Texas as an enticement to join the war as a German ally. The telegram was passed to the U.S. by Captain Hall, and a scheme was devised (involving a still unknown agent in Mexico and a burglary) to conceal how its plaintext had become available and also how the U.S. had gained possession of a copy. The telegram was made public by the United States, which declared war on Germany on 6 April 1917, entering the war on the Allied side.

Interwar period

With the importance of interception and decryption firmly established by the wartime experience, countries established permanent agencies dedicated to this task in the interwar period.

These agencies carried out substantial SIGINT work between the World Wars, although the secrecy surrounding it was extreme. While the work carried out was primarily COMINT, ELINT also emerged, with the development of radar in the 1930s.

United Kingdom

In 1919, the British Cabinet's Secret Service Committee, chaired by Lord Curzon, recommended that a peace-time codebreaking agency should be created, a task given to the then-Director of Naval Intelligence, Hugh Sinclair. Sinclair merged staff from the British Army's MI1b and Royal Navy's Room 40 into the first peace-time codebreaking agency: the Government Code and Cypher School (GC&CS). The organization initially consisted of around 25–30 officers and a similar number of clerical staff. It was titled the "Government Code and Cypher School", a cover-name chosen by Victor Forbes of the Foreign Office.

Alastair Denniston, who had been a leading member of Room 40, was appointed as its operational head. It was initially under the control of the Admiralty, and located in Watergate House, Adelphi, London. Its public function was "to advise as to the security of codes and cyphers used by all Government departments and to assist in their provision", but also had a secret directive to "study the methods of cypher communications used by foreign powers". GC&CS officially formed on 1 November 1919, and produced its first decrypt on 19 October.

By 1922, the main focus of GC&CS was on diplomatic traffic, with "no service traffic ever worth circulating" and so, at the initiative of Lord Curzon, it was transferred from the Admiralty to the Foreign Office. GC&CS came under the supervision of Hugh Sinclair, who by 1923 was both the Chief of SIS and Director of GC&CS. In 1925, both organisations were co-located on different floors of Broadway Buildings, opposite St. James's Park. Messages decrypted by GC&CS were distributed in blue-jacketed files that became known as "BJs".

In the 1920s, GC&CS was successfully reading Soviet Union diplomatic ciphers. However, in May 1927, during a row over clandestine Soviet support for the General Strike and the distribution of subversive propaganda, Prime Minister Stanley Baldwin made details from the decrypts public.

By 1940, GC&CS was working on the diplomatic codes and ciphers of 26 countries, tackling over 150 diplomatic cryptosystems.

Germany

From the mid-twenties, German Military Intelligence Abwehr began intercepting and cryptanalyzing diplomatic traffic. Under Hermann Göring, the Nazi Research Bureau (Forschungsamt or "FA") had units for intercepting domestic and international communications. The FA was penetrated by a French spy in the 1930s, but the traffic grew to a point that it could not easily be forwarded.

In addition to intercept stations in Germany, the FA established an intercept station in Berne, Switzerland. German code breaking penetrated most cryptosystems, other than the UK and US. German Condor Legion personnel in the Spanish Civil War ran COMINT against their opponents.

United States

The US Cipher Bureau was established in 1919 and achieved some success at the Washington Naval Conference in 1921, through cryptanalysis by Herbert Yardley. Secretary of War Henry L. Stimson closed the US Cipher Bureau in 1929 with the words "Gentlemen do not read each other's mail."

Luckily for US COMINT, the Army offered a home to William Friedman after Stimson closed the Yardley operation. There, largely manual cylindrical and strip ciphers were developed, but, as a result of Friedman's advances in cryptanalysis, machine ciphers became a priority, such as the M134, also known as the SIGABA. While the SIGABA was a rotor machine like the German Enigma machine, it was never known to be cracked. It was replaced by electronic encryption devices.

The American Sigint effort began in the early 1930s with mounting tensions with the Japanese. The Navy started implementing high frequency DF (HF/DF) at eleven planned locations, primarily on the Atlantic Coast. The first operational intercept came from what would later be called Station CAST, at Cavite in the Philippines. In July 1939, the function turned from training and R&D to operations, and the Navy officially established a Strategic Tracking Organization under a Direction Finder Policy.

By December 1940, the Navy's communication organization, OP-20-G, had used HF/DF on German surface vessels and submarines. Training continued and cooperation with the British began. In April 1941, the British gave the US Navy a sample of their best HF/DF set from Marconi.

World War II

The use of SIGINT had even greater implications during World War II. The combined effort of intercepts and cryptanalysis for the whole of the British forces in World War II came under the code name "Ultra" managed from Government Code and Cypher School at Bletchley Park. By 1943, such was the extent of penetration of Axis communications and the speed and efficiency of distribution of the resulting intelligence, messages sometimes reached allied commanders in the field before their intended recipients. This advantage failed only when the German ground forces retreated within their own borders and they began using secure landline communications. For this reason, the Battle of the Bulge took the allies completely by surprise.

A true world war, SIGINT still tended to be separate in the various theaters. Communications security, on the part of the Allies, was more centralized. From the Allied perspective, the critical theater-level perspectives were the Ultra SIGINT against the Germans in the European theater (including the Battle of the Atlantic, the Mediterranean Theater of Operations, and MAGIC against the Japanese in the Pacific Theater and the China-Burma-India theater.

The entire German system of high command suffered from Hitler's deliberate fragmenting of authority, with Party, State, and military organizations competing for power. Hermann Göring also sought power for its own sake, but was much less effective as the war went on and he became more focused on personal status and pleasure.

Germany enjoyed some SIGINT success against the Allies, especially with the Merchant Code and, early in the war, reading American attaché traffic. German air intelligence, during the Battle of Britain, suffered from the structural problem that subordinated intelligence to operations. Operations officers often made conclusions that best fit their plans, rather than fitting conclusions to information.

In contrast, British air intelligence was systematic, from the highest-level, most sensitive Ultra to significant intelligence product from traffic analysis and cryptanalysis of low-level systems. Fortunately for the British, German aircraft communications discipline was poor, and the Germans rarely changed call signs, allowing the British to draw accurate inferences about the air order of battle.

Japan was the least effective of the major powers in SIGINT. In addition to the official Allies and Axis battle of signals, there was a growing interest in Soviet espionage communications, which continued after the war.

British SIGINT

Flow of information from an intercepted Enigma message at Bletchley Park.

The British Government Code and Cypher School moved to Bletchley Park, in Milton Keynes, Buckinghamshire, at the beginning of the Second World War. A key advantage was Bletchley's geographical centrality.

Commander Alastair Denniston was operational head of GC&CS. Key GC&CS cryptanalysts who moved from London to Bletchley Park included John Tiltman, Dillwyn "Dilly" Knox, Josh Cooper, and Nigel de Grey. These people had a variety of backgrounds – linguists, chess champions, and crossword experts were common, and in Knox's case papyrology. In one 1941 recruiting stratagem The Daily Telegraph was asked to organise a crossword competition, after which promising contestants were discreetly approached about "a particular type of work as a contribution to the war effort".

Denniston recognised, however, that the enemy's use of electromech­anical cipher machines meant that formally trained mathematicians would be needed as well; Oxford's Peter Twinn joined GC&CS in February 1939; Cambridge's Alan Turing and Gordon Welchman began training in 1938 and reported to Bletchley the day after war was declared, along with John Jeffreys. Later-recruited cryptanalysts included the mathematicians Derek Taunt, Jack Good, Bill Tutte, and Max Newman; historian Harry Hinsley, and chess champions Hugh Alexander and Stuart Milner-Barry. Joan Clarke (eventually deputy head of Hut 8) was one of the few women employed at Bletchley as a full-fledged cryptanalyst.

Properly used, the German Enigma and Lorenz ciphers should have been virtually unbreakable, but flaws in German cryptographic procedures, and poor discipline among the personnel carrying them out, created vulnerabilities which made Bletchley's attacks just barely feasible. These vulnerabilities, however, could have been remedied by relatively simple improvements in enemy procedures, and such changes would certainly have been implemented had Germany any hint of Bletchley's success. Thus the intelligence Bletchley produced was considered wartime Britain's "Ultra secret" – higher even than the normally highest classification Most Secret  – and security was paramount.

Initially, a wireless room was established at Bletchley Park. It was set up in the mansion's water tower under the code name "Station X", a term now sometimes applied to the codebreaking efforts at Bletchley as a whole. Due to the long radio aerials stretching from the wireless room, the radio station was moved from Bletchley Park to nearby Whaddon Hall to avoid drawing attention to the site.

Subsequently, other listening stations – the Y-stations, such as the ones at Chicksands in Bedfordshire, Beaumanor Hall, Leicestershire (where the headquarters of the War Office "Y" Group was located) and Beeston Hill Y Station in Norfolk – gathered raw signals for processing at Bletchley. Coded messages were taken down by hand and sent to Bletchley on paper by motorcycle despatch riders or (later) by teleprinter.

Bletchley's work was essential to defeating the U-boats in the Battle of the Atlantic, and to the British naval victories in the Battle of Cape Matapan and the Battle of North Cape. In 1941, Ultra exerted a powerful effect on the North African desert campaign against German forces under General Erwin Rommel. General Sir Claude Auchinleck wrote that were it not for Ultra, "Rommel would have certainly got through to Cairo". "Ultra" decrypts featured prominently in the story of Operation SALAM, László Almásy's daring mission across the Libyan Desert behind enemy lines in 1942. Prior to the Normandy landings on D-Day in June 1944, the Allies knew the locations of all but two of Germany's fifty-eight Western-front divisions.

Winston Churchill was reported to have told King George VI: "It is thanks to the secret weapon of General Menzies, put into use on all the fronts, that we won the war!" Supreme Allied Commander, Dwight D. Eisenhower, at the end of the war, described Ultra as having been "decisive" to Allied victory. Official historian of British Intelligence in World War II Sir Harry Hinsley, argued that Ultra shortened the war "by not less than two years and probably by four years"; and that, in the absence of Ultra, it is uncertain how the war would have ended.

German codes

Enigma machine in use, 1943.

Most German messages decrypted at Bletchley were produced by one or another version of the Enigma cipher machine, but an important minority were produced by the even more complicated twelve-rotor Lorenz SZ42 on-line teleprinter cipher machine.

Five weeks before the outbreak of war, in Warsaw, Poland's Cipher Bureau revealed its achievements in breaking Enigma to astonished French and British personnel. The British used the Poles' information and techniques, and the Enigma clone sent to them in August 1939, which greatly increased their (previously very limited) success in decrypting Enigma messages.

The bombe was an electromech­anical device whose function was to discover some of the daily settings of the Enigma machines on the various German military networks. Its pioneering design was developed by Alan Turing (with an important contribution from Gordon Welchman) and the machine was engineered by Harold 'Doc' Keen of the British Tabulating Machine Company. Each machine was about 7 feet (2.1 m) high and wide, 2 feet (0.61 m) deep and weighed about a ton.

At its peak, GC&CS was reading approximately 4,000 messages per day. As a hedge against enemy attack most bombes were dispersed to installations at Adstock and Wavendon (both later supplanted by installations at Stanmore and Eastcote), and Gayhurst.

Luftwaffe messages were the first to be read in quantity. The German navy had much tighter procedures, and the capture of code books was needed before they could be broken. When, in February 1942, the German navy introduced the four-rotor Enigma for communications with its Atlantic U-boats, this traffic became unreadable for a period of ten months. Britain produced modified bombes, but it was the success of the US Navy bombe that was the main source of reading messages from this version of Enigma for the rest of the war. Messages were sent to and fro across the Atlantic by enciphered teleprinter links.

SIGINT played a most important role for the Royal Navy, in its protection of merchant ships during the Battle of the Atlantic. While Ultra cryptanalysis certainly played a role in dealing with German submarines, HF/DF and traffic analysis were complementary.

A Mark 2 Colossus computer. The ten Colossi were the world's first programmable electronic computers.

It is unclear why the German submarine command believed that frequent radio communications were not a hazard to their boats, although they seemed confident in the security of their Enigma ciphers, both in the initial three-rotor and subsequent four-rotor versions (known as Triton to the Germans and Shark to the Allies). There was an apparent, mutually reinforcing belief that wolfpack attacks by groups of submarines were much more deadly than individual operations, and confidence the communications were secure. Arguably, the Germans underestimated HF/DF even more than they did British cryptanalysis. Apparently, the Germans did not realize that the Allies were not limited to slow, manually operated direction finders, and also underestimated the number of direction finders at sea. On the other hand, the introduction of a new secure communication system would have interrupted submarine operations for a long time since a gradual shift to a new system was out of the question.

The Lorenz messages were codenamed Tunny at Bletchley Park. They were only sent in quantity from mid-1942. The Tunny networks were used for high-level messages between German High Command and field commanders. With the help of German operator errors, the cryptanalysts in the Testery (named after Ralph Tester, its head) worked out the logical structure of the machine despite not knowing its physical form. They devised automatic machinery to help with decryption, which culminated in Colossus, the world's first programmable digital electronic computer. This was designed and built by Tommy Flowers and his team at the Post Office Research Station at Dollis Hill. The first was delivered to Bletchley Park in December 1943 and commissioned the following February. Enhancements were developed for the Mark 2 Colossus, the first of which was working at Bletchley Park on the morning of D-day in June. Flowers then produced one Colossus a month for the rest of the war, making a total of ten with an eleventh part-built. The machines were operated mainly by Wrens in a section named the Newmanry after its head Max Newman.

The "Radio Security Service" was established by MI8 in 1939 to control a network of Direction Finding and intercept stations to locate illicit transmissions coming from German spies in Britain. This service was soon intercepting a network of German Secret Service transmissions across Europe. Successful decryption was achieved at an early stage with the help of codes obtained from the British XX (Double Cross) System that "turned" German agents and used them to misdirect German intelligence. The combination of double agents and extensive penetration of German intelligence transmissions facilitated a series of highly successful strategic deception programmes throughout WWII.

Italian codes

Breakthroughs were also made with Italian signals. During the Spanish Civil War the Italian Navy used the K model of the commercial Enigma without a plugboard; this was solved by Knox in 1937. When Italy entered the war in 1940 an improved version of the machine was used, though little traffic was sent by it and there were "wholesale changes" in Italian codes and cyphers. Knox was given a new section for work on Enigma variations, which he staffed with women ("Dilly's girls") who included Margaret Rock, Jean Perrin, Clare Harding, Rachel Ronald, Elisabeth Granger; and Mavis Lever – who made the first break into the Italian naval traffic. She solved the signals revealing the Italian Navy's operational plans before the Battle of Cape Matapan in 1941, leading to a British victory.

On entering World War II in June 1940, the Italians were using book codes for most of their military messages. The exception was the Italian Navy, which after the Battle of Cape Matapan started using the C-38 version of the Boris Hagelin rotor-based cipher machine, particularly to route their navy and merchant marine convoys to the conflict in North Africa. As a consequence, JRM Butler recruited his former student Bernard Willson to join a team with two others in Hut 4. In June 1941, Willson became the first of the team to decode the Hagelin system, thus enabling military commanders to direct the Royal Navy and Royal Air Force to sink enemy ships carrying supplies from Europe to Rommel's Afrika Korps. This led to increased shipping losses and, from reading the intercepted traffic, the team learnt that between May and September 1941 the stock of fuel for the Luftwaffe in North Africa reduced by 90%. After an intensive language course, in March 1944 Willson switched to Japanese language-based codes.

Japanese codes

Allidina Visram school in Mombasa was the location of the Far East Combined Bureau codebreaking outpost during World War II.

An outpost of the Government Code and Cypher School was set up in Hong Kong in 1935, the Far East Combined Bureau (FECB), to study Japanese signals. The FECB naval staff moved in 1940 to Singapore, then Colombo, Ceylon, then Kilindini, Mombasa, Kenya. They succeeded in deciphering Japanese codes with a mixture of skill and good fortune. The Army and Air Force staff went from Singapore to the Wireless Experimental Centre at Delhi, India.

In early 1942, a six-month crash course in Japanese, for 20 undergraduates from Oxford and Cambridge, was started by the Inter-Services Special Intelligence School in Bedford, in a building across from the main Post Office. This course was repeated every six months until war's end. Most of those completing these courses worked on decoding Japanese naval messages in Hut 7, under John Tiltman. By mid-1945 well over 100 personnel were involved with this operation, which co-operated closely with the FECB and the US Signal intelligence Service at Arlington Hall, Virginia. Because of these joint efforts, by August of that year the Japanese merchant navy was suffering 90% losses at sea. In 1999, Michael Smith wrote that: "Only now are the British codebreakers (like John Tiltman, Hugh Foss, and Eric Nave) beginning to receive the recognition they deserve for breaking Japanese codes and cyphers".

US SIGINT

During the Second World War, the US Army and US Navy ran independent SIGINT organizations, with limited coordination, first on a pure personal basis, and then through committees.

After the Normandy landings, Army SIGINT units accompanied major units, with traffic analysis as - or more - important than the tightly compartmented cryptanalytic information. General Bradley's Army Group, created on August 1, 1944, had SIGINT including access to Ultra. Patton's subordinate Third Army had a double-sized Signal Radio Intelligence Company attached to his headquarters, and two regular companies were assigned to the XV and VIII Corps.

The US Navy used SIGINT in its anti-submarine warfare, using shore or ship-based SIGINT to vectored long-range patrol aircraft to U-boats.

Allied cooperation in the Pacific Theater included the joint RAN/USN Fleet Radio Unit, Melbourne (FRUMEL), and the Central Bureau which was attached to the HQ of the Allied Commander of the South-West Pacific area.

At first, Central Bureau was made up of 50% American, 25% Australian Army and 25% Royal Australian Air Force (RAAF) personnel, but additional Australian staff joined. In addition, RAAF operators, trained in Townsville, Queensland in intercepting Japanese telegraphic katakana were integrated into the new Central Bureau.

Until Central Bureau received replacement data processing equipment for that which was lost in the Philippines , as of January 1942, U.S. Navy stations in Hawaii (Hypo), Corregidor (Cast) and OP-20-G (Washington) decrypted Japanese traffic well before the U.S. Army or Central Bureau in Australia. Cast, of course, closed with the evacuation of SIGINT personnel from the Philippines. Central Bureau broke into two significant Japanese Army cryptosystems in mid-1943.

Japanese codes

LCDR Joseph Rochefort of the US Navy led and handpicked many of the key codebreakers at Station HYPO.

The US Army shared with the US Navy the Purple attack on Japanese diplomatic cryptosystems. After the creation of the Army Signal Security Agency, the cryptographic school at Vint Hill Farms Station, Warrenton, Virginia, trained analysts. As a real-world training exercise, the new analysts first solved the message center identifier system for the Japanese Army. Until Japanese Army cryptosystems were broken later in 1943, the order of battle and movement information on the Japanese came purely from direction finding and traffic analysis.

Traffic analysts began tracking Japanese units in near real time. A critical result was the identification of the movement, by sea, of two Japanese infantry divisions from Shanghai to New Guinea. Their convoy was intercepted by US submarines, causing almost complete destruction of these units.

Army units in the Pacific included the US 978th Signal Company based at the Allied Intelligence Bureau's secret "Camp X", near Beaudesert, Queensland south of Brisbane. This unit was a key part of operations behind Japanese lines, including communicating with guerillas and the Coastwatcher organization. It also sent radio operators to the guerillas, and then moved with the forces invading the Philippines.

US Navy strategic stations targeted against Japanese sources at the outbreak of the war, included Station HYPO in Hawaii, Station CAST in the Philippines, station BAKER on Guam, and other locations including Puget Sound, and Bainbridge Island. US COMINT recognized the growing threat before the Pearl Harbor attack, but a series of errors, as well as priorities that were incorrect in hindsight, prevented any operational preparation against the attack. Nevertheless, that attack gave much higher priority to COMINT, both in Washington, D.C. and at the Pacific Fleet Headquarters in Honolulu. Organizational tuning corrected many prewar competitions between the Army and Navy.

Perhaps most dramatically, intercepts of Japanese naval communications yielded information that gave Admiral Nimitz the upper hand in the ambush that resulted in the Japanese Navy's defeat at the Battle of Midway, six months after the Pearl Harbor attack.

The US Army Air Force also had its own SIGINT capability. Soon after the Pearl Harbor attack, Lieutenant Howard Brown, of the 2nd Signal Service Company in Manila, ordered the unit to change its intercept targeting from Japanese diplomatic to air force communications. The unit soon was analyzing Japanese tactical networks and developing order of battle intelligence.

They learned the Japanese air-to-ground network was Sama, Hainan Island, with one station in Indochina, one station near Hong Kong, and the other 12 unlocated. Two Japanese naval stations were in the Army net, and it handled both operations and ferrying of aircraft for staging new operations. Traffic analysis of still-encrypted traffic helped MacArthur predict Japanese moves as the Fil-American forces retreated in Bataan.

An Australian-American intercept station was later built at Townsville, Queensland. US Air Force Far East, and its subordinate 5th Air Force, took control of the 126th in June 1943. The 126th was eventually placed under operational control of U.S. Air Force Far East in June 1943 to support 5th Air Force. Interception and traffic analysis from the company supported the attack into Dutch New Guinea in 1944.

Cold War

After the end of World War II, the Western allies began a rapid drawdown. At the end of WWII, the US still had a COMINT organization split between the Army and Navy. A 1946 plan listed Russia, China, and a [redacted] country as high-priority targets.

From 1943 to 1980, the Venona project, principally a US activity with support from Australia and the UK, recovered information, some tantalizingly only in part, from Soviet espionage traffic. While the Soviets had originally used theoretically unbreakable one-time pads for the traffic, some of their operations violated communications security rules and reused some of the pads. This reuse caused the vulnerability that was exploited.

Venona gave substantial information on the scope of Soviet espionage against the West, but critics claim some messages have been interpreted incorrectly, or are even false. Part of the problem is that certain persons, even in the encrypted traffic, were identified only by code names such as "Quantum". Quantum was a source on US nuclear weapons, and is often considered to be Julius Rosenberg. The name, however, could refer to any of a number of spies.

US Tactical SIGINT

After the Beirut deployment, Lieutenant General Alfred M. Gray, Jr. did an after-action review of the 2nd Radio Battalion detachment that went with that force. Part of the reason for this was that the irregular units that presented the greatest threat did not follow conventional military signal operating procedures, and used nonstandard frequencies and callsigns. Without NSA information on these groups, the detachment had to acquire this information from their own resources.

Recognizing that national sources simply might not have information on a given environment, or that they might not make it available to warfighters, Lieutenant General Gray directed that a SIGINT function be created that could work with the elite Force Reconnaissance Marines who search out potential enemies. At first, neither the Force Reconnaissance nor Radio Battalion commanders thought this was viable, but had orders to follow.

Initially, they attached a single Radio Battalion Marine, with an AN/GRR-8 intercept receiver, to a Force Reconnaissance team during an exercise. A respected Radio Marine, Corporal Kyle O'Malley was sent to the team, without any guidance for what he was to do. The exercise did not demonstrate that a one-man attachment, not Force Recon qualified, was useful.

In 1984, Captain E.L. Gillespie, assigned to the Joint Special Operations Command, was alerted that he was to report to 2nd Radio Battalion, to develop a concept of operations for integrating SIGINT capabilities with Force Recon, using his joint service experience with special operations. Again, the immediate commanders were not enthusiastic.

Nevertheless, a mission statement was drafted: "To conduct limited communications intelligence and specified electronic warfare operations in support of Force Reconnaissance operations during advance force or special operations missions." It was decided that a 6-man SIGINT team, with long/short range independent communications and SIGINT/EW equipment, was the minimum practical unit. It was not practical to attach this to the smallest 4-man Force Recon team.

General Gray directed that the unit would be called a Radio Reconnaissance Team (RRT), and that adequate planning and preparation were done for the advance force operations part of the upcoming Exercise Solid Shield-85. Two six-man teams would be formed, from Marines assigned from the Radio Battalion, without great enthusiasm for the assignment. One Marine put it"There is nothing that the Marine Corps can do to me that I can't take." Force Recon required that the RRT candidates pass their selection course, and, to the surprise of Force Recon, they passed with honors. Both teams were assigned to the exercise, and the RRTs successfully maintained communications connectivity for Force Recon and SEALs, collected meaningful intelligence, disrupted opposing force communications, and were extracted without being compromised.

From 1986 on, RRTs accompanied MEU(SOC) deployments. Their first combat role was in Operation Earnest Will, then Operation Praying Mantis, followed by participation in the 1989 United States invasion of Panama

Recent history

Threat from terrorism

Terrorism from foreign groups became an increasingly major concern beginning in the late 20th century, as seen with the 1992 al-Qaeda attack in Yemen, the 1993 truck bombing of the World Trade Center, the 1995 Khobar Towers bombing in Saudi Arabia and the 1998 bombings of the US embassies in Dar es Salaam, Tanzania and Nairobi, Kenya. Third world and non-national groups, with modern communications technology, in many ways are a harder SIGINT target than a nation that sends out large amounts of traffic. According to the retired Commandant of the US Marines, Alfred M. Gray, Jr., some of the significant concerns of these targets are:

• Inherently low probability of intercept/detection (LPI/LPD) because off-the-shelf radios can be frequency agile, spread spectrum, and transmit in bursts.
• Additional frequencies, not normally monitored, can be used. These include citizens band, marine (MF, HF, VHF) bands, personal radio services such as MURS, FRS/GMRS and higher frequencies for short-range communications
• Extensive use of telephones, almost always digital. Cellular and satellite telephones, while wireless, are challenging to intercept, as is Voice over IP (VoIP)
• Commercial strong encryption for voice and data
• "Extremely wide variety and complexity of potential targets, creating a "needle in the haystack" problem"

As a result of the 9/11 attacks, intensification of US intelligence efforts, domestic and foreign, were to be expected. A key question, of course, was whether US intelligence could have prevented or mitigated the attacks, and how it might prevent future attacks. There is a continuing clash between advocates for civil liberties and those who assert that their loss is an agreeable exchange for enhanced safety.

Another possibility is the use of software tools that do high-performance deep packet inspection. According to the marketing VP of Narus, "Narus has little control over how its products are used after they're sold. For example, although its lawful-intercept application has a sophisticated system for making sure the surveillance complies with the terms of a warrant, it's up to the operator whether to type those terms into the system...

"That legal eavesdropping application was launched in February 2005, well after whistle-blower Klein allegedly learned that AT&T was installing Narus boxes in secure, NSA-controlled rooms in switching centers around the country. But that doesn't mean the government couldn't write its own code to do the dirty work. Narus even offers software-development kits to customers ". The same type of tools with legitimate ISP security applications also have COMINT interception and analysis capability.

Former AT&T technician Mark Klein, who revealed AT&T was giving NSA access, said in a statement, said a Narus STA 6400 was in the NSA room to which AT&T allegedly copied traffic. The Narus device was "known to be used particularly by government intelligence agencies because of its ability to sift through large amounts of data looking for preprogrammed targets."

European Space Systems cooperation

France launched Helios 1A as a military photo-reconnaissance satellite on 7 July 1995. The Cerise (satellite) SIGINT technology demonstrator also was launched in 1995. A radio propagation experiment, S80-T, was launched in 1992, as a predecessor of the ELINT experiments. Clementine, the second-generation ELINT technology demonstrator, was launched in 1999.

Financial pressures in 1994-1995 caused France to seek Spanish and Italian cooperation for Hélios 1B and German contributions to Helios 2. Helios 2A was launched on 18 December 2004. Built by EADS-Astrium for the French Space Agency (CNES), it was launched into a Sun-synchronous polar orbit at an altitude of about 680 kilometers.

The same launcher carried French and Spanish scientific satellites and four Essaim ("Swarm") experimental ELINT satellites

Germany launched their first reconnaissance satellite system, SAR-Lupe, on December 19, 2006. Further satellites were launched at roughly six-month intervals, and the entire system of this five-satellite synthetic aperture radar constellation achieved full operational readiness on 22 July 2008. SAR is usually considered a MASINT sensor, but the significance here is that Germany obtains access to French satellite ELINT.

The joint French-Italian Orfeo Programme, a dual-use civilian and military satellite system, launched its first satellite on June 8, 2007. Italy is developing the Cosmo-Skymed X-band polarimetric SAR, to fly on two of the satellites. The other two will have complementary French electro-optical payloads. The second Orfeo is scheduled to launch in early 2008.

See also

• Signals intelligence by alliances, nations and industries

Bibliography

• Aldrich, R. J. (2010), GCHQ: The Uncensored Story of Britain's Most Secret Intelligence Agency, London: HarperCollins Press, ISBN 978-0-00-727847-3 
• Alvarez, David (2001), "Most Helpful and Cooperative: GC&CS and the Development of American Diplomatic Cryptanalysis, 1941-1942", in Smith, Michael; Erskine, Ralph, Action This Day: Bletchley Park from the Breaking of the Enigma Code to the Birth of the Modern Computer, Bantam Press, ISBN 978-0593049105 
• Budiansky, Stephen (2000), Battle of wits: The Complete Story of Codebreaking in World War II, Free Press, pp. 176–179, ISBN 978-0684859323, https://archive.org/details/battleofwitscomp00budi/page/176 
• Holmes, W. J. (1979), Double-edged Secrets: U.S. Naval Intelligence Operations in the Pacific During World War II, Annapolis, Maryland: Blue Jacket Books/Naval Institute Press, ISBN 1-55750-324-9 
• Kahn, David (1996), The Codebreakers: The Comprehenisve History of Secret Communication from Ancient Times to the Internet, New York: Scribner, ISBN 0684831309 
• Lee, Bartholomew (2006), Radio Spies -- Episodes in the Ether Wars, http://www.trft.org/TRFTPix/spies9eR2006.pdf, retrieved 2007-10-08 
• Parker, Frederick D., A Priceless Advantage: U.S. Navy Communications Intelligence and the Battles of Coral Sea, Midway, and the Aleutians, National Security Agency, Central Security Service, http://www.ibiblio.org/hyperwar/PTO/Magic/COMINT-CoralSea/index.html, retrieved November 20, 2006 
• Prados, John (1995), Combined Fleet Decoded: The Secret History of American Intelligence and the Japanese Navy in World War II, New York: Random House, ISBN 0-679-43701-0