Thursday, July 13, 2017

Working on the Dark Side of the Moon: Life Inside the National Security Agency

A new book on the NSA has been published recently. The book in question is ‘Working on the Dark Side of the Moon: Life Inside the National Security Agency’ by Thomas Reed Willemain.

Maitland, FL (May 19, 2017) –Working on the Dark Side of the Moon provides the first, ground-level look inside the super-secret National Security Agency (NSA) and a shadowy think tank affiliated with it. The author, a software entrepreneur and statistics professor, volunteered for a year-long sabbatical tour of duty in the NSA. He ended up spending several years moving between the business and academic worlds and the secret world. This book records his impressions of people and places never before described in such intimate detail.

A deeply personal account of the years spent within the most secretive organization in the world, Working on the Dark Side of the Moon explores the range of emotions an outsider experiences while crossing over to the “inside.” It also shows the positive side of an Agency whose secrecy hides dedicated men and women devoted to protecting the country while honoring the Constitution.

Willemain writes, "The very secrecy that enables NSA to be effective also cripples its ability to explain its positive contributions. Into this void are projected grossly distorted views of what NSA does and what NSA people are like. This book puts a human face on the people who work in this secret world: their character, motivations, frustrations, sense of humor. Readers can develop a more balanced and nuanced view of NSA and its people."

About the Author

Dr. Thomas Reed Willemain served as an Expert Statistical Consultant to the National Security Agency (NSA) at Ft. Meade, MD and as a member of the Adjunct Research Staff at an affiliated think-tank, the Institute for Defense Analyses Center for Computing Sciences (IDA/CCS). He is Professor Emeritus of Industrial and Systems Engineering at Rensselaer Polytechnic Institute, having previously held faculty positions at Harvard’s Kennedy School of Government and Massachusetts Institute of Technology. He is also co-founder and Senior Vice President/Research at Smart Software, Inc. He is a member of the Association of Former Intelligence Officers, the Military Operations Research Society, the American Statistical Association, and several other professional organizations. Willemain received the BSE degree (summa cum laude, Phi Beta Kappa) from Princeton University and the MS and PhD degrees from Massachusetts Institute of Technology.

His other books include: Statistical Methods for Planners, Emergency Medical Systems Analysis (with R. C. Larson), and 80 articles in peer-reviewed journals on topics in statistics, operations research, health care and other topics.

Q&A with Thomas Reed Willemain

The author was kind enough to answer some of my questions.

1). Can you give an overview of your career prior to working for the NSA?

I’ve had overlapping careers:  About 40 years as an academic, and about 30 years as a software entrepreneur. I have been a professor at Massachusetts Institute of Technology, Harvard Kennedy School of Government, and Rensselaer Polytechnic Institute. I am now Professor Emeritus of Industrial and Systems Engineering at RPI. I am also co-founder and Senior Vice President/Research at Smart Software, Inc. in Boston. A common thread has been the study of statistics, forecasting -- anything involving randomness.

2). How/why did you consider working for the NSA?

I was looking for a challenging and useful sabbatical leave. I’d previously spent a sabbatical leave at the Federal Aviation Administration and made some contributions there, even though I’d not had any formal background in aviation. I was wary of applying to NSA, since I was not in synch with the Bush administration. But I wanted another period of public service. I also knew that there would be no more intriguing place for a statistician to work. And I suspected, correctly, that when I came back to RPI I would have more to contribute to my students. That turned out to be correct, in that my courses were richer (and more computational) afterwards.

3). What did you expect working at the NSA would be like and were your expectations accurate or not?

I was very wrong about some things. One was politics, or the lack thereof. I mentioned my misgivings about President Bush. The woman who handled the sabbatical program was very diplomatic and not put off by my questions. When I finally met her in person, it turned out that she was a lesbian with an “Anybody but Bush” bumper sticker on her car – not at all fitting my stereotype of an NSA person. During the McCain-Obama election campaign, the bumper stickers in the vast parking lots were about 50:50, and there was no whiff of politics inside the wire. The only person who talked (incessantly) about the election was somebody from another country embedded with us. I did expect a high level of expertise, and that was definitely true.

Something I should have expected but did not was how radically different the culture was from my university life. I was going back and forth between “inside” and “outside”. The academic culture encourages the question “Hey, what are you working on?” I had to learn to not ask that question on the inside unless it was behind a locked door, and not always then.

Now, the NSA is a big place. And one of the people described in my book pointed out that I was in the Research Directorate, which is more like a playground for uber-geeks than most of the rest of the Agency, where a mix of civilians and service members grind out massive amounts of work every day. So my book must present a partial picture of “Life inside the National Security Agency”. I may have been the proverbial blind man feeling the best part of the proverbial elephant.

4). Why did you decide to write a book about your experiences working for the NSA and was it difficult to gain approval from the agency?

I’ll be 70 years old soon, and I found myself slowing down on the math side of things, so I looked for another way to contribute. I had a plan to begin substituting my words for my equations, and writing the book would be a good way to test the feasibility of that plan. But I was also motivated by a desire to continue serving as best I could. Most every depiction of NSA in the media has been negative, and distorted stereotypes about the people and the Agency are rampant. I wanted to offset that with an insider’s look at the reality. The Snowden affair in particular prompted me to try to offset that. It turned out that, without knowing what I was contributing to, some of my technical work helped the Agency offset some of the damage Snowden did. The book let me do more on that front.

Getting the book cleared through NSA’s pre-publication review was a slow-motion crucifixion. It delayed the book by five months and blacked out about 15% of the book. There was some lying and bullying involved. Call it a character-building moment. I wrote about the process in the LawFare blog and discussed it with the House Permanent Select Committee on Intelligence, who were already reviewing the pre-pub process. The basic problem is that the process knows only one word: “No”. I tried to get the strategic communications people involved so there would be someone to say “Yes” to the idea of permitting a pro-Agency book to be published, but so far no luck. The Agency claimed, with perhaps dubious legality, that anybody described in my book, though anonymously, could require me to remove them from the book. If they had all done so, there would have been no book. But only one insisted that she be removed. She is now a large black rectangle.

5). What new information is available from your book compared to previous studies of the NSA?

I’m fairly certain that this is the only grunt-level memoir of service in the NSA. There are a few faux-memoirs that are works of fiction. Folks at the top levels have written books (e.g., Michael Hayden), but daily life below the top has been, well, rather like the dark side of the moon. There have been policy-oriented and history-oriented books about NSA, but not people-oriented books. So what it feels like to work there has been mysterious. Much of my book is centered on descriptions of about 40 people that I worked with, and the book is about their stories as much as mine. I also paid a lot of attention to comparing life inside against life outside, especially regarding the intellectual and administrative climates (including personnel evaluation systems). There are not many “insider/outsider” stories to tell, and mine is the only one in print.

Actually, part way through my time inside, several of us academics were “traded” to NSA-affiliated think tanks. So my book is also the first to expose the inner workings of the Institute for Defense Analyses Center for Computing Sciences. That must be the world’s most comfortable SCIF, and it’s full of sharp, colorful characters. I think the director of IDA/CCS was even more opposed to publication of my book than the NSA itself, even though my book might be very helpful to recruiting people to take my place there.

6). What is your opinion on the recent Snowden revelations regarding the NSA interception of US civilian communications?

I have mixed feeling about Snowden, mostly negative. Perhaps some of his motivation was idealistic. But what he did was very damaging to the tracking of foreign targets, so he definitely belongs in jail. He also appears to be a narcissistic liar. He permitted a persona to be presented in the movie “Snowden” that was just not true. As I watched the movie, I kept thinking “That’s not true. And that’s not true. And that doesn’t really happen.” For instance, I write about my struggles to pass the repeated exams I had to take to certify that I knew about the practical implementation of the Fourth Amendment prohibitions as applied to foreign intelligence. The public should know how seriously the Agency regards those things. It is certain that something as powerful as the NSA bears constant watching, but facts ought to be the basis for judgment.

Sunday, July 9, 2017


1). I’ve made some corrections and also changed the title in The Japanese FUJI diplomatic cipher 1941-43 (for example instead of saying the Germans or OKW/Chi solved the code I mention the specific department etc)

2). I uploaded the file containing the Japanese decodes of US diplomatic traffic from Diplomatic records Office, Tokyo, ‘U.S.-Japan Relations, Miscellaneous Diplomatic Correspondence-Special Information File’ (A-1-3-1, 1-3-2). Link here.

Also fixed the broken links in Japanese codebreakers of WWII.

Saturday, July 1, 2017

The quest for the missing NAASt 5 reports - Update

In April I said that I’ve been trying to locate the two missing reports of NAASt 5, which was the cryptanalytic centre of KONA 5 (Kommandeur der Nachrichtenaufklärung - Signals Intelligence Regiment).

KONA 5 covered Western Europe and the cryptanalysts of NAASt 5 were able to solve the US M-209 cipher machine in 1944.

According to the TICOM report IF-272 - TAB ‘D’ the following NAAS 5 reports survived the war:

E-Bericht Nr. 1/44 der NAAst 5 dated 10.1.44

E-Bericht Nr. 2/44 der NAAst 5

E-Bericht Nr. 3/44 der NAAst 5 (Berichtszeit 1.4-30.6.44)

E-Bericht 4/44 der NAAst 5 (Berichtszeit 1.7-30.9.44) dated 10.10.44 

E-Bericht der NAAst 5 (Berichtszeit 1.10.44-30.12.44) dated 14.1.45

The first three can be found in the US national archives, collection RG 457 - Entry 9032 - box 22, titled ‘German deciphering reports’.

Unfortunately the last two (covering the second half of 1944) are not there.

Initially the NSA FOIA office told me that the NAASt 5 reports had been transferred to the US National archives as part of transfer group TR-0457-2016-0014. However when the NARA FOIA office checked these files they were unable to locate any report titled E-Bericht NAAs 5.

I then asked the NSA FOIA office again about these files, since it seems they made a mistake and I was told to check transfer group TR-0457-2017-0010.

Now the response from the NARA research office regarding this transfer group has been the following:

‘We have received the records of which you speak and they must first of all undergo formal accessioning and any necessary preservation. Then they will need to be archivally described and professionally arranged before they will be available for research.   ALL of those steps will depend on how many previous accessions are in line to be processed.

Although you have the most up-to-date information on these record transfers, our archival processing steps must be done prior to making the records available for public use.’

So it seems that I’ll have to wait for NARA to process the transfer group TR-0457-2017-0010 and then they can search it for the NAASt 5 reports (assuming they are there).

Thursday, June 22, 2017

German signals intelligence successes during operation Barbarossa

On June 22 1941 the military forces of Nazi Germany attacked the Soviet Union, thus starting the largest land campaign in history.

Codebreaking and signals intelligence played a major role in the German war effort. Army and Luftwaffe units relied on signals intelligence in order to monitor enemy units and anticipate major actions.

For a summary of German signal intelligence operations read Compromise of Soviet codes in WWII.

Saturday, June 10, 2017

Information on the Enigma cipher machine found in the War Diary of Inspectorate 7/VI

During WWII the German Army made extensive use of signals intelligence and codebreaking in its operations against enemy forces. German commanders relied on signals intelligence in order to ascertain the enemy’s order of battle and track the movements of units.

The German Army’s signal intelligence agency operated a number of fixed intercept stations and also had mobile units assigned to Army Groups. These units were called KONA (Kommandeur der Nachrichtenaufklärung - Signals Intelligence Regiment) and each had an evaluation centre, a stationary intercept company, two long range signal intelligence companies and two close range signal intelligence companies.

The KONA units did not have the ability to solve complicated Allied cryptosystems. Instead they focused on exploiting low/mid level ciphers and even in this capacity they were assisted by material sent to them by the central cryptanalytic department in Berlin. This was the German Army High Command’s Inspectorate 7/VI

The War Diary of Inspectorate 7/VI

Some files of the German army signal intelligence service survived WWII and were retrieved in 1947 from a camp in Glasenbach, Austria, where they had been buried at the end of the war.

The War Diary of Inspectorate 7/VI for the years 1939-45 can be found in the US National Archives, in collection RG457 and in the TICOM collection of the German Foreign Ministry’s Political Archive.

The reports of departments 1, 7, 13 and F occasionally have information on the Enigma cipher machine (both commercial and plugboard versions).

Initially department 1 was responsible for general cryptanalytic research but in 1941 department 7 was created to look into the security of German cipher systems. For a time both 1 and 7 did general crypto research. In November 1942 department 13 was created and from then on department 7 dealt solely with German hand systems, while department 13 was responsible for German cipher machines. In 1943 department F (Forschung/Research) was created to do general cryptanalytic research.

I’ve copied the relevant passages from the War Diary and used google translate. However many terms were not translated correctly so it was up to Frode Weierud, an expert on Enigma history, to correct these passages.

Thus I present the War Diary entries dealing with the Enigma machine for the years 1941-45 (I’m afraid I don’t have the files of 1939-40):

Friday, June 9, 2017

Secure ciphers - Insecure messages

In the construction and use of tactical cryptosystems there are two conflicting requirements. One is security and the other is ease of use. If a system is highly secure but hard and time consuming to use then important messages might be secure from cryptanalysis but they could arrive too late, with disastrous consequences. On the other hand if a system is extremely easy to use but insecure then the messages will get through on time but the enemy will also be able to read them.

The Slidex code, used by the US and British armies in WWII was easy to use but it could be solved in a few hours by the German codebreakers.

However the British Army’s double transposition cipher and the US Army’s M-209 cipher machine were basically secure systems, since they could only be solved through mistakes in encipherment. It seems that contrary to regulations the Allied troops did not always use these systems in the field since it took too long to encipher their messages.

UK example

Letter from the War Office to the Commanders in Chief 21st Army Group, Home Forces, Middle East, Persia-Iraq (dated February 1945):

I am commanded by the Army Council to inform you that further consideration has been given to the suitability for operational purposes of the Low-Grade cipher "Double Transposition" which was introduced for use throughout the Army by War Office letter 32/Tels/943 dated 5th November, 1943.
2. Experience shows that while this cipher affords adequate security, unit personnel find it difficult and slow to operate. There is, therefore, a tendency to avoid the use of cipher with a consequent possibility of overstrain of other safe means of communication or the use of wireless in clear to a dangerous extent.
3. It has, therefore, been decided to adopt a new Low Grade cipher, called LINEX, details of which are given in appendices A to D, in place of Double Transposition.’

US example

Report of interview with S/Sgt, Communications Section 79 Inf Div, 7th Army. (dated March 1945):

"The US Army code machine #209 was found to be something that hampered operations. It would take at least half hour to get a message through from the message center by use of this code machine and as a result the codes of particular importance or speed, for instance mortar messages, were sent in the clear."

Sources: British national archives WO 193/211 ‘Wireless, cable and signal (including cipher) communications: policy and codes: action from report of Godwin-Austen Committee’, US national archives - collection RG457 - Entry 9032 - box 1.024 - US COMSEC reports.