Sunday, December 13, 2015

Claude Shannon and Norbert Wiener

Bill Collins, Executive Director, Thumb Land Conservancy

Once in a while, I decide to read about something I haven’t before. Could be just about anything. Yes, I am one of those people that will occasionally read through a dictionary. I’ve learned quite a bit by doing that. I realize there’s a lot of knowledge out there in books and other print that will probably never make it to the web, but I’m still amazed by the internet. I try not to take it for granted. There are some days that the choices feel too overwhelming, or I just don’t care for anything new, even the weather. But most of the time, I’m still fascinated by the opportunity to go almost anywhere, into almost any subject, with one search on a browser. Life is just too short though.

So, while writing the previous blog article about plant ecology and monitoring the Michigan Road Preserve, I decided to look up Shannon of the Shannon-Wiener Species Diversity Index. I really wasn’t expecting much, like maybe some abstract of a university ecology paper or mention of someone that worked for the US Fish and Wildlife Service, or something like that. Ha ha. Boy was I wrong.

Claude Shannon
The first thing I read was that Shannon was Claude Elwood Shannon, born in Petoskey, Michigan in 1916, the same year as my grandfather, and he grew up in Gaylord, Michigan of all places. That was completely unexpected. But the rest of his story is far more fascinating. Together with the story of Norbert Wiener, this little tangent led to some of that rare historical treasure that I sometimes dig up. But now, knowing about these two men, I feel a little stupid for not having known about them a long time ago.

Gaylord and Petoskey, Michigan, in the northern Lower Peninsula.
As a boy, Shannon was inspired by Thomas Edison, who grew up in Port Huron, Michigan as many of us Michiganders know, my home town. Shannon was a young tinkerer just like Edison, and later he even found out they were distantly related. Shannon studied electrical engineering at the University of Michigan. Then he went off to the Massachusetts Institute of Technology where he ended up working with an analog computer called the “Differential Analyzer”. Conceived in the 1920’s by MIT Dean of Engineering, Vannevar Bush, the Differential Analyzer was built and working by 1931. It was a mechanical computer, the only electrical parts being the motors that drove the gears. Preparing it to work with one equation and obtaining an answer could take almost a week. As a work-study student, Shannon helped visiting scientists set up the analyzer by adjusting mechanical linkages to correspond to their mathematical equations.

Claude Elwood Shannon. Wikipedia:

Shannon was intrigued by the operation of the many relay switches of the Differential Analyzer and enjoyed watching them work. Multiple relay switches were either on or off at a given time, and this reminded him of symbolic logic he studied back at the University of Michigan, a binary logic in which statements are either true or false, essentially on or off. Shannon made the connection between the relay switches and the potential to create mechanical and electronic logic. Apparently, this had not occurred to anyone before. Vannevar Bush suggested that Shannon study the operation of the Differential Analyzer relay circuits for his Master’s Thesis. In 1937, Shannon wrote what is described in many references as possibly the most important Master’s Thesis of the Twentieth Century, A Symbolic Analysis of Relay and Switching Circuits, describing how Boolean algebra and binary arithmetic could be used with electrical relays to solve mathematical problems. He demonstrated how such binary circuits could perform complex mathematical functions such as “if the number X equals the number Y, then do operation A.” As an illustration of this, he showed how relay switches could be arranged to create an electronic lock that opened only if a series of buttons was pressed in a certain order. Shannon’s conclusions in his 1937 thesis became the basis of modern computers and all digital technology since. The operation of a machine utilizing an internal program not only led to the development of the computer that you are most likely reading this article on right now, but also initiated the field of artificial intelligence. Shannon once said that he had more fun doing this work than anything else in his life.

Vannevar Bush with his differential analyzer. Bush’s Analog Solution. Computer History museum:

Claude Shannon. DaTuOpinion:
At a few points while reading about Shannon, I was wondering when he made a switch to ecology and what motivated him. Was it because he grew up in Gaylord, surrounded by forest, lakes and the northern Michigan wilderness? I was thinking back on my history. I had three years of electronics at Port Huron Northern High School taught by the great and ever-entertaining Paul Johnson. As I recall, it was a 2-hour class every day. My plan was, encouraged by Mr. Johnson, to study electrical engineering at Michigan Technological University up in Houghton, Michigan. Not only was MTU considered a leading university for electrical engineering, but I loved the Upper Peninsula. Eventually, my interest in nature and the environment led me to study natural resources and ecology at Michigan State University. Probably should have stuck with electronics. Anyway, turns out that development of the Shannon-Wiener Diversity Index had nothing to do with ecology.

In 1941, Shannon was hired by Bell Laboratories where he worked on automated anti-aircraft firing controls and cryptography, both of which became critical as the US entered World War II. In 1943, he had some association with the famous British cryptanologist and mathematician Alan Turning, who, a few years earlier, had broken the Naval Enigma machine used by the Nazi’s. From about this time, Shannon became more involved with cryptography and communication theory. His interest in transmitting and receiving information steadily grew as he realized that perfect, undistorted information transmission was theoretically possible using the digital technology that he developed a few years earlier. He wrote to his MIT graduate advisor, Vannevar Bush, “Off and on, I have been working on an analysis of some of the fundamental properties of general systems for the transmission of intelligence, including telephony, radio, television, telegraphy, etc.”. I think he actually meant that “Off and on” as a joke. In his spare time, while at Bell Laboratories, he continued to work on the mathematics of his digital information ideas, but according to several sources, he kept it mostly to himself and no one realized what he was doing. Shannon said he was motivated simply by curiosity and that publication was a “painful” process. I got that idea while at MSU, which is partly why I never went on to earn a graduate degree. I mean, back in the mid 1980’s they barely had reliable word processing programs. Publishing in the 1940’s had to be very difficult.

Claude Shannon with Theseus the intelligent electronic mouse.

But finally, in 1948, Shannon once again published a ground-breaking article, A Mathematical Theory of Communication in the July and October issues of the Bell System Technical Journal. Before this paper, there was apparently no concept of unifying all forms of communication into digital format, including radio, television, audio, visual, data, and so on. Equally as important, Shannon demonstrated that perfect transmission or communication was possible, within limits, regardless of the amount of background noise or distortion, or how faint the signal. Shannon’s “fundamental theorem” of information theory, as he called it, surprised even him. Before this, it seemed that signal noise would always be a problem. Again, this was a huge leap forward in thinking that enabled more technological advances than I care to list here, not the least of which is space exploration. Obviously, his work is a foundation of the internet as well, without which, again, you would likely not be reading this article.

Claude Shannon. Virtual Panorama. The Universe Of Information At Your Service:
What became the equation for the Shannon-Wiener Diversity Index was originally developed as a way to quantify the entropy or uncertainty of information in groups of text for the purpose of deciphering messages and codes, and to facilitate transmission and reception of data according to Shannon’s ideas of information theory. The more different letters or other data characters there are in a line of text or other group of characters, and the more equal their proportional abundance, the more difficult it is to correctly predict the next characters in a sequence. The Shannon entropy equation quantified the uncertainty of each new character.

The Shannon-Wiener Diversity Index, or entropy, equation.

Someone, I don’t know who yet, later got the idea this same equation could be a useful measure of species diversity in a given sampling area. The same concept applies to species or any other group of assorted things as lines of data characters. In a given group of species, the more different species there are, and the more even or proportional their abundance, the more diverse the group is and the harder it would be to predict which species you might encounter in the sampling area. For example, if there were only a few species and abundance was skewed toward one or two species, you could easily predict which species would be encountered most of the time. If there are many species with fairly equal abundance, it would be much more difficult to predict what species would be encountered and this would be described as a very diverse community. There have since been more refined equations developed, but the Shannon-Wiener Index essentially gets at what species diversity is.

Reading about Shannon was one surprise after another. Throughout his life, Shannon was drawn to physical activities that required dexterity and coordination. He was a gymnast in college, a juggler, and rode a unicycle. His wife, Betty, said he liked “circus things”. She bought him a unicycle one Christmas, which he mastered within a few days. He is famous for riding his unicycle down the hallways of Bell Laboratories while juggling, which I read they subsequently banned after Shannon was gone. He felt this kind of activity helped his intellectual work. In 1973, he even co-founded the Unicycling Society of America. Well, I used to ride a skateboard and I can juggle. As of a few years ago, I’m still (barely) able to juggle and pass 6 objects, such as plastic fruit, with one of my old roommates from MSU. But I've always had a hard time with mathematics, so maybe I need to try a unicycle.

Claude Shannon:

Shannon loved to build complex and odd machines. It might even be more correct to say that he worked professionally on the side. He was quoted in a 1987 OMNI magazine article, which I used to read, “I was always interested in building things with funny motions”. This goes back to his youth in Gaylord, Michigan, where most boys were probably more interested in hunting and fishing, but instead, he was busy building model planes, radios, a radio-controlled model boat, and a telegraph. So, at the University of Michigan in 1932, he said he had no hesitation about majoring in electrical engineering. The many unique things he built throughout his life included: rocket-powered flying discs; a motorized pogo stick; a flame-throwing trumpet; the “THROBAC” or “THrifty ROman-numerical BAckward-looking Computer” which calculated and displayed Roman numerals; the “Ultimate Machine”, a box with a switch on the side that when switched on would activate an arm to reach out from a lid on the box, switch itself off, and return inside the box; a mechanical version of W.C. Fields that could juggle; a machine that could solve the Rubik’s Cube; one of the first computer chess programs; and his famous mechanical mouse “Theseus” that learned how to navigate mazes with the help of telephone relay circuits. Shannon’s “ultimate machine” recreations have been popular in internet videos in recent years. He used to keep his, the original, on his desk at work. You can also watch video of his mouse “Theseus” in action with a detailed explanation provided by Shannon while he was working for Bell Laboratories. He also co-invented a wearable computer that he used while playing roulette in Las Vegas. By this point, he had gotten into game theory, which allowed him to do very well in the casinos and the stock market.

Shannon’s approach to his work was out of genuine curiosity and fascination with the complexities of the world, with a spirit of fun and playfulness. In later years, he witnessed what he felt was a lot of misinterpretation and hype regarding information theory, and he largely withdrew from public, wanting to avoid what he perceived as undue celebrity. He continued his pursuits for many years, but eventually was afflicted with Alzheimer’s disease and died in Medford, Massachusetts in 2001.

There are a few videos on YouTube with Claude Shannon and some of his creations.

Claude Shannon Demonstrates Machine Learning. In Their Own Words:

Claude Shannon Juggling:

Claude Shannon - Ultimate Machine - Leave Me Alone Box - um 1952:
A replication of Shannon's "ultimate machine".

Norbert Wiener
So who was Wiener? Norbert Wiener, the man credited with co-developing the Shannon-Wiener Diversity Index, was a boy genius from Columbia, Missouri of Polish, German, and Jewish ancestry. His father, a professor of Slavic languages at Harvard, was determined that his son should be a prominent scholar and home-schooled Norbert until he was 9. He graduated from high school at 11, earned a Bachelor’s degree in mathematics at 14, and a PhD in mathematics from Harvard at the age of 18. In 1914, he went off to Cambridge University in England and the University of Goettingen in Germany to study philosophy, logic, and mathematics. Wiener seemed to be interested in all kinds of things. He even studied zoology. At the outbreak of World War I, he returned to the US and taught philosophy at Harvard. He then worked as an engineer for General Electric, and wrote for the Encyclopedia Americana.

Young student Norbert Wiener. Polymath Matematica e ist:

Wiener had a strong sense of social justice and often acted on his convictions. He was briefly a journalist for the Boston Herald, where he wrote an article about the poor working conditions of mill workers in Lawrence, Massachusetts. He got fired from the paper because he refused to write favorable articles about a politician they wanted to promote. Although he was a pacifist, and perhaps driven by his experience of studying in England and Germany in 1914, he tried to enlist in the military a few times to help the US effort in World War I, but was rejected due to his poor eyesight. For a short time, in the summer of 1918, Wiener worked for the military privately with other mathematicians on ballistics at the Aberdeen Proving Ground in Maryland, but he still felt he should serve as a soldier, just as any other common man. He tried to enlist again and was finally admitted into the Army in 1918, but the war ended shortly after and he was discharged.

In 1919, Wiener started teaching mathematics at the Massachusetts Institute of Technology. While remaining with MIT, in 1926, he returned to Cambridge and the University of Goettingen as a Guggenheim Scholar where he had studied when he was 20. There, he worked on things like Brownian motion, the Fourier integral, Dirichlet's problem, harmonic analysis, and the Tauberian theorems. He became a professor at MIT in 1931, and worked there the remainder of his career. He said MIT gave him “… the encouragement to work and the freedom to think" in contrast to Harvard. Wiener became quite famous at MIT and enabled them to put together a research team that explored all kinds of things, including cybernetics, cognitive science, neuropsychology, and the mathematics and biophysics of the nervous system. Members of this MIT research group later made significant contributions to computer science and artificial intelligence. It seems there was just no limit to what some of these guys could do.

Young Norbert Wiener. Norbert Wiener Portraits:

It should be noted that Wiener was known for a diverse collection of mathematical notions that bear his name, including the Wiener process, the Wiener equation, the Wiener measure used to represent Brownian motion, the Wiener filter used to minimize noise in signals, and the abstract Wiener space, to name a few. Well, I’m not intimidated. I hypothesized and proved the existence of The Antelopean Surface as early as 1977. Wiener was attracted to quantum mechanics, collaborating with some of the leading researchers on relativistic quantum theory, the fifth dimension, unification of gravity and electromagnetism, and a theory of statistical hidden variables using differential space. Although I have a pretty good foundation in physics, I wish I could honestly say that I have a grasp on all that.

Professor Wiener in the MIT classroom with the tricycle cart.
Norbert Wiener, father of cybernetics and prophet forgotten:
Throughout his academic career, Wiener participated in many international gatherings and collaborated with academics across the world, especially several French mathematicians. Perhaps influenced by his father, Wiener was interested in foreign languages and preferred to exchange ideas with his collaborators in their native language. At least 16 of his articles are in French, five in German, and one in Spanish. He said, "One cannot understand a nation without knowing its language". He spent a year in China where he learned Mandarin. He spent a few years working in Mexico City, which had an important influence on his ideas about cybernetics. He continued to spend a lot of time in France, and in his later years lectured in the Netherlands, Italy, India, and Japan. He was a strong advocate for automation as a means of improving living conditions, and was very influential on the government of India in the 1950’s. All indications are that he truly had an international outlook on the world, and likely had a strong sense of being a citizen of the world, which unfortunately, sometimes brought him into conflict with the powers that be. Wiener was especially criticized for sharing ideas with researchers in the Soviet Union, which resulted in suspicions about him during the Cold War. After World War II, he became quite openly critical of political interference with research and the militarization of science, and especially of nuclear weapons. The January 1947 issue of The Atlantic Monthly featured his article, A Scientist Rebels, in which he urged scientists to consider the ethical implications of their work.

Norbert Wiener and Baidyanath swami at ISI, 1954 (Credits Indian Statistical Institute [ISI], Kolkata). Thinking Machines in the Physical World. IEEE 2016 Conference on Norbert Wiener in the 21st Century:

Norbert Wiener and Mrs Mahalanobis (2) at ISI, 954 (Credits Indian Statistical Institute [ISI], Kolkata). Thinking Machines in the Physical World. IEEE 2016 Conference on Norbert Wiener in the 21st Century:

Page of letter to his sister Bertha. Wiener Autograph Image:
Wiener seems to have always been a maverick in his own unimposing way, investigating and writing about the limitations of mathematical thinking early in his career, even as he was studying the depths of mathematics. Aspects of philosophy and a sense of social morality underlay much of his investigations, including cybernetics. In his last book, God & Golem, Inc., A Comment on Certain Points where Cybernetics Impinges on Religion, Wiener discusses the confrontation of technology and ethics, and the responsibilities of religion and politics.

As in his youth, Wiener’s interests remained broad throughout his life, including mathematics and the sciences of course, but also philosophy, literature, and fine arts. He wrote two short stories, a novel, and his autobiography in two parts, Ex-Prodigy: My Childhood and Youth in 1953, and I Am a Mathematician in 1956. He was described as being meditative, appreciating his personal solitude, which his wife helped protect from too many outside pressures. He enjoyed life in the country and spent his summers in “Tamarack Cottage” in South Tamworth, New Hampshire, often at a black board in the attic. Despite the many demands of his career, he was described as generous and once said, "I want to be the master of nobody". Wiener died in Stockholm, Sweden at the age of 69 in 1964.

Dr Norbert Wiener (credit Life Magazine) 
Enroque de ciencia. Norbert Wiener, un hombre despistado:

Unfortunately, I couldn't find any video featuring the actual Norbert Wiener, but there are a few videos about him that are interesting.

Norbert Wiener Today (1981):
This video is very well done. An actor portrayal of Wiener begins at 9:00. I was wondering why a guy from Missouri had a slightly British accent, but I supposed it could be a New England university thing.

Norbert Wiener:
This video is a little strange but has some good information.

Documental Norbert Wiener: 
This video is in Spanish, but you can set the subtitles to English and get a rough idea of what is said.

During his work with automation of anti-aircraft guns for the military in World War II, Norbert Wiener began developing what became cybernetics and information theory. Independently, Claude Shannon was also developing his ideas and is widely considered the father of information theory, the basis for all of our modern digital technology. As a student at MIT, Shannon met up with Wiener who was on the MIT faculty. This is apparently the basis of Shannon co-crediting Wiener with his development of his equation for entropy, which was eventually used for species diversity. Both Shannon and Wiener were known for being generous in crediting others for contributions to their ideas and work.

So, the short of it is, all of this was quite unexpected when I set out to read about the creators of an equation for species diversity. Some of the top minds in history are responsible for this little formula that I use. I can't help but project myself as I read about people, which maybe all of us do. Based on what little I know of each man on a personal level, I feel more of a kinship with Wiener, while I'd love to have the agile and upbeat mind of Shannon.

It appears Norbert Wiener was an ecologist after all.

As impressed as I am with Shannon and Wiener, after having read about them through several different sources, I have become a little more skeptical of solely crediting individual scientists with major accomplishments and labeling this man or that, “the father of” whatever. Sorry ladies, but it seems rarely in science have I read about “the mother of”. Clearly, certain people are responsible for major breakthroughs, but if you dig a little deeper you will often find how much they were influenced by someone else. There is the recurring phenomenon of two individuals, completely isolated from each other, hitting upon the same idea or discovery at about the same time, as though they are two particles in quantum entanglement. But, then there are those relationships that are obvious, where one person very clearly built upon the foundation created by someone else. So, not to take away from the impressive and fascinating achievements of Shannon and Wiener, but as I write, I constantly feel like maybe I should be reading more about the work of 10 or 15 other people who are mentioned in passing.