The Beginning

The origins of Neural Networks date as far back in time as the 1950s. In the year 1943, neurophysiologist and cybernetician Warren McCulloch, known popularly for certain brain theories, along with logician Walter Pitt who worked in the field of computational neuroscience wrote a paper titled “A Logical Calculus of Ideas Immanent in Nervous Activity”. The groundbreaking work went down in history to become the very first mathematical model of a neural network. This work paved the way for research in both biological processes and applications of Neural Networks to Artificial Intelligence.

In 1949, Donald O. Hebb in his work “The Organization of Behaviour” pointed out that humans learn by repeated use of neural pathways. He argued that when two nerves fire at the same time the connection between them is enhanced. This came to be known as Hebbien learning which is unsupervised learning. W. A. Clark used Hebbian learning in the computational machines called “calculators”, to simulate a Hebbian network in 1954. By 1956, more neural network computational machines were still emerging.

The Perceptron

On July 13, 1958, the New York Times reported “The Navy last week demonstrated the embryo of an electronic computer named the Perceptron which when completed in about a year is expected to be the first non-living mechanism able to ‘perceive, recognize and identify its surroundings without human training or control’.”

In the year 1957, Frank Rosenblatt invented the Mark 1 Perceptron machine, based on the findings of McCulloch and Pitt. The perceptron was designed specifically for image recognition. It was a physical neural network where the binary neural units were connected through adjustable weights. It seemed to be what one could call the mechanical representation of a human brain.

Fig: The Perceptron

Architecture: 

The Mark 1 perceptron, now kept in the Smithsonian National Museum of American History is a binary classifier with 3 layers:

1. S-units: Each input is connected to a set of sensory units (S-units) or input retina. Each S-unit corresponds to an input feature and is connected to the input neurons. The number of input neurons equals the input features.
2. A-units: A hidden layer containing 512 perceptrons that play a crucial role in learning and processes form the Association units (A-units). This layer contains the weights that are adjusted based on the learning rate, the summation function to produce the weighted sum, and the activation function (step function) to produce the output.
3. R-units: The output layer of 8 perceptrons forms the response unit (R-unit). The layer produces the final output which processes the result of the activation function.

Fig: The Mark 1 Perceptron Architecture

The perceptron, however, was found to be flawed as it would sometimes fail even on extremely simple tasks like the exclusive-or problem as pointed out by computer scientists M. L. Minsky and S. A. Papert. Only the use of multiple hidden layers could fix this problem, but no algorithm could run such a neural network.

It took 17 years for such an algorithm to be devised and it was none other than “Back Propagation”. However, it was later known that the concept of backpropagation had already been invented when the book was published. Now, with the newfound algorithm that paved the way for neural networks to theoretically approximate any function, the multi-layered perceptrons were finally trained to yield better results.

Follow my blog to dive deep into the world of the development and working of deep learning and neural networks.