The history of AI is not a single replacement chart
If you think of AI's history as a single evolutionary chart — old AI, then Machine Learning, then Deep Learning, then Transformer, then LLM — you lose sight of important differences.
That's because this sequence mixes together different kinds of concepts:
- broad research fields
- learning methods
- types of models
- architecture
- groups of models in use
Also, the fact that a new method appeared does not mean every earlier method disappeared. Rather than memorizing a complete history of AI, this chapter looks at several of the threads that lead to today's AI, as a map.
Approaches based on symbols and rules
AI includes an approach that represents knowledge as symbols and rules and solves problems by manipulating them. This approach based on symbols and rules has held an important place since the early stages of AI's history. Expert systems, which represent specialized knowledge as rules, are a representative example. The later spread of data-driven learning methods did not mean that approaches based on symbols and rules completely disappeared. Research still exists today that tries to combine symbolic thinking with neural networks. So we avoid the understanding that symbolic AI completely disappeared once statistical AI appeared.
Learning from data
Rather than a human writing every rule, methods that learn regularities and tendencies from data also developed. In natural language processing in particular, statistical methods using data such as corpora became prominent from the late 1980s into the 1990s. Here too, there's no need to think that "the old method was completely replaced by the new method." What matters is that the emphasis in how problems were approached shifted.
Neural networks predate Deep Learning
The idea of neural networks itself was studied long before today's Deep Learning. Early models were proposed in the 1940s, and the perceptron appeared in the 1950s. In the 1980s, research on back-propagation — adjusting weights in networks that include hidden units — also held an important place. So it is not appropriate to understand this as Deep Learning appearing and neural networks being born as a result.
Neural network research went through periods of rising expectations as well as periods where technical limitations were strongly felt. That said, care is needed not to treat any single paper or book as the sole cause of a stagnation or revival in the entire research field. Technical limitations and historical changes in research, society, or funding are not the same question.
Deep Learning is not cut off from neural networks
Deep Learning is not an unrelated, separate line of technology from neural networks. It can be understood as a development of neural networks that learn representations using multiple layers. From the 2000s into the 2010s, Deep Learning drew increasing attention alongside changes in available data, computational resources, and learning methods. AlexNet in 2012 is often cited as an example that greatly raised attention toward Deep Learning in computer vision.
However, this single example does not let us conclude that "AI began in 2012" or that "every prior method became unnecessary."
Transformer as an architecture
The Transformer was published in 2017. It was presented as a neural-network architecture for handling sequences. What matters here is that Transformer is not the name for AI as a whole, and Transformer and LLM are not the same word either. Various models use Transformers. Some models, such as BERT, use Transformers while serving purposes different from the conversational, generative LLMs commonly imagined today. On the other hand, many of today's well-known LLMs use Transformer-family architectures. In other words, we do not treat Transformer as being equal to LLM.
What changed in the LLM era
Modern LLMs involve multiple factors beyond just Transformer-family architecture, including:
- model scale
- Pretraining
- available data
- behavior shaped by instructions or examples in context
So it is not accurate to say that "Transformer simply became LLM." The discussion of architecture, the discussion of model scale, the discussion of learning method, and the discussion of behavior at use time need to be kept separate.
Seeing history as a "map"
The flow so far is not a single, linear path like this:
Rather, approaches based on symbols and rules, statistical methods, neural-network methods, Deep Learning, Transformer, and LLM overlap with one another in different relationships. These labels include broad research fields, learning methods, types of models, architectures, and groups of models currently in use. Simply lining things up in chronological order cannot capture this difference.
Chapter summary
This chapter did not treat AI's history as "a history of new technology sequentially erasing old technology." Keep the following points in mind:
- thinking based on symbols and rules did not simply vanish once later AI methods appeared
- statistical methods and neural-network methods overlap, and cannot be organized by a simple sequence alone
- neural networks were studied before Deep Learning
- Deep Learning has a continuous relationship with the development of neural networks
- Transformer is an architecture, not LLM itself
- the changes of the LLM era involve multiple factors beyond architecture, such as scale and Pretraining
- AI's history is not a single replacement chart
The next chapter returns from history to the present-day use case, looking at what we can actually confirm from the outside when we use an AI service.