Developers
July 21, 2020

New models for Long Term memory To Advance AI

The progress of long term memory can improve computational systems, the development of AI, data analysis, and data prediction.
Source: Pixabay

Today we will talk about a new model that improves memory. Earlier this year, DeepMind introduced a new long-range memory model, called the Compressive Transformer, to advance the developments in memory models and language modeling. Comparing to computer memory, human capacity to remember is vast and varies a lot. We can remember things that happened minutes ago, months, years, even decades.

When we read a book, we can remember chapters that have already passed. We can even stop reading and at the time of picking up the book again remember the storyline. But is it possible to let computational systems select, filter and integrate data like the way human brain does? This is the goal of AI researchers and scientists want to achieve. Now, it’s getting more progress.

Brain, Sensory Perception, Memory

The brain receives information about the world through sensory perception. It then filters and selects the most relevant information to be processed. Based on this process, we can then think about past things and anticipate future moments.

We are not here to talk only about neuroscience, but it is relevant at the time of understanding what the next move in AI development is. Current AI researchers are working nonstop on the development of these abilities for computational systems and AI models.

It is not an easy task to convert a machine into a remembering and intelligent entity. In the past decades, this has advanced notably. The current focus is on how natural language modeling can design longe range memory systems.

Machines need better memory systems, better memory mechanisms to achieve the desired goal. What is this goal? Make machines reason like humans do.

The currently used memory architecture is the recurrent neural network (RNN), it is known as the Long Short Term Memory (LSTM). The LSTM has a compact memory which accesses read, write, and forget operations. It was developed based on synthetic tasks that learned logical operations as bits. It has become a model of data sequences. It recognizes handwritten notes and predicts kidney injuries.

The reason why LSTM is looked at as something to be improved is its reduced capacity.  Each unit of memory can influence every other unit of memory. This makes the system computationally inefficient.

Researchers that work at DeepMind have a solution for this memory shortage. The Differentiable Neural Computer (DNC) raises the LSTM memory capacity to new standards.

The DNC uses attention operations that reads from the memory directly. Memory models can attend particular events of the past. This attention operation needs some parameters. This way, the memory capacity of the model is increased.

Humans, use visual attention, where our eyes focus on certain objects in a specific visual scene. Certain factors change on what we focus, for example, emotion. If one is having an emotional moment, one may focus on the eyes of the other person more than in their clothing.

DNC shows promise in the translation and questions answering fields. These models can reason using two memory mechanisms: the LSTM memory and large external memory. Researchers at the Google Brain Team proposed the Transformer. This would remove the LSTM entirely, using only attention to transmit information.

Replacing the recurrent neural network

The transformer was proposed to take the place of the recurrent neural network. The network has many uses such as machine translation, applications in natural language processing, question answering, document summarization, sentiment classification, and modeling of natural language.

As AI continues to grow and be developed, the simile between a human mind and a thinking machine becomes closer. The purpose of AI is not to make a machine think like a machine, but to be able to reason like a human does. In one of its maximum capacities, to be able to relate to a human with developed intelligence.

There are not so many machine learning tasks that accelerate the development of better memory and at the same time make artificial intelligence better. For now, Statistical language modeling is the task that is considered a big possibility. ,

Language models predict the next word in a line of text in a sequential way. These models can be used to model existing texts and to generate new ones. As they model the past, their predictions become more efficient and precise. This relationship between past analysis and future forecast is based on logic, and it uses the same thought process.

In conclusion, some new models replace old ones when it comes to Memory and the use of AI. Current machines need better memory systems and memory architecture. Not in the sense of storing data, but in the sense of being able to remember like a human does. This next step in the evolution of Artificial Intelligence will make a quantum jump in how machines process data from the past and predict data for the future.

The current use model "Long Short Term Memory" LSTM, is proposed to be replaced by the Transformer. Neuroscience explains our ability to process information based and process on sensory information. Machines don't have senses, but the way they process data from the received data point to the processed data point can be replicated and that’s where the focus has to be to make AI produce the next evolution in Computer Science.

TagsAILong Term MemoryLong Short Term Memory (LSTM)
Lucas Bonder
Technical Writer
Lucas is an Entrepreneur, Web Developer, and Article Writer about Technology.

Related Articles

Back
DevelopersJuly 21, 2020
New models for Long Term memory To Advance AI
The progress of long term memory can improve computational systems, the development of AI, data analysis, and data prediction.

Today we will talk about a new model that improves memory. Earlier this year, DeepMind introduced a new long-range memory model, called the Compressive Transformer, to advance the developments in memory models and language modeling. Comparing to computer memory, human capacity to remember is vast and varies a lot. We can remember things that happened minutes ago, months, years, even decades.

When we read a book, we can remember chapters that have already passed. We can even stop reading and at the time of picking up the book again remember the storyline. But is it possible to let computational systems select, filter and integrate data like the way human brain does? This is the goal of AI researchers and scientists want to achieve. Now, it’s getting more progress.

Brain, Sensory Perception, Memory

The brain receives information about the world through sensory perception. It then filters and selects the most relevant information to be processed. Based on this process, we can then think about past things and anticipate future moments.

We are not here to talk only about neuroscience, but it is relevant at the time of understanding what the next move in AI development is. Current AI researchers are working nonstop on the development of these abilities for computational systems and AI models.

It is not an easy task to convert a machine into a remembering and intelligent entity. In the past decades, this has advanced notably. The current focus is on how natural language modeling can design longe range memory systems.

Machines need better memory systems, better memory mechanisms to achieve the desired goal. What is this goal? Make machines reason like humans do.

The currently used memory architecture is the recurrent neural network (RNN), it is known as the Long Short Term Memory (LSTM). The LSTM has a compact memory which accesses read, write, and forget operations. It was developed based on synthetic tasks that learned logical operations as bits. It has become a model of data sequences. It recognizes handwritten notes and predicts kidney injuries.

The reason why LSTM is looked at as something to be improved is its reduced capacity.  Each unit of memory can influence every other unit of memory. This makes the system computationally inefficient.

Researchers that work at DeepMind have a solution for this memory shortage. The Differentiable Neural Computer (DNC) raises the LSTM memory capacity to new standards.

The DNC uses attention operations that reads from the memory directly. Memory models can attend particular events of the past. This attention operation needs some parameters. This way, the memory capacity of the model is increased.

Humans, use visual attention, where our eyes focus on certain objects in a specific visual scene. Certain factors change on what we focus, for example, emotion. If one is having an emotional moment, one may focus on the eyes of the other person more than in their clothing.

DNC shows promise in the translation and questions answering fields. These models can reason using two memory mechanisms: the LSTM memory and large external memory. Researchers at the Google Brain Team proposed the Transformer. This would remove the LSTM entirely, using only attention to transmit information.

Replacing the recurrent neural network

The transformer was proposed to take the place of the recurrent neural network. The network has many uses such as machine translation, applications in natural language processing, question answering, document summarization, sentiment classification, and modeling of natural language.

As AI continues to grow and be developed, the simile between a human mind and a thinking machine becomes closer. The purpose of AI is not to make a machine think like a machine, but to be able to reason like a human does. In one of its maximum capacities, to be able to relate to a human with developed intelligence.

There are not so many machine learning tasks that accelerate the development of better memory and at the same time make artificial intelligence better. For now, Statistical language modeling is the task that is considered a big possibility. ,

Language models predict the next word in a line of text in a sequential way. These models can be used to model existing texts and to generate new ones. As they model the past, their predictions become more efficient and precise. This relationship between past analysis and future forecast is based on logic, and it uses the same thought process.

In conclusion, some new models replace old ones when it comes to Memory and the use of AI. Current machines need better memory systems and memory architecture. Not in the sense of storing data, but in the sense of being able to remember like a human does. This next step in the evolution of Artificial Intelligence will make a quantum jump in how machines process data from the past and predict data for the future.

The current use model "Long Short Term Memory" LSTM, is proposed to be replaced by the Transformer. Neuroscience explains our ability to process information based and process on sensory information. Machines don't have senses, but the way they process data from the received data point to the processed data point can be replicated and that’s where the focus has to be to make AI produce the next evolution in Computer Science.

AI
Long Term Memory
Long Short Term Memory (LSTM)
About the author
Lucas Bonder -Technical Writer
Lucas is an Entrepreneur, Web Developer, and Article Writer about Technology.

Related Articles