Google developing next-generation AI

Google’s Pathways developing new multipurpose neural network architectures. 

Today’s neural networks are aimed at performing a single task. For example, a neural network used to correct spelling errors can be retrained to detect grammatical mistakes, but it’s likely to “forget” its previous functions of how to correct misspellings.

According to Jeff Dean, the Senior Vice President of Google Research, instead of extending existing models to learn new tasks, they train new models from scratch to complete a single task, which takes more time and also uses more data.

Neural networks must perform a variety of diverse tasks, the company believes.

The neural network architecture of the Pathways project will not only be multitasking; the model will be trainable. As Jeff Dean noted, they plan to train the model to use and combine already developed skills to learn new tasks faster and more efficiently. For example, how aerial images can predict the elevation of a landscape – could help it learn another task — say, predicting how flood waters will flow through that terrain. Dean compared this technology to mammalian brains that can generalize different tasks. 

Google developers promise that Pathways will enable multimodal models that can simultaneously process and understand incoming visual, acoustic and language data. For instance, hearing someone say the word "leopard" or in general, dealing with this word, the machine could analyze the video of a running leopard, and the reaction would be the same in all cases: the machine would recognize the idea of a leopard. 

Project development solves another important problem — today's machine learning models expect the involvement of all neural nodes of the network, regardless of how simple or complex the task is. Google believes that it is possible to achieve "sparse" activation; it means to direct new tasks only to separate arrays of neural nodes. This approach is also much less energy-consuming.

Thus, the neural network will consume only one tenth of the energy that would have to be spent on a more traditional neural network, where all neural nodes are activated at once. This concept is already being applied in projects like Google Switch Transformer, Models for Natural Language Understanding, and Gshard.

Google has high expectations for the Pathways project - in the future it should lead to the creation of neural networks that can perform millions of different tasks, conduct generalization between them, combine them as needed, understand different types of data and work with them more efficiently than any network of theirs today. The goal of the project is to move “from the era of single-tasking models that simply recognize patterns, to multitasking intelligent systems that reflect a deeper understanding of our world and can adapt to new needs. 

Georgy Lagoda, Deputy General Director of the Software Product Group of Companies, believes that the stated goal looks exceptionally ambitious, but with all this, Google may well have the resources to implement it. Optimization of modern approaches to the creation and training of neural networks requires a radical revision of existing architectures.

Lagoda believes that if Google succeeds in realizing this idea, then it will be a significant step towards true artificial intelligence. Indeed, today there are no such systems.