Welcome to this new episode of The Context. Today, I want to talk to you about Jolting AI. The increasing rate of acceleration in artificial intelligence applications. I often talk about accelerating change. Which measures the rate of change in a given period of time and then comparing the rates of change tries to interpolate and to understand what are the phenomena underlying it.
15863358630000000
The visual representation in various kinds of charts of this rate of change. Of course depends on what you want to highlight. When the rate of change is very small and the variation in the rate of changes also small then perhaps a linear chart is going to be fine. On the y-axis, you will have units of whatever you want to represent: 1, 2, 3, 4, 5. However, when we have an accelerating rate of change typically we use a logarithmic chart where the y-axis represents orders of magnitude. Every unit is going to be an increasing order of magnitude: 1, 10, 100, 1,000 and so on. So when we are talking about an accelerating change the mathematical function that embodies this is the exponential function. For example, two to the power of x. And the exponential function will be a rapidly increasing curve, when represented on a linear chart. But when represented on a logarithmic chart the exponential function will be a line.
15863358630000000
Now when we are talking about the technologies that enable society to take advantage of various types of innovation, which is in turn analyzed and expressed in terms of accelerating technological change, we must of course reference the law of accelerating returns that was formulated by Ray Kurzweil in his book of 1999, The Age Of Spiritual Machines. And then of course again further analyzed and and represented in various ways in for example, The Singularity Is Near.
15863358630000000
What Ray formulates is that the traditional understanding that we have, a diminishing return from increasing investment in a given technology in a given industry, is true on a small scale, but if we look at a larger scale we can actually observe the opposite. An equal investment will generate a higher than expected return. So how can both of these be true?
15863358630000000
What happens is that we have for any given technology the traditional S curve of a technology be being experimented with, then an ever increasing understanding of how the technology can be exploited and applied. When we squeeze out every possible advantage of the technology, we will have a plateau where effectively further investments are not going to provide the kind of returns that we expect and this S curve is for any given single technology.
15863358630000000
But the law of accelerating returns, the exponential curves that we talk about when we talk about accelerating technologies looks at a successive technologies substituting each other and designing the curve that we are finally looking at. One of the most famous examples of using this kind of paradigm is more slaw that formulated over 50 years ago said that electronic circuits would.
15863358630000000
05:55
15863358630000000
Double the density of their components every 18 months and then adjust it to every two years. And this is not a natural law, it is a self fulfilling prophecy a projection of our desires and expectation of of our abilities. In order for many competing teams around the world to strive to be the first to arrive at a given breakthrough and.
15863358630000000
06:29
15863358630000000
Ray actually generalized this and looked. At many technologies that preceded the sixties of the 20th century when Gordon Moore in formulated his observation he looked at electromechanical relays he looked at the old tubes he looked at many things that went through their individual as curves and then smoothly one on after another fulfilled the same kind of expectation of doubling the competition.
15863358630000000
07:09
15863358630000000
Al power in a given period of time, that could have been. More or less what more slaw also formulated. So more slaw applies to transistors that are then put together in integrated circuits that in turn form the CPUs the central processing units that we have in our computers in our mobile phones in our servers where we connect.
15863358630000000
07:45
15863358630000000
Over the internet when we browse web servers and so on. And an important. Additional. Component in understanding what is going on in the world of technology is what is called the innovators dilemma? Formulated by Clayton Christensen who recently passed away. It is the pretty dramatic decision that an industry.
15863358630000000
08:21
15863358630000000
Leader has to make to stop serving its current set of customers. And invest instead in the development of the technologies that when put into a given set of products will serve their future customers and the dilemma is in the fact that any investment in that future is subtracting resources in investing in the already successful present.
15863358630000000
08:58
15863358630000000
And the short-sightedness of those leaders that can't understand the necessity of this. Stems from the fact that if they don't do it, somebody else is going to do it so they will be disrupted and they will stop being the leader of that future generation of products. And it is indeed the case that it is very difficult to disrupt oneself it is extremely difficult to say I will take away resources from improving my current generation of products because I realize that I have to embrace a new maybe unproven technology.
15863358630000000
09:53
15863358630000000
Because I understand I believe in the fact that it is going to be. An essential technology of the leading solutions of tomorrow. So. Even though. Intel is the leader an undisputed leader of the era of CPU computing of personal computers of servers intel is not a leader in the next generation of computing that we are already seeing around us which is based on GPUs in graphical processing units the leader of that is Nvidia.
15863358630000000
10:44
15863358630000000
GPUs are made of transistors just like CPUs but their architecture is massively parallel, they are optimized rather than executing an arbitrary kind of program that is taken sequentially their architecture is optimized for executing those kinds of programs that can be broken down in hundreds or thousands of or millions of similar parts that are executed.
15863358630000000
11:19
15863358630000000
Simultaneously across the architecture of the chip. The simplistic example of these kinds of programs are the video games that we play where calculating and designing and representing the scene whether we are fighting against aliens or driving in a car simulator in a racing game or any other kind of graphically intensive task.
15863358630000000
11:54
15863358630000000
The kind of calculations that the computer has to make are basically the same pixel by pixel not the result of the calculations but a kind of calculations. Are in some other companies recognized this early and they created chips specialized for these graphically intensive tasks. And they became important in that they became a leader in that.
15863358630000000
12:32
15863358630000000
Now. It is today. Frequently reported in mainstream media articles, or even some specialized articles that more slaw is ending. And that is mistakenly equated to innovation in computers ending as well. Now, the era of traditional CPUs becoming ever more powerful may be. Progressing towards an end for many reasons and but the age of innovation in computers is definitely not.
15863358630000000
13:21
15863358630000000
So let's get back to what I started with. The increasing rate of acceleration in artificial intelligence. About ten years ago. It was observed and then fully embraced that the then leading type of AI. Architecture was approachable efficiently by using GPUs. That kind of approach is still the leading approach today.
15863358630000000
14:01
15863358630000000
It is a subset of machine learning called artificial neural networks and especially deep learning which is the type of artificial neural networks where there are many many layers hundreds or thousands or tens of thousands of layers connecting the inputs with the outputs. And each of these layers makes some calculation on the data and passes the calculation on to the next layer and the calculation and the optimization and then the execution of the optimized what we say the trained neural network is extremely efficient.
15863358630000000
14:54
15863358630000000
If rather than run on traditional CPUs if run on GPUs instead. So what? Is happening is that there is a progressive learning curve. That is applied not only within a given technology but across generations of technologies and this learning curve enables the acceleration of change except that we are not talking about the mirror acceleration of change anymore we are talking about an increasing rate of acceleration that derives from the learning curve being applied across many generations of technologies.
15863358630000000
15:49
15863358630000000
And the rapid rapid coming together and employment of that learning by not only the specialists in hardware, but also the specialists in software in infrastructure architectures and so on. Actually in terms of technology generations, we are now talking about specialized. AI chips that go beyond optimizing the architecture of the chip and the integrated circuit using cell transistors.
15863358630000000
16:31
15863358630000000
Not only recognizing that the parallel nature of GPUs is is great but that we can go beyond and implementing in hardware the kind of calculations that artificial neural networks apply the in deep learning need achieving either even greater results, so for example. Google designed such a chip and they are calling it TPUs.
15863358630000000
17:06
15863358630000000
Tensor processing units from the name of the mathematical calculations that these specialized the chips have to execute so we we went from CPUs to GPUs and now specialize the AI chips such as for example, the TPUs. Now. A few months ago. Stanford University published their. Hundred plus page report on the state of the AI industry and on page.
15863358630000000
17:47
15863358630000000
I think 65 or something of that report they. Published a chart that. Represented how the rate of doubling in the performance of the computer infrastructures if we take into account the infrastructure available for artificial intelligence applications. Changed. From following more slaw for the past 50 years to following a different curve over the course of the past 10 years.
15863358630000000
18:33
15863358630000000
And they calculated the amount of computation available compute in terms of the global infrastructure that a given set of problems require. Expressed in petaflop per second days and this unit of measure is similar to what you would look at in the energy consumption of your home where the power available is expressed in kilowatt and kilowatt hours is the amount of energy that your house consumes and and needs in order to function.
15863358630000000
19:22
15863358630000000
So a typical Western European house has three kilowatt of power available and then if you turn on your washer and dryer and your hair dryer and your dishwasher at the same time. You often end up exceeding that power available and and your energy provider will trigger a breaker circuit and you will realize oh my god, I have to turn some appliance off.
15863358630000000
19:59
15863358630000000
So similarly to that we can look at what are the applications that we can practically and usefully attack. At the availability of a given power in computation, how many petaflops per second we can deploy. If we need 10,000 years to complete a task, we will just not do it.
15863358630000000
If we can train a neural network to solve a given challenge fast enough, and then that neural network can be applied to the task after being trained usefully, all of this within budget and within a given amount of time then we can actually solve problems that were unsolvable before.
15863358630000000
So if we were on the curve of Moore’s law exponential on a linear chart linear on a logarithmic chart. Over the course of the past eight ten years, we would have seen. A 7-fold improvement in this availability of compute and in the type of problems that as a consequence we are able to attack.
15863358630000000
Instead as Stanford University mapped the availability of. Computer power and the application of that computer power how much we were able to dedicate to a given problem set. They saw that over the course of the past eight years which they went to look. We had a three hundred thousand fold improvement.
15863358630000000
So they took a given route and together with Sanford University, also OpenAI, another organization dedicated to the analysis and implementation of advanced artificial intelligence applications looked at this dataset and they said okay, let's do a linear interpolation of the first set and then another linear interpolation of the second set. They concluded that the doubling used to be two years according to Moore’s law before, and now the doubling is between three-four months.
15863358630000000
Now, why not? That kind of simplistic approach is possible. But I propose a little more sophisticated approach rather than doing a linear interpolation, we can draw an exponential curve on the logarithmic chart and say that what we are looking at is the increase in the rate of acceleration of our computing infrastructure when we take into account the latest software and hardware architectures.
15863358630000000
That is why I say that AI is jolting. Jolt is the first derivative of acceleration. Jolt represents an increasing rate of acceleration and what we are seeing today, is that AI is jolting. So what are the consequences? What does this imply? First of all that we can expect, even within the current set of applications, potentially the rate to increase further. But if my paradigm is correct, what we have to watch out for is that there will be an even more important increase in the availability of compute for next generation applications. Will that doubling increase be one month instead of three, will that be one week? What will that mean and when will that be available?
15863358630000000
We have to study the numbers better we have to try and forecast what kind of software and hardware components are going to be available, but I expect that we will have this new disruption. Somebody like Stanford or OpenAI will once again simplistically represent it through another linear interpolation, but it is more appropriately represented by an exponential curve on the logarithmic chart.
15863358630000000
Quantum computers are going to be applied to AI problems or maybe the reverse when we will use AI systems to design better quantum computers. There are already teams that are studying what each of these could be, what does it mean to design a neural network that runs natively rather than on GPUs or TPUs AI chips, that runs on quantum computers.
15863358630000000
Quantum computers are so massively parallel as to require an entirely new understanding of how the universe works. Or rather multiple universes since one of the interpretations of quantum phenomena is the multiverse view of the universe. What does it mean to structure an AI application such that the output of that AI application is a better quantum computer? Most likely that AI application will already be running on a quantum computer.
15863358630000000
The technological singularity is the hypothetical moment in time in a future when the rate of change in the world is such that unaided humans are unable to comprehend it and whether it comes from self-modifying artificial intelligence leading to the so-called intelligence explosion, whether it comes from other factors, it feels a little bit like we may starting to be there.
15863358630000000
Because we haven't been designing microchips with pencil and paper for a long time. We haven't been programming line by line for the past 10 years. First we gave up designing hardware ourselves, we used computer aided design, we used computers to design hardware.
15863358630000000
Now for the past 10 years we gave up, willingly so in order to be more effective and more efficient, designing software. We use neural networks that are designing the software instead. So, when this comes together, and software designing hardware designing software is going to be applied to a rapidly increasing set of problems, that is in many ways what we can call the singularity.
15863358630000000
| Word Count | Unique Lemmas | Characters | Lemmas Density |
0 |
0 |
0 |
0 |
| Modularity | 0 |
Influence Distribution | 0 % |
|---|
| Topics | Nodes in Top Topic | Components | Nodes in Top Comp |
0 |
0 % |
0 |
0 % |
| Nodes | Av Degree | Density | Weighed Betweenness |
0 |
0 |
0 |
0 |