Introduction
I’m going to try to sum up in a few simple paragraphs the problems I see with A.I. as it exists today and why I’m skeptical of many of the current claims and predictions.
Nothing New
The technology being used by existing A.I. goes back roughly to the mid 1980s when work by Rumelhart and others on the Back Propagation Algorithm led to supervised learning for more complex data. For instance, handwriting could be reliably read by machine for the first time. Today’s work (more or less) does this at a much larger scale. There are no particular patents or trade secrets indicating something significantly new has been discovered to drive the current AI enthusiasm.
Battle of the Giants
This isn’t the old tech story of two guys in a garage scaling up a new idea. Todays AI competition is some of the largest companies in the world trying to outspend each other. There is a sense that it will be Winner Take All like other areas of tech. But this isn’t Microsoft or Dell or Cisco in the 1980s. There may be many “winners” given the lack of any “first mover” advantage, or any advantage, for that matter.
Who’s Gonna Pay for This?
As billions get spent on AI, we still don’t see any applications that would cover the huge expense. Not self driving cars. Not chatBots. It could happen at some point in the future but to even cover the power bills for some of these large AI data centers makes profitability difficult. More on that later.
Accuracy Issues
AI and similar technologies are great for “faking it” in the absence of actual data. There are places in the world where this sort of behavior is useful. I’m thinking mostly entertainment related applications, like movies or even music, where there may not be a “right” or “wrong” solution. Most of the world, especially the computing world, wants accurate answers. Faking it might be ok, at least for a while, in some contexts. Customer support chatBots is an example. But how much bad information is going to be allowed? And can we even measure or control this?
The Why
AI can often give answers but usually can’t tell exactly why those answers were given. And even if we find an error, there isn’t much that can be done to correct such errors. In particular any change to fix one error may introduce other errors in other places. There is no easy way to test large scale AI and to address deficiencies.
Algorithmic Complexity
Because existing AI technology goes back decades, why the big burst of excitement now? Well basically AI has been used quite successfully to solve smaller problems until now. With large data centers, compute power to tackle larger problems, particularly things like video, has led to renewed interest in AI.
While “Deep Fake” images and videos are impressive demonstrations, they aren’t an obvious source of large scale revenues. They also happen to be expensive to produce, at least today. But what about the future?
At the lowest level, AI algorithms are all more or less related to Matrix Arithmetic, which has a mathematical complexity of O(n^3), or cubic complexity. This means to get 2x the performance (either 2x the speed or a result 2x larger or some combination) requires 2^3 or 8x the compute power.
This is a difficult problem of scaling. While it took decades to get from handwriting recognition to deep fake videos, there were enormous gains in computing power over that time. The next gains won’t be so easy.
Moore’s Law
A long standing “law” of semiconductors is Moore’s Law which says, more or less, that we can double the number of transistors on a chip every 18 months or so.
This isn’t a “law” like the laws of physics, It is really more of a guideline and perhaps even a form of corporate collusion. If all the people involved in making semiconductors aim at this 18 months target for doubling the transistor count on a chip, it will happen. Nothing really magical. Of course it could go slower or faster. It mostly depends on the amount of investment.
This 18 month doubling is perhaps unprecedented in human history. It has led to the rapid increase in computing power witnessed in the last half century. The mobile phone in your pocket today has more compute power than the largest most expensive computers in the world just a short time ago.
Dennard Scaling
Computer chips are made of transistors. Millions, sometimes billions of transistors. They take electricity to run, but for decades a chip took more or less the same amount of power, no matter how many transistors it contained. This is because as the transistors shrank so did their operating power. This is known as Dennard Scaling,
Unfortunately, the power the transitors consume when doing nothing at all is a constant. As the number of transistors grew large so did this “leakage” power. This leakage is a very small number per transistor and was safely ignored for decades, but it adds up when the number of transistors gets large.
By about 2000 the handwriting was on the wall. Many people knew about this but it was first popularly documented by Trevor Mudge in his paper “Power: A First Class Architectural Design Constraint“. Instead of being limited by how small transistors could be made, future computers would be limited by how much power they could safely dissipate.
The Data Center Arms Race
Compute power now scales with actual electrical power. For decades you could essentially double your computing power and keep the same electrical power by waiting 18 months and buying the next generation semiconductors. Those days are gone. The good news is your desktop PC and even your mobile phone doesn’t have to be upgraded as often. The bad news is faster computers will now take more power.
This isn’t such a problem for desktop PCs where there is no real need for large performance increases. In the data center, however, the need for higher performance to support new AI workloads is causing a proportional increase in power consumption. To get 2x the raw performance will take about 2x the (electric) power. For 2x the performance on AI workloads we can expect 8x (electric) power consumption. This is why data centers for large AI players are exploding in size and power consumption.
Nowhere Left to Go
Where does the AI race go from here? It’s anyone’s guess. My personal concern is that this is an “arms race” building weapons for a battle that may never occur. Right now AI needs some highly profitable application or applications to emerge to pay for these data centers and to pay their large one-time build out costs and ongoing electricity bills.
And should the Winner Take All model be the actual outcome, there will be several losers with very expensive data centers sitting idle. Those losses will have to be absorbed elsewhere by the corporations or somehow in the larger economy, or both.
Right now AI isn’t a market. There is essentially one hardware vendor making vast amounts of money, and a few large corporations spending that vast amount of money on data centers using this hardware, with little to show for it. When will we see businesses and individuals begin to use, and pay for, AI remains to be seen. And even as the capital costs of the data centers are paid off, the large electric power bills will also have to be reckoned with. Where and how businesses will make profitable use of AI given these constraints is still the open question.
Steven A. Guccione 