Why do you assume that the current economics, caused directly by our level of technology, will remain the same? Are we at the level of "Everything that can be invented has been invented"? Unlikely.
Junior programmers can now progress through understanding & experience much faster than it used to take. LLMs are pretty good always-available computer science and software development coaches. I suspect there will be waves of senior-ish developers who learn quickly this way.
Hi Stephen, I'm not sure I made the assumption that you're ascribing to me. But if there's going to be some kind of economic revolution based on a new level of technology, it had better arrive pretty soon. Your position seems like a version of "And then a miracle occurs...."
I'm not ruling that out, just saying that it seems unlikely. As far as I can tell, everyone in the AI industry is betting on some version of "maybe this will all work out somehow," but no-one sees it as their problem to actually solve.
Meanwhile, the losses keep piling up and the aggregate numbers keep getting bigger. Maybe you're right and it does all work out, but (a) simple-minded me doesn't see how, and (b) that seems like a form of blind faith right now, and should be acknowledged as such.
It is quite common for people to appreciate recent & cumulative technical advances yet not anticipate that more advances arrive constantly, although not predictably. But it is predictable that new advances of some kind will keep arriving. Based on early Internet economics (at each stage from the early 90s through 2010 at least), Internet streaming of video or even just audio made no economic sense. There was no way it could be profitable. It seemed like the costs would always be prohibitive. I actually pointed that out at one point. Yet here we are: Free YouTube streaming is one of the highest sources of media consumption. We don't even bother with much audio streaming: Just send videos of talking heads because it apparently costs the same.
About 20-25 years ago, on the FoRK mailing list, a lot of smart PhD friends were arguing about Peak Oil and the necessary decline the world would have. The point I made is that, generally, society (especially certain inventive societies) perform just-in-time innovation: We solve problems about the time that we need to solve them. Sometimes a little ahead, sometimes with some lag. But we always come up with a solution. We clearly did that with oil, and in parallel got renewables & EVs off the ground. Just in time.
Right now, machine learning is running on a handful of working solutions. Everything has scaled up because it is amazing that it works as well as it does. There is some serious friction for finding better methods, but huge benefit from doing so. We have human (and animal) brains as an existence proof that a much better mechanism is possible. We can't easily delve into the wetware solution space. But it is quite clear that we will find much more efficient methods soon. No way to tell how the market will evolve with that. But it certainly will happen, and relatively soon. We are at the Singularity after all: Virtually no predictions of the past are applicable now. Our best predictions now are that the future is unpredictable.
We are already in the age of miracles: Machine Learning works, far beyond what anyone expected this soon. Follow on miracles are not that surprising.
I look at it this way: Already several open models are quite useful for a broad range of tasks. More and more of those models are reasonably able to run on a local computer. Last year, I built myself a very powerful (and expensive) ML system. I can now run unlimited ML operations on that computer for just the price of electricity and insurance. For me, the current economics are working very well, even before the next miracle.
This is what I could afford personally: 2 RTX PRO 6000 GPUs (96GB VRAM each), AMD Ryzen Pro 64c/128t, 768GB DDR5 6400Mt/s RAM, and a lot of fast storage with 10Gbps nics. With some approaches, that can run 600B parameter models at a good speed. I calculated that at typical cloud GPU prices, the breakeven is 2-12 months. I haven't switched coding to that, but it looks promising so far. There are new options every day.
Terrifyingly precise analysis, Mark. The point about economics is the elephant in the room—right now, the tech is being heavily subsidized by VC burn-rates, and the cliff edge is real. But your point on expertise is what really resonates. We are essentially generating technical debt at unprecedented speeds, and the burden of validation is falling on a shrinking pool of senior talent. If we stop training juniors, we’re eating our own seed corn. It feels like we are rapidly heading toward a world where 'prompting' is easy, but verifying the output is everything. I fully expect to see LinkedIn job postings for 'AI Bullshit Manager' popping up any day now.
Hi Bob, the problem of eating seed corn seems to be a repeating problem in CS. Maybe it happens in other fields too, but I know I've seen other versions of this particular movie before (the first time when I was a mere undergrad in the '80s). I expect we'll sort it out eventually, but it might get a little ugly first.
Why do you assume that the current economics, caused directly by our level of technology, will remain the same? Are we at the level of "Everything that can be invented has been invented"? Unlikely.
Junior programmers can now progress through understanding & experience much faster than it used to take. LLMs are pretty good always-available computer science and software development coaches. I suspect there will be waves of senior-ish developers who learn quickly this way.
Hi Stephen, I'm not sure I made the assumption that you're ascribing to me. But if there's going to be some kind of economic revolution based on a new level of technology, it had better arrive pretty soon. Your position seems like a version of "And then a miracle occurs...."
I'm not ruling that out, just saying that it seems unlikely. As far as I can tell, everyone in the AI industry is betting on some version of "maybe this will all work out somehow," but no-one sees it as their problem to actually solve.
Meanwhile, the losses keep piling up and the aggregate numbers keep getting bigger. Maybe you're right and it does all work out, but (a) simple-minded me doesn't see how, and (b) that seems like a form of blind faith right now, and should be acknowledged as such.
It is quite common for people to appreciate recent & cumulative technical advances yet not anticipate that more advances arrive constantly, although not predictably. But it is predictable that new advances of some kind will keep arriving. Based on early Internet economics (at each stage from the early 90s through 2010 at least), Internet streaming of video or even just audio made no economic sense. There was no way it could be profitable. It seemed like the costs would always be prohibitive. I actually pointed that out at one point. Yet here we are: Free YouTube streaming is one of the highest sources of media consumption. We don't even bother with much audio streaming: Just send videos of talking heads because it apparently costs the same.
About 20-25 years ago, on the FoRK mailing list, a lot of smart PhD friends were arguing about Peak Oil and the necessary decline the world would have. The point I made is that, generally, society (especially certain inventive societies) perform just-in-time innovation: We solve problems about the time that we need to solve them. Sometimes a little ahead, sometimes with some lag. But we always come up with a solution. We clearly did that with oil, and in parallel got renewables & EVs off the ground. Just in time.
Right now, machine learning is running on a handful of working solutions. Everything has scaled up because it is amazing that it works as well as it does. There is some serious friction for finding better methods, but huge benefit from doing so. We have human (and animal) brains as an existence proof that a much better mechanism is possible. We can't easily delve into the wetware solution space. But it is quite clear that we will find much more efficient methods soon. No way to tell how the market will evolve with that. But it certainly will happen, and relatively soon. We are at the Singularity after all: Virtually no predictions of the past are applicable now. Our best predictions now are that the future is unpredictable.
We are already in the age of miracles: Machine Learning works, far beyond what anyone expected this soon. Follow on miracles are not that surprising.
I look at it this way: Already several open models are quite useful for a broad range of tasks. More and more of those models are reasonably able to run on a local computer. Last year, I built myself a very powerful (and expensive) ML system. I can now run unlimited ML operations on that computer for just the price of electricity and insurance. For me, the current economics are working very well, even before the next miracle.
This is what I could afford personally: 2 RTX PRO 6000 GPUs (96GB VRAM each), AMD Ryzen Pro 64c/128t, 768GB DDR5 6400Mt/s RAM, and a lot of fast storage with 10Gbps nics. With some approaches, that can run 600B parameter models at a good speed. I calculated that at typical cloud GPU prices, the breakeven is 2-12 months. I haven't switched coding to that, but it looks promising so far. There are new options every day.
Terrifyingly precise analysis, Mark. The point about economics is the elephant in the room—right now, the tech is being heavily subsidized by VC burn-rates, and the cliff edge is real. But your point on expertise is what really resonates. We are essentially generating technical debt at unprecedented speeds, and the burden of validation is falling on a shrinking pool of senior talent. If we stop training juniors, we’re eating our own seed corn. It feels like we are rapidly heading toward a world where 'prompting' is easy, but verifying the output is everything. I fully expect to see LinkedIn job postings for 'AI Bullshit Manager' popping up any day now.
Hi Bob, the problem of eating seed corn seems to be a repeating problem in CS. Maybe it happens in other fields too, but I know I've seen other versions of this particular movie before (the first time when I was a mere undergrad in the '80s). I expect we'll sort it out eventually, but it might get a little ugly first.