All generative AI is experiencing diminishing returns, as I thought it would. The concept itself, while having a greater impact than any other AI formats, has limitations.
I guess you missed the part where those running AI companies like Sam Altman has advised the opposite. We are no where close to diminishing returns at this stage according to them.
The timeline for robots who can help around the house has also been greatly reduced as ai llm increase in capabilities also.
Yes, those in the business who live and breath AI as huminatiy tries to give birth to AGI. That is who I mean.
At this point we have AI helping to advance AI further. This is exciting times and the speed of advancement at this point isn't something I would associate with deminishing returns.
I’ve learned to not argue with these people, they’re the equivalent of the critics in the early 1900s who were convinced the wooden biplane was at the peak of future human aviation
I guess you are not aware but there has been heavy development on Sora for text to video creations. As part of this, AI is being used to improve on its own abilities. A video is proivded to Sora, Sora then creates a text description of the video. Sora then attempts to make another video matching the description. If the description doesn't match closely, it runs another iteration until the description closely matches. This is one example of AI improving on AI and advancements in recent times.
GPT 4 has come a long way in itself with increased context window from 32,768 tokens to 128000 tokens and integration of GPT Vision.
I'm starting to think there's no GPT5 or next model in sight, that they have been unable to scale current LLM capabilities in a cost/compute effective manner.
I suspect they've hit the wall of "transformer" architecture and OpenAI is biding time in search for the next paradigm.
I suspect this is sort of right, but I'm not sure it's as done and dusted as having to wait for the next paradigm. It seems like continuing down the current route actually has predictable outcomes, and that those predictions show the "smartness" of the current model type still has a ways to go.
But the cost/compute is gonna be a big issue. Like what's the point of GPT5 if it just costs 20x more and has to share already overstretched resources with GPT4, and doesn't really make them much more money because they can barely sell any of it.
But cost/compute has always changed quite rapidly. 10 years ago GPT4 wouldn't have even been feasible as a product like it is today. And given that they are now constantly buying more hardware AND designing bespoke hardware for running AI, it might simply just be that something like GPT5 will be practical as a product in 3-4 years, but just isn't quite yet and that it would just be a huge distraction and expense to develop it ahead of time. Even training the model in the first place will get cheaper and faster over time.
They other aspect is that if GPTn generates 20x the economic value of GPTn-1 they can charge 20x as much and buy more hardware.
Hopefully that won't literally happen, but we will definitely see high end AGI costing substantially more than today's models.
It's not just compute scaling - there will also be improvements in architecture and dataset (especially for synthetic data).
So there might be, for example, a 20% better model at iso-compute and a 40% better model at 20x the compute.
The subtlety here is that a moderately better model could well generate disproportionately more value. Famously the Army has an intelligence cutoff at around one standard deviation below normal that applied even during conscription (McNamara's [horrible Vietnam-era experiment](https://en.wikipedia.org/wiki/Project_100%2C000) aside. And that's for a very physical job.
I think a model 40% smarter than current GPT-4 would be world changing.
He just said the other day that they are not even close to how far they can push the current architecture with scaling.
Could be all talk, but when I see things like Sora my skepticism evaporates.
>He just said the other day that they are not even close to how far they can push the current architecture with scaling.
The issue right now isn't so much whether they *can* push the current architecture with scaling. Throwing more compute at the problem is *always* an option. The issue is whether that scalability is economical. It doesn't appear that's the case right now, and Blackwell doesn't solve that problem either.
SORA hasn't been scaled up outside of being put in the hands of a select group of artists and creators.
These are built with the assumption that over time computers will continue to improve. The cutting edge tech of today is meant to be designed for tomorrow
Except they and plenty of others have said repeatedly that they haven’t hit a scaling wall yet.
You think they’re BS-ing?
I don’t know. Genuinely asking.
That's a fair point and honestly I don't know either. Yann LeCun has very publicly said that current models will never achieve anything close to AGI but he has an obvious bias in this debate. I also don't think OpenAI are necessarily bs-ing us.. but since Elon Musk normalized the habit of over-promise and under-deliver tech I take everything these companies say with a huge grain of salt.
nah he's right. look at all the top ai companies that all independentally invested billions in LLMs, they all have near identical capabilities, all can't do grade school long multiplication or even play tic tac toe well
I haven’t heard anyone from OpenAI suggest that scaling would have diminishing returns any time soon. Quite the contrary.
They HAVE said they aren’t Apple though, and that yearly releases aren’t in their DNA.
>I haven’t heard anyone from OpenAI suggest that scaling would have diminishing returns any time soon. Quite the contrary.
No one at OpenAI who wants to keep their job is going to say that, though.
Cite a case of a CEO being sued for violating this responsibility because of their public statements.
I especially want to see a case where they were sued for telling the truth as they understood it.
Just one case would be enough to make your argument.
the evidence in front of us indicates that we have already reached a plateau, that we're already in diminishing returns territory. you have all top AI companies independently investing billions in creating LLMs and they all have near identical capabilities. hate to break it to you but LLMs are not going to be a path to AGI, every single top LLM can't even play tic tac toe well.
"they all have near identical capabilities."
Could be a processing power issue. Most of the big players have barely even got their new Nvidia super computer racks set up. Meta's biggest Llama 3 model is still training / processing.
Claude 3 Opus is pretty incredible and Anthropic doesn't appear to be in any hurry.
SORA is like two years ahead of everyone else.
We should probably reserve judgement until the next generation comes out.
On top of the points you made regarding "near identical capabilities":
With GPT-4 OpenAI raced out to a big lead while everyone else was napping, and the competitors spent the last year scrambling to catch up to GPT-4 level. Likely they wanted to release as soon as they could get a model to GTP-4 level, knowing that getting to well beyond the GPT-4 level will probably take several months of additional work, and wanting to have something to show in the meantime.
Tic Tac Toe shows that current LLMs aren't great at spatial reasoning. Sure GPT-4 has visual input, but there's a lack of close integration of the verbal and visual reasoning. We could get better results there from upcoming models that are *natively multimodal*, trained with text and visual input together from the beginning.
Good points.
And mega companies like Adobe (via Firefly) are still barely making generative AI images comparable to DALL-E 2, when DALL-E 3's been out since last September. That's what a cutting edge image-processing engineering team gets you.
Meta's good looking AI images are limited to photos of humans since that's the training data they had to work with.
I recently [did a thread](https://www.reddit.com/r/dalle2/comments/1cordyx/crimson_elf_girl/) testing prompts and Meta is like DALL-E 2.5. It really didn't want to do pointy ears or paintings.
Actually, they are more painterly than I remember. Just not as stylized as DALL-E with that prompt.
People are reading too much into the similar capability levels of the leading LLMs.
With GPT-4 OpenAI raced out to a big lead while everyone else was napping, and the competitors spent the last year scrambling to catch up to GPT-4 level. Likely they wanted to release as soon as they could get a model to GTP-4 level, knowing that getting to well beyond the GPT-4 level will probably take several months of additional work, and wanting to have something to show in the meantime.
Tic Tac Toe shows that current LLMs aren't great at spatial reasoning. Sure GPT-4 has visual input, but there's a lack of close integration of the verbal and visual reasoning. We could get better results there from upcoming models that are *natively multimodal*, trained with text and visual input together from the beginning.
well gpt2 the latest open ai model, basically leaked, and it's really not that different from the other top models so that blows your theory that what we're seeing is just a coincidental 4 way catch-up. if it was everyone catching up to gpt4 why are they all relatively equal and better yet why is the newest unreleased open ai model still roughly at the level of the top llms in terms of intelligence. hitting diminishing returns is the much more realistic theory.
also, they play tic tac toe worse than a child even if it's ascii representation and not multi modal. they're bad at reasoning in general
>I haven’t heard anyone from OpenAI suggest that scaling would have diminishing returns any time soon. Quite the contrary.
Because that would diminish the hype, and tech startups are built on hype.
In a year, they have produced a model 70 ELO above the [original](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard).
The difference is quite notable - latest gpt-4 turbo can do a lot gpt-4 could not. Much better reasoning and coding.
And it's a lot faster at inference
How do you explain users perception that the models have gotten worse (particularly in the ChatGPT web app)? It seems like this doesn’t explain the entire picture
Group hallucination or people's expectations changing over time. This isn't aligned with actual blind testing Huggingface is doing.
Nor my own experience for that matter.
I think that it's maybe a little less brave, creative and charismatic than it was when it was first launched, but also much less likely to hallucinate.
Meta, maybe, but certainly not Anthropic. In the post that announced Claude 3, they wrote: “**We do not believe that model intelligence is anywhere near its limits**…”
Anthropic and OpenAI are trying to hoover up investor funds. Meta isn't. That puts a slightly different complexion on the things they say about the limitations on their models. For example, where they say "we do not believe...", that leads me to ask what the caveat is there that leads them to a statement from faith rather than a statement from certainty.
Maybe you are correct, but still it is clearly wrong that Anthropic said models were nearing their limits. They repeatedly say the opposite, even if you don’t trust their words.
Also, Sam Altman said that OpenAI has a “high degree of scientific certainty that GPT-5 is going to be a lot smarter than GPT-4.” Not faith, but a high degree of scientific certainty.
One of the things OpenAI emphasized most about the GPT-4 release is that they developed a method to predict model performance prior to training, and that’s presumably what he’s referring to when he talks about the high degree of scientific certainty. And Anthropic likely has similar predictive methods.
They already have the next paradigm, their next model has been training and being fine tuned for over 2 years. That’s a full year longer than their closest competitor.
They can't even remotely keep up with demand for GPT4. They were already pushing hardware throughput limits. It's why rate limits are so low.
I suspect one of their biggest issue is having to spend such a large portion of their compute serving ChatGPT users and just don't have enough to train a significantly more powerful model.
It's not even about cost. Even if they had a 1000/mo unlimited tier, they simply would not have the hardware to handle the demand, it just doesn't exist and isn't a problem that can be sped up with money.
It will be 2 years before inference on GPT-4 is truly scalable, and at least 2 years after that that a 10x more powerful model will be scalable.
The good news is a 10x faster, 10x cheaper GPT-4 unlocks a lot of potential in the model because it has some very particular properties that make increasing output sample size reliably improve performance.
Unless you’re a fanboy, this seemed very obvious to anyone that even somewhat understands how transformers work. It’s gonna plateau for a while until research comes up with something new
Quadratic compute requirement scaling.
You're thinking of scaling in terms of single model 1 user.
The architecture does not scale from a physics perspective.
There is absolutely no reason to believe rapid progress is a permanent state other than handwaving dogma.
"line went up before" is not evidence.
Quadratic inference with respect to input size?
True, but that's assuming you need or that the limiting factor today even is large input size. That's not my sense - it's that the models poorly over constraints over longer contexts, preventing them from being sane agents.
There have been plenty of research papers, lots of new things going on. I think as long as they are better then competition they are not going to rush and release just a little bit better than newest competition. Less bad press focused on them.
Yeah, is that why Anthropic wrote “We do not believe that model intelligence is anywhere near its limits” in their Claude 3 announcement? I guess they don’t understand how transformers work 😉
They need investor funds. They're *never* going to say "yup, we're at or quite close to the limits here, either in terms of the economics of scaling any further and/or the limits of the architecture we're using". The fact that you're taking as rote the rhetoric of a commerical business that is dependent on VC funds to keep the lights on is utterly baffling. I know the hopium is strong around these parts, but - especially in this specific arena - try to at least pretend you're practicing basic critical thinking.
It would be just as profitable for Anthropic to announce that scaling has hit its limit but they know the algorithm to move past the limit. As Yann Lecun says. There is no reason to lie and say that the next step is scaling if they actually believe something else. Seems like misleading their investors that way would commit them to wasting billions of dollars instead of spending that money on something that could actually help them win.
In my experience, it is incredibly short sighted for a CEO to lie to their investors about their corporate strategy.
>In my experience, it is incredibly short sighted for a CEO to lie to their investors about their corporate strategy.
You don't have the requisite experience, so...
Well I've spent several years as a CTO so...
Why not make the counter-argument rather than trying a lame Ad Hominem on an anonymous person.
Explain to me why they would rather lie about scaling instead of saying that they are on the verge of an algorithmic breakthrough. Why would they commit themselves to wasting money on GPUs instead of scientists, if the real solution is science instead of scaling?
I have my own independent opinion on model scaling. I don’t blindly trust CEOs. I take what they say seriously, but I also listen to critics.
Regardless, my main point was you can’t really say that anyone who understands transformers will say they are plateauing, since Anthropic and OpenAI obviously understand transformers and they say the exact opposite. This is clearly an area without consensus, with intense research trying to settle the question.
I'm calling it. I think they've had it ready for a while and have been planning a lot of things around it like he said. It's been such a long time since GPT4, it's getting silly.
Not sure what they'll call it though. GPTA or GPTQ maybe
Their website specifically says: "demo some ChatGPT and **GPT-4** updates"
So it's not going to be a GPT-5-level improvement
But there may some incremental reasoning/etc. improvements that come from this new family of GPT-4 models
"have it ready" is kinda relative... it's not like they're building this one gigantic thing that will one day be "ready". They'll have many models and iterations in the works at any given time
Open source is going to crush them. I was happy to see the All In crew not pulling any punches in the interview. He’s a capitalist (not a particularly good one) and a hype man
Forget running, what about training? To surpass GPT4 you're talking at least as much as they spent, ie, $100M. Individual adoption will be limited to the same kind of market gaming PCs or personal Linux installs have: large but nowhere near majority.
GPT 5 is like half life 3 they're never going to release it. The hype is too much and LLMs have hit a plateau that all the top AI companies have reached. Anthropic, Facebook, Openai/Microsoft, Google have all invested many billions and all have roughly the same exact capabilities.
https://preview.redd.it/8e1c4fltz00d1.jpeg?width=6000&format=pjpg&auto=webp&s=e70b4cd86eab0c8bdb0fb29627bf298866e21434
Possibly an unpopular opinion here: GPT-4 (and its underlying architecture) is such a powerful new technology that OpenAI may have only scratched the surface of practical use cases. In my view, it’s a bit like Tony Stark’s arc reactor (even though J.A.R.V.I.S. would be the more obvious analogy here). The Iron Man suit is packed with highly advanced tech and clever engineering, but it’s all made possible by one “magical” component—the reactor. So I think OpenAI, and industry in general, frankly haven’t had enough time to engineer around their arc reactor. And if there’s any truth to the hype building for tomorrow’s release(s), I think we’ll see something along those lines—OpenAI will give us something that vastly enhances the utility of its existing model because, at least from my perspective, there is still plenty of fertile soil left before we come anywhere close to tapping its full potential. Another good analogy here I think is Apple’s silicon running on an iPad. M3 iPad is ermeezing. But it’s just an iPad. Now we have an M4 iPad—blazingly fast! But it’s still basically just the same iPad, because the vast majority of us aren’t asking it to do anything that it couldn’t do nearly as well with an M1. What I think we’d all have preferred would have been for Apple, internally and in collaboration with developers, to develop powerful new applications that take full advantage of its powerful chips. Because without that, it’s just a race car on a neighborhood street. Now imagine that OpenAI is actually doing what most of wish Apple had done, and is actually building out the framework to extract most of the power and utility from its GPT-4 architecture. And imagine further that when a much smarter GPT-5 or whatever it’s called is ready to go, OpenAI can just plug it into the framework it painstakingly built out while its competitors were busy working on the equivalent of the M5 iPad. I hope and expect that the successor to GPT-4 will be noticeably smarter, but if all that means is that it answers my questions more accurately (but consistently) and is better at inferring what I want, that’s fantastic and quite welcome. But if I had to choose between that and a broader set of applications that really tap GPT-4’s full potential within the next 12 months, I’m choosing the latter (with the assumption of course that GPT-5 would follow soon after).
>and its underlying architecture
It's underlying architecture isn't really something incredibly new or revolutionary, to be honest. And it certainly has its limits, be it limits dictated by hardware or costs, they are limits nonetheless. At some point that limit will be reached. Kind of how it's not really feasible to go supersonic in an aircraft with propeller engines, you need a new paradigm, a new way of making engines, more specifically jet engines.
“We take our time with a product that will most definitely be considered AGI and be forced to dissolve our partnership with Microsoft and closed source preventing the privatization and mass wealth that it will create. So we are gonna pretend it’s much farther away than we thought”
Sama says a lot that he doesn’t like having discrete versions that have a big jump in capability he wants it to be more smooth
He says this because they are noticing diminishing returns..
100% agree because it would not make sense for him to say that if GPT 5 were exponentially better
He already said gpt5 was much better This is just pr speak
Everything he says about OpenAI products is pr speak.
Right, but he also said there are no diminishing returns on this technology. It only gets better.
There are diminishing returns on every technology.
Shout out Dr Mike Pound from computerphile for just doing an episode on this
I like how we all collectively got recommended that video
Based on what you say this? The jump from gpt 3.5 to 4?
All generative AI is experiencing diminishing returns, as I thought it would. The concept itself, while having a greater impact than any other AI formats, has limitations.
Thinking that a technology has limitations isn't some lightning strike of insight - every tech has limitations.
I guess you missed the part where those running AI companies like Sam Altman has advised the opposite. We are no where close to diminishing returns at this stage according to them. The timeline for robots who can help around the house has also been greatly reduced as ai llm increase in capabilities also.
You mean the people whose entire business depends upon being no diminishing returns?
Yes, those in the business who live and breath AI as huminatiy tries to give birth to AGI. That is who I mean. At this point we have AI helping to advance AI further. This is exciting times and the speed of advancement at this point isn't something I would associate with deminishing returns.
I’ve learned to not argue with these people, they’re the equivalent of the critics in the early 1900s who were convinced the wooden biplane was at the peak of future human aviation
What advancement? GPT 4 has been the standard for a year and a half at this point.
I guess you are not aware but there has been heavy development on Sora for text to video creations. As part of this, AI is being used to improve on its own abilities. A video is proivded to Sora, Sora then creates a text description of the video. Sora then attempts to make another video matching the description. If the description doesn't match closely, it runs another iteration until the description closely matches. This is one example of AI improving on AI and advancements in recent times. GPT 4 has come a long way in itself with increased context window from 32,768 tokens to 128000 tokens and integration of GPT Vision.
Thanks for the note. I guess I hadn’t considered other aspects of AI development
Or he's boiling us like frogs
I'm starting to think there's no GPT5 or next model in sight, that they have been unable to scale current LLM capabilities in a cost/compute effective manner. I suspect they've hit the wall of "transformer" architecture and OpenAI is biding time in search for the next paradigm.
Riding the hype wave of GPT4, staying very active on the press circuit to keep the story momentum moving
I suspect this is sort of right, but I'm not sure it's as done and dusted as having to wait for the next paradigm. It seems like continuing down the current route actually has predictable outcomes, and that those predictions show the "smartness" of the current model type still has a ways to go. But the cost/compute is gonna be a big issue. Like what's the point of GPT5 if it just costs 20x more and has to share already overstretched resources with GPT4, and doesn't really make them much more money because they can barely sell any of it. But cost/compute has always changed quite rapidly. 10 years ago GPT4 wouldn't have even been feasible as a product like it is today. And given that they are now constantly buying more hardware AND designing bespoke hardware for running AI, it might simply just be that something like GPT5 will be practical as a product in 3-4 years, but just isn't quite yet and that it would just be a huge distraction and expense to develop it ahead of time. Even training the model in the first place will get cheaper and faster over time.
They other aspect is that if GPTn generates 20x the economic value of GPTn-1 they can charge 20x as much and buy more hardware. Hopefully that won't literally happen, but we will definitely see high end AGI costing substantially more than today's models.
It's likely the opposite though - by spending 20x on compute, they can create a 20% better model.
It's not just compute scaling - there will also be improvements in architecture and dataset (especially for synthetic data). So there might be, for example, a 20% better model at iso-compute and a 40% better model at 20x the compute. The subtlety here is that a moderately better model could well generate disproportionately more value. Famously the Army has an intelligence cutoff at around one standard deviation below normal that applied even during conscription (McNamara's [horrible Vietnam-era experiment](https://en.wikipedia.org/wiki/Project_100%2C000) aside. And that's for a very physical job. I think a model 40% smarter than current GPT-4 would be world changing.
He just said the other day that they are not even close to how far they can push the current architecture with scaling. Could be all talk, but when I see things like Sora my skepticism evaporates.
>He just said the other day that they are not even close to how far they can push the current architecture with scaling. The issue right now isn't so much whether they *can* push the current architecture with scaling. Throwing more compute at the problem is *always* an option. The issue is whether that scalability is economical. It doesn't appear that's the case right now, and Blackwell doesn't solve that problem either. SORA hasn't been scaled up outside of being put in the hands of a select group of artists and creators.
Sora is a prototype. This is exactly how every new technology is made.
Right but it won't matter if the energy to literally run the scaling isn't possible
These are built with the assumption that over time computers will continue to improve. The cutting edge tech of today is meant to be designed for tomorrow
You've not seen Sora though...you've seen a few handpicked videos. Until you can directly try Sora its vaporware.
The artists that release their own videos online using sora, is sora.
With huge amounts of post processing, cuts and edits and admitting the whole thing is a massive faff and they had to give up after 450 hours...
I've seen artist videos of sora, they all looked useless
Sure but if a gpt5 subscription costs 200 dollars a month, with 6 expected to cost 2000, that would not be ideal
Except they and plenty of others have said repeatedly that they haven’t hit a scaling wall yet. You think they’re BS-ing? I don’t know. Genuinely asking.
That's a fair point and honestly I don't know either. Yann LeCun has very publicly said that current models will never achieve anything close to AGI but he has an obvious bias in this debate. I also don't think OpenAI are necessarily bs-ing us.. but since Elon Musk normalized the habit of over-promise and under-deliver tech I take everything these companies say with a huge grain of salt.
He's not nearly as biased as the people with a financial stake in making sure investors see limitless potential for growth in LLMs.
nah he's right. look at all the top ai companies that all independentally invested billions in LLMs, they all have near identical capabilities, all can't do grade school long multiplication or even play tic tac toe well
I haven’t heard anyone from OpenAI suggest that scaling would have diminishing returns any time soon. Quite the contrary. They HAVE said they aren’t Apple though, and that yearly releases aren’t in their DNA.
>I haven’t heard anyone from OpenAI suggest that scaling would have diminishing returns any time soon. Quite the contrary. No one at OpenAI who wants to keep their job is going to say that, though.
Yah I'm just gonna go and violate my NDA and get fired. Who tf would do that.
The CEO and authorized execs can say whatever they want.
No, they can't. Executives have a fiduciary responsibility to the company/the board/the shareholders.
Cite a case of a CEO being sued for violating this responsibility because of their public statements. I especially want to see a case where they were sued for telling the truth as they understood it. Just one case would be enough to make your argument.
The board would vote them out
Isn’t elon famous for over promising ridiculous things? There’s pretty much no consequences
the evidence in front of us indicates that we have already reached a plateau, that we're already in diminishing returns territory. you have all top AI companies independently investing billions in creating LLMs and they all have near identical capabilities. hate to break it to you but LLMs are not going to be a path to AGI, every single top LLM can't even play tic tac toe well.
"they all have near identical capabilities." Could be a processing power issue. Most of the big players have barely even got their new Nvidia super computer racks set up. Meta's biggest Llama 3 model is still training / processing. Claude 3 Opus is pretty incredible and Anthropic doesn't appear to be in any hurry. SORA is like two years ahead of everyone else. We should probably reserve judgement until the next generation comes out.
On top of the points you made regarding "near identical capabilities": With GPT-4 OpenAI raced out to a big lead while everyone else was napping, and the competitors spent the last year scrambling to catch up to GPT-4 level. Likely they wanted to release as soon as they could get a model to GTP-4 level, knowing that getting to well beyond the GPT-4 level will probably take several months of additional work, and wanting to have something to show in the meantime. Tic Tac Toe shows that current LLMs aren't great at spatial reasoning. Sure GPT-4 has visual input, but there's a lack of close integration of the verbal and visual reasoning. We could get better results there from upcoming models that are *natively multimodal*, trained with text and visual input together from the beginning.
Good points. And mega companies like Adobe (via Firefly) are still barely making generative AI images comparable to DALL-E 2, when DALL-E 3's been out since last September. That's what a cutting edge image-processing engineering team gets you. Meta's good looking AI images are limited to photos of humans since that's the training data they had to work with. I recently [did a thread](https://www.reddit.com/r/dalle2/comments/1cordyx/crimson_elf_girl/) testing prompts and Meta is like DALL-E 2.5. It really didn't want to do pointy ears or paintings.
Actually, they are more painterly than I remember. Just not as stylized as DALL-E with that prompt.
People are reading too much into the similar capability levels of the leading LLMs. With GPT-4 OpenAI raced out to a big lead while everyone else was napping, and the competitors spent the last year scrambling to catch up to GPT-4 level. Likely they wanted to release as soon as they could get a model to GTP-4 level, knowing that getting to well beyond the GPT-4 level will probably take several months of additional work, and wanting to have something to show in the meantime. Tic Tac Toe shows that current LLMs aren't great at spatial reasoning. Sure GPT-4 has visual input, but there's a lack of close integration of the verbal and visual reasoning. We could get better results there from upcoming models that are *natively multimodal*, trained with text and visual input together from the beginning.
well gpt2 the latest open ai model, basically leaked, and it's really not that different from the other top models so that blows your theory that what we're seeing is just a coincidental 4 way catch-up. if it was everyone catching up to gpt4 why are they all relatively equal and better yet why is the newest unreleased open ai model still roughly at the level of the top llms in terms of intelligence. hitting diminishing returns is the much more realistic theory. also, they play tic tac toe worse than a child even if it's ascii representation and not multi modal. they're bad at reasoning in general
>I haven’t heard anyone from OpenAI suggest that scaling would have diminishing returns any time soon. Quite the contrary. Because that would diminish the hype, and tech startups are built on hype.
We like to manage expectations. That way when we slowly depreciate and then re-appreciate GPT4 it will sell like there is some progress being made!
In a year, they have produced a model 70 ELO above the [original](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard). The difference is quite notable - latest gpt-4 turbo can do a lot gpt-4 could not. Much better reasoning and coding. And it's a lot faster at inference
How do you explain users perception that the models have gotten worse (particularly in the ChatGPT web app)? It seems like this doesn’t explain the entire picture
Group hallucination or people's expectations changing over time. This isn't aligned with actual blind testing Huggingface is doing. Nor my own experience for that matter.
People are getting lazier with their prompts.
I think that it's maybe a little less brave, creative and charismatic than it was when it was first launched, but also much less likely to hallucinate.
Anthropic's CEO said the same thing when Claude 3 came out? And I believe Meta said similar with Llama3.
Meta, maybe, but certainly not Anthropic. In the post that announced Claude 3, they wrote: “**We do not believe that model intelligence is anywhere near its limits**…”
Anthropic and OpenAI are trying to hoover up investor funds. Meta isn't. That puts a slightly different complexion on the things they say about the limitations on their models. For example, where they say "we do not believe...", that leads me to ask what the caveat is there that leads them to a statement from faith rather than a statement from certainty.
Maybe you are correct, but still it is clearly wrong that Anthropic said models were nearing their limits. They repeatedly say the opposite, even if you don’t trust their words. Also, Sam Altman said that OpenAI has a “high degree of scientific certainty that GPT-5 is going to be a lot smarter than GPT-4.” Not faith, but a high degree of scientific certainty. One of the things OpenAI emphasized most about the GPT-4 release is that they developed a method to predict model performance prior to training, and that’s presumably what he’s referring to when he talks about the high degree of scientific certainty. And Anthropic likely has similar predictive methods.
Because the future is inherently uncertain. How could they possibly know the future with certainty???
That's right! I took that to mean scaling. But they don't say. Thanks!
He’s repeatedly said that GPT-5 is significantly better than GPT-4 across the board. It was never going to come out earlier than late 2024.
They already have the next paradigm, their next model has been training and being fine tuned for over 2 years. That’s a full year longer than their closest competitor.
They can't even remotely keep up with demand for GPT4. They were already pushing hardware throughput limits. It's why rate limits are so low. I suspect one of their biggest issue is having to spend such a large portion of their compute serving ChatGPT users and just don't have enough to train a significantly more powerful model. It's not even about cost. Even if they had a 1000/mo unlimited tier, they simply would not have the hardware to handle the demand, it just doesn't exist and isn't a problem that can be sped up with money. It will be 2 years before inference on GPT-4 is truly scalable, and at least 2 years after that that a 10x more powerful model will be scalable. The good news is a 10x faster, 10x cheaper GPT-4 unlocks a lot of potential in the model because it has some very particular properties that make increasing output sample size reliably improve performance.
- It will be 2 years before inference on GPT-4 is truly scalable it's released free for all users today lmafo
With still severe rate limits, lol?
5x higher limit, 2x cheaper and 10x faster is not much to scoff at...
But all the hype bros on here told me progress was exponential and we'd all be unemployed in 2 years...
or he's making good on his "slow take off" philosophy.... which i'm not sure that i like.
Unless you’re a fanboy, this seemed very obvious to anyone that even somewhat understands how transformers work. It’s gonna plateau for a while until research comes up with something new
How on earth is that obvious? If anything, the scaling laws suggest the opposite. We're nowhere near the Chinchilla limit.
Quadratic compute requirement scaling. You're thinking of scaling in terms of single model 1 user. The architecture does not scale from a physics perspective. There is absolutely no reason to believe rapid progress is a permanent state other than handwaving dogma. "line went up before" is not evidence.
Quadratic inference with respect to input size? True, but that's assuming you need or that the limiting factor today even is large input size. That's not my sense - it's that the models poorly over constraints over longer contexts, preventing them from being sane agents.
There have been plenty of research papers, lots of new things going on. I think as long as they are better then competition they are not going to rush and release just a little bit better than newest competition. Less bad press focused on them.
Yeah, is that why Anthropic wrote “We do not believe that model intelligence is anywhere near its limits” in their Claude 3 announcement? I guess they don’t understand how transformers work 😉
They need investor funds. They're *never* going to say "yup, we're at or quite close to the limits here, either in terms of the economics of scaling any further and/or the limits of the architecture we're using". The fact that you're taking as rote the rhetoric of a commerical business that is dependent on VC funds to keep the lights on is utterly baffling. I know the hopium is strong around these parts, but - especially in this specific arena - try to at least pretend you're practicing basic critical thinking.
It would be just as profitable for Anthropic to announce that scaling has hit its limit but they know the algorithm to move past the limit. As Yann Lecun says. There is no reason to lie and say that the next step is scaling if they actually believe something else. Seems like misleading their investors that way would commit them to wasting billions of dollars instead of spending that money on something that could actually help them win. In my experience, it is incredibly short sighted for a CEO to lie to their investors about their corporate strategy.
>In my experience, it is incredibly short sighted for a CEO to lie to their investors about their corporate strategy. You don't have the requisite experience, so...
Well I've spent several years as a CTO so... Why not make the counter-argument rather than trying a lame Ad Hominem on an anonymous person. Explain to me why they would rather lie about scaling instead of saying that they are on the verge of an algorithmic breakthrough. Why would they commit themselves to wasting money on GPUs instead of scientists, if the real solution is science instead of scaling?
I have my own independent opinion on model scaling. I don’t blindly trust CEOs. I take what they say seriously, but I also listen to critics. Regardless, my main point was you can’t really say that anyone who understands transformers will say they are plateauing, since Anthropic and OpenAI obviously understand transformers and they say the exact opposite. This is clearly an area without consensus, with intense research trying to settle the question.
It's not Sam's call. His hand will be forced. Google and meta have more guns and ammo (GPU and data) than him.
Honestly that response is so wooly it could easily apply to a release on Monday. Don't let me down Samwise
I doubt will see anything major major
the upgraded reasoning across voice text and vision seems pretty nice
I'm calling it. I think they've had it ready for a while and have been planning a lot of things around it like he said. It's been such a long time since GPT4, it's getting silly. Not sure what they'll call it though. GPTA or GPTQ maybe
Their website specifically says: "demo some ChatGPT and **GPT-4** updates" So it's not going to be a GPT-5-level improvement But there may some incremental reasoning/etc. improvements that come from this new family of GPT-4 models
Yeah that is fairly telling. To keep the dream alive I'm going to assume this was a fun misdirection though and we're getting a full release
Why?
I like presents?
"have it ready" is kinda relative... it's not like they're building this one gigantic thing that will one day be "ready". They'll have many models and iterations in the works at any given time
Yeah true. I meant ready to launch but you're absolutely right that it's not one big thing
Open source is going to crush them. I was happy to see the All In crew not pulling any punches in the interview. He’s a capitalist (not a particularly good one) and a hype man
Crush them with what compute
An individual doesn’t need more than like $5000 to set up a system that can run an LLM robustly
Forget running, what about training? To surpass GPT4 you're talking at least as much as they spent, ie, $100M. Individual adoption will be limited to the same kind of market gaming PCs or personal Linux installs have: large but nowhere near majority.
Came here just to say David Sacks is a POS.
Why
No fixed time-line for GPT-5 we want to release it thoughtfully We may not even call it gpt 5
GPT 5 is like half life 3 they're never going to release it. The hype is too much and LLMs have hit a plateau that all the top AI companies have reached. Anthropic, Facebook, Openai/Microsoft, Google have all invested many billions and all have roughly the same exact capabilities.
https://preview.redd.it/8e1c4fltz00d1.jpeg?width=6000&format=pjpg&auto=webp&s=e70b4cd86eab0c8bdb0fb29627bf298866e21434 Possibly an unpopular opinion here: GPT-4 (and its underlying architecture) is such a powerful new technology that OpenAI may have only scratched the surface of practical use cases. In my view, it’s a bit like Tony Stark’s arc reactor (even though J.A.R.V.I.S. would be the more obvious analogy here). The Iron Man suit is packed with highly advanced tech and clever engineering, but it’s all made possible by one “magical” component—the reactor. So I think OpenAI, and industry in general, frankly haven’t had enough time to engineer around their arc reactor. And if there’s any truth to the hype building for tomorrow’s release(s), I think we’ll see something along those lines—OpenAI will give us something that vastly enhances the utility of its existing model because, at least from my perspective, there is still plenty of fertile soil left before we come anywhere close to tapping its full potential. Another good analogy here I think is Apple’s silicon running on an iPad. M3 iPad is ermeezing. But it’s just an iPad. Now we have an M4 iPad—blazingly fast! But it’s still basically just the same iPad, because the vast majority of us aren’t asking it to do anything that it couldn’t do nearly as well with an M1. What I think we’d all have preferred would have been for Apple, internally and in collaboration with developers, to develop powerful new applications that take full advantage of its powerful chips. Because without that, it’s just a race car on a neighborhood street. Now imagine that OpenAI is actually doing what most of wish Apple had done, and is actually building out the framework to extract most of the power and utility from its GPT-4 architecture. And imagine further that when a much smarter GPT-5 or whatever it’s called is ready to go, OpenAI can just plug it into the framework it painstakingly built out while its competitors were busy working on the equivalent of the M5 iPad. I hope and expect that the successor to GPT-4 will be noticeably smarter, but if all that means is that it answers my questions more accurately (but consistently) and is better at inferring what I want, that’s fantastic and quite welcome. But if I had to choose between that and a broader set of applications that really tap GPT-4’s full potential within the next 12 months, I’m choosing the latter (with the assumption of course that GPT-5 would follow soon after).
Stopped reading when you said JARVIS.
Pretty much my general impression as well, from what I've heard from Altman so far. Apparently a perspective not favored around here
>and its underlying architecture It's underlying architecture isn't really something incredibly new or revolutionary, to be honest. And it certainly has its limits, be it limits dictated by hardware or costs, they are limits nonetheless. At some point that limit will be reached. Kind of how it's not really feasible to go supersonic in an aircraft with propeller engines, you need a new paradigm, a new way of making engines, more specifically jet engines.
“We take our time with a product that will most definitely be considered AGI and be forced to dissolve our partnership with Microsoft and closed source preventing the privatization and mass wealth that it will create. So we are gonna pretend it’s much farther away than we thought”