Ask HN: How much better can the LLMs become assuming no AGI
OpenAI, Anthropomorphic and every company is putting a lot into training. How is the functionality pipeline filled out for LLMs? What is missing in today’s LLMs that they need to plan for? Just trying to get some insights into the planning process and also what the community sees as the northstar for LLMs without saying AGI
For the most part, LLMs just remember.
They don't think or learn or create on their own --- at least not anywhere close to a human level. Otherwise, they wouldn't require so much "training"
Essentially, they are best characterized as a huge database with a natural language interface.
Once the internet had been consumed and indexed, this sort of approach starts to hit a wall. There is no more data readily available for "training".
I don't know what the next breakthrough will be but I firmly believe one will be required to push performance to any significantly higher level.
In terms of bits seen during training, LLM are more akin to a 3 year old. Robots roaming around and learning to interact with the in environment and sharing knowledge might be a game changer, assuming that the current methodologies are sufficient (LLM + RL)
Right now I get timed out on my thinking queries use of VM in 60 seconds and as a result the responses are less than adequate as it tries to take shortcuts to stay within the time out limit. I can imagine that in the future maybe there won't be a time out limit which would greatly increase the quality of the responses. And more recently the model seems to get stuck reaffirming facts that we already established and moved on from. For some reason it feels it wants to remind me. Additionally, we may have moved on, but then it applies the fact to what we have moved on to so the context needs to be improved.
[dead]