The conversation

zsoltkacsandiHNSource author
Lessons learnedExperience excerpt · English

Once I tested this, I gave the same task for a model after the release and a couple weeks later

… Once I tested this, I gave the same task for a model after the release and a couple weeks later. In the first attempt it produced a well-written code that worked beautifully, I started to worry about the jobs of the software engineers. Second attempt was a nightmare, like a butcher acting as a junior developer performing a surgery on a horse. Is this empirical evidence? And this is not only my experience. Calling this phychological is gaslighting.

Original discussionClaude Opus 4.5506 source comments · at import
4

What was this responding to?

Parent context · original excerpt

LU
lukevHN

There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift. However, benchmarks exist. And I haven't seen any empirical evidence that the performance of a given model version grows worse over time on benchmarks (in general.) Therefore, some combination of two things are true: 1. The nerf is psychologial, not actual. 2. The nerf is real but in a way that is perceptual to humans, but not benchmarks. #1 seems more plausible to me a priori, but if you aren't inclined to believe that, you should be positively intrigued by #2, since it points towards a powerful paradigm shift of how we think about the capabilities of LLMs in general... it would mean there is an "x-factor" that we're entirely unable to capture in any benchmark to date.

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 4

Direct replies available in this import. Original-language excerpts.

AC
ACCount37HN

No, it's entirely psychological. Users are not reliable model evaluators. It's a lesson the industry will, I'm afraid, have to learn and relearn over and over again.

Full original
LU
lukevHN

> Is this empirical evidence? Look, I'm not defending the big labs, I think they're terrible in a lot of ways. And I'm actually suspending judgement on whether there is ~some kind of nerf happening. But the anecdote you're describing is the definition of non-empirical. It is entirely subjective, based entirely on your experience and personal assessment.

Full original
RO
roywigginsHN

https://en.wikipedia.org/wiki/Regression_toward_the_mean The way this works is: 1) x% of users have an exceptional first experience by chance. Nobody who has a meh first experience bothers to try a second time. 2) x²% of users also have an exceptional second experience by chance 3) So a lot of people with a great first experience think the model started off great and got suddenly worse Suppose it's 25% that have a really great first experience. 25% of them have a great second experience too, but 75% of them see a sudden decline in quality and decide that it must be intentional. After the third experience this population gets bigger again. So by pure chance and sampling biases you end up convincing a bunch of people that the model used to be great but has gotten worse, but a much smaller population of people who thought it was terrible but got better because most of them gave up early . …

Full original
WO
WowfunhappyHN

I think this is pretty easy to explain psychologically. The first time you see a dog that can make pancakes, you’re really focused on the fact that a dog is making pancakes . After a few weeks of having them for breakfast, you start to notice that the pancakes are actually kind of overcooked and don’t taste that good. Sure it’s impressive that a dog made them, but what use are sub-par pancakes? You’re naturally more focused on what it can’t do than what it can .

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.