The conversation

marcus_holmesHNSource author
Lessons learnedExperience excerpt · English

Then I started running tests and getting into the details, and found that if you ran the same …

… Then I started running tests and getting into the details, and found that if you ran the same report on the same data, it generated completely different, still very plausible, results. I could run the same source data through the assessment process 10 times and get 10 very different results. We had to can the project and go a different route. LLMs are designed to produce plausible results, not factual results. We can fix this when using them for software dev by using linters and tests (though we've all had the experience where the LLM invents an API endpoint). I would not trust raw LLM output in any situation where that kind of testing and verification capability isn't present.

Original discussionI used Claude Code to get a second opinion on my MRI715 source comments · at import
6

What was this responding to?

Parent context · original excerpt

AU
AurornisHN

I have multiple LLM subscriptions at any given time, plus an array of local models. When I ask a question outside of my domain of expertise I like to ask all of the LLMs I have access to. I also create separate sessions and ask the same question multiple ways. It’s revealing to see how many different and contradictory answers I get, most of which are presented confidently. The last time I ran a medical question through Claude I couldn’t even get consistent answers between sessions. It’s also scary how easily you can lead each LLM to the answer you have in mind. When I would start asking questions about different options that other LLMs had presented, each session would drift toward that explanation.

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 6

Direct replies available in this import. Original-language excerpts.

AD
adamddev1HN

Linters and tests help of course, but they cannot "fix" the problem since tests cannot prove the absence of bugs.

Full original
SU
SuppaflyHN

What's crazy is that there are ton of businesses building processes around LLMs that haven't done this exercise and fully believe the LLM is giving them accurate data.

Full original
XB
xbmcuserHN

Yup I use llm to write scripts for me to process data I don't ask the llm to process the data themselves. Even when I wrote something for my day trading I used llm write scripts that do all the processing and predict price movement from that the more data is pre processed the more all the llm come up with similar trades.

Full original
YU
yubblegumHN

> LLMs are designed to produce plausible results, not factual results. They are true to their name: Language models. It is precisely the same problem in a language: a grammatically correct sentence is not necessarily true.

Full original
BO
bondarchukHN

It's funny that if the LLMs had all given the same result each time (it sounds like) you would have considered it more valid, even though it might just be giving a single wrong answer more consistently.

Full original
GR
greenailHN

you can set the "temperature" which is a lever on how stochastic the prediction is. If you are doing your own inference this is clear and easy. If you are consuming tokens this is outsourced.

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.