> I'd love to know what are some of the hidden challenges to making a useful product with agents? …

I use already a few ai tools even without perfect accuracy. And a LLM who only needs to call to a few API calls isn't hard. Very little need perfect accuracy and for that we still have classical software.

> You can do handwritten digit recognition with 90% accuracy? Sounds pretty good, but if you need to turn that into recognizing a 12 digit account number you now have a 70% chance of getting at least one digit incorrect. You are assuming that the probability of failure is independent, which couldn't be further from the truth. If a digit recogniser can recognise one of your "hard" handwritten digits, such as a 4 or a 9, it will likely be able to recognise all of them. The same happens with AI agents. They are not good at some tasks, but really really food at others.

Yea that's a good point. Now I am curious, what are some tasks that can accept a model that is at 80% as good as a human, but is 100x cheaper?(or, 100x faster?)

> The biggest thing any ML practitioner realizes when they step out of a research setting is that for most tasks accuracy has to be very high for it be productizable. I think that ChatGPT's success might be partly attributable to its chat interface. For whatever reason, a lot of people - including me! - are much more forgiving of inconsistencies, slip-ups, and inaccuracies when in a conversational format. Kind of like how you might forgive a real human for making a mistake in conversation. I don't think that's necessarily good, and might not have much connection to attempts to build new non-conversational products on top of LLMs, but maybe it has some explanatory power for the current situation.

One interesting thing about LLMs is that they can actually recover (and without error loops). You can have a step that doesn't work right, and a later step can use its common-sense knowledge to ignore some of the missing results, conflicting information, etc. One of the problems with developing with LLMs is that the machine will often cover up bugs! You think it's giving sub-par results but actually you've given it conflicting or incomplete instructions. Another opportunity is that you can have less steps or more shared context. One interesting thing about Whisper is that it's not just straight speech recognition but can also be prompted and given context to understand what sort of thing the speech may be about, increasing its accuracy considerably. LLM Vision models also do this with things like OCR. This might not help it with the individual digits in an account number, but it does hel…

The first think any ML practictioner realizes is that accuracy is about the single worst performance metric you can use for most real-world tasks, lol

This has been a perfect description of my experience doing this. I had written some code to go through reasonably complex web onboarding flows and it basically played out exactly like you predicted in your comment. In addition, I've been working with some vendors that have been trying to do the same thing and they're finding that it works out just like you describe. The handwritten automations have performed better and the issues are reproducible, so even when there are issues, there's some sense of forward progress as you fix them. With handing it all over to an agent, it really feels like running around in circles. I think there's probably something here, but it's less trivial than just tossing a webpage at chatGPT and hoping for the best.

You're 100% right - but I do think there are more lower accuracy cases than I initially expected, *especially* if you assume a human-in-the-loop. Still 10x better than status quo. Ex. Content generation + zero-shot classification/mapping are powerful, and with a human in the loop (somewhat) responsible for accuracy, they can move much faster.