socketcluster 2 days ago

It is true that they don't tend to give simple answers.

But it reveals something about human biases that we tend to seek out simple answers to complex questions. We tend to think that because a question is short, that it's simple. We forgot how many details are involved and how much is actually going on underneath. Reality itself is often annoyingly complex.

All living creatures, including humans, are conditioned to simplify extremely complex things into simple concepts which we can react to. We only understand things to the extent that we believe they affect us.

  • goodmythical 2 days ago

    That's just a caloric efficiency thing, isn't it?

    Far easier to simply spread peanut butter and jelly on bread than to consider how the peanuts were harvested, how many bugs were in the fruits, how close the wheat was to molding, how much sugar is in the bread, how many years it tooks us to perfect such tasty and long lasting ingredients, what potenitally substandard conditions the labor involved endured...etc, every time one wants a sandwhich.

    And then a multiple of that easier as well in conversing about a peanut butter and jelly sandwhich as few if any of the parties will think about any of the above.

    We see the same in computers. A web dev can make it quite far without once considering the OSI model. A game dev can complete an entire career without ever worrying about the underlying bytecode.

    There are situational benefits to diving deeper, of course, but I don't imagine we'd make it very far if we had to reinvent the universe for every word in every conversation or chain of thought.

    • jvanderbot 2 days ago

      Calorie, bit also speed of decision and easier transmission to peers and kiddos.

      A quick rhyme encoding a useful heuristic? They'll remember that for generations!

    • grayclhn 2 days ago

      >> But it reveals something about human biases that we tend to seek out simple answers to complex questions. We tend to think that because a question is short, that it's simple. We forgot how many details are involved and how much is actually going on underneath. Reality itself is often annoyingly complex.

      > That's just a caloric efficiency thing, isn't it?

      perfect, no notes

  • proc0 2 days ago

    Simple answers are the result of a problem being solved all the way through. It abstracts away all the details and exposes the essential part of the solution. This is the compression perspective of intelligence. We want things as compressed as possible because it's how intelligence more or less works.

    • Shorn 2 days ago

      Whitehead said it in 1911:

      > “Civilization advances by extending the number of important operations which we can perform without thinking about them.”

      He followed it with:

      > “Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.”

    • socketcluster 2 days ago

      Agreed but often, even the simplest possible solution is still complicated.

  • kzrdude 2 days ago

    In a lot of sciences we actually give simple answers, while keeping the door open for "it's complicated, go this way to find out more". And you can repeat that with four or five layers, always adding details to make it 'more correct'.

    Our whole academic system is structured that way :)

  • wolvesechoes 2 days ago

    But the things their answers are not detailed and nuanced, but trivial and verbose. They lay down the most banal stuff like it some enlightening wisdom.

    • socketcluster 2 days ago

      I agree that they don't typically phrase it in the simplest way possible. Also, they have a bias towards increasing complexity so they tend to frame problems in a complex way so that they may then suggest solving them by making things more complicated. Many times, the best solution is to avoid the problem entirely. LLMs almost never see this kind of solution.

  • watwut 1 day ago

    > But it reveals something about human biases that we tend to seek out simple answers to complex questions.

    There is a value in a simple answer to simple question whose correct answer is actually very simple.

b3ing 2 days ago

Problem is tech is full of PMs and other managers that don’t know technology. Things that could be algorithms, functions, programs, regex filters, etc are now LLMs because they don’t know anything else to solve problems

  • cookiengineer 2 days ago

    > Problem is tech is full of PMs and other managers that don’t know technology. Things that could be algorithms, functions, programs, regex filters, etc are now LLMs because they don’t know anything else to solve problems

    Is this the right approach though?

    Computers should be usable by humans, not the other way around. How CPUs are built, how assembler came to be, how programming languages came to be... ultimately, they were all just compromises along the way to make computers more usable by humans.

    With LLMs, however, the threshold barrier for being able to modify and use a computer effectively is now zero, which in my opinion is a good thing.

    I can recommend watching "the mother of all demos" talk from 1968 [1] which basically established the design patterns for modern computers. That's how old the concept of desktop environments is. There hasn't been any meaningful invention eversince, apart from touchscreens/smartphones, and now LLMs as a generalized interface design.

    [1] https://www.youtube.com/watch?v=yJDv-zdhzMY

    • b3ing 2 days ago

      Well LLMs are more expensive and take longer to setup. If non tech people want to build and leverage APIs and so forth for something they could have done in 1 hr or less of coding I guess that’s up to them. But I’ve seen them planning thousands of dollars costs and a team of people working on something simple

  • whycombinetor 2 days ago

    This year I listened to someone suggest using an LLM call to go from the UI field contents of a web form to the JSON to send over the wire. Like, a .map statement.

    • perrygeo 2 days ago

      This year I listened to someone use an LLM call to sort a list of dictionaries by key.

      The LLM did indeed sort the list - after several seconds.

      My editor can sort the list in single-digit milliseconds.

      Do you really aspire to be the 0.001x developer? Weird flex, bro. It's just embarrassing.

      There are operations best done locally. Learn your tools. Outsourcing these task simply because you're unaware is not something to be proud of.

      • jamesfinlayson 2 days ago

        My old boss proudly told all how he'd used Claude to create three tickets in Jira. He said it took 10 minutes and normally it would take him 2 minutes, but he was still thrilled.

        • hdjrudni 2 days ago

          Interesting. I use my LLM to create tickets too but it saves time. Often I just need a bug # or a reminder to do something later. The bugs it writes might not be perfect but they're more detailed than I would have written myself.

  • hdjrudni 2 days ago

    The software engineers should be pushing back on PMs when they overcomplicate things though. That's our job as SWEs. In fact, PMs shouldn't really be suggesting the technology we use at all. Just tell us what you want the product to do and we'll tell you how to get it there.

    • ffalex 2 days ago

      True, they should. But in ugly corporate world of today there are consequences. Reality nowadays is that push back is perceived as reluctance and not being open to the great new world of AI era, sometimes even as not wanting to adapt. And that brings a lot of frustration to both sides. Kinda waiting for all this to blow up in their faces. In real life tho most of the times different people will have to deal with consequences.

      • pixl97 1 day ago

        And when many companies are laying off large swaths of employees looking like you don't want to adapt is risky.

  • dumfries 2 days ago

    It's not just that they don't know. It's also that chasing the new shiny things helps them at their annual review.. I've worked with several PMs that were not that interested to work on certain project but once the term LLM was dropped, even as a possible option for something simple, they all were interested.

  • mike_hearn 2 days ago

    But this article is about replacing hard-coded algorithms and functions with what is actually indeed an LLM. Decision models like Jev are large language models, they're just using a slightly different neural architecture designed for one-shot selection from a list instead of generation of tokens. But they're still processing language neurally.

dzink 2 days ago

PostHog should not be using them on customer data, period.

  • jakeydus 2 days ago

    Every time one of these consumer data brokers/gatherers shows some insight into how they gather the data it makes me feel sick. I know they're doing it, I know it's granular, it's still gross.

    • goosejuice 2 days ago

      I mean look at what they sell. Surveillanceware.

    • nixosbestos 2 days ago

      Gathering it and pumping it all through Anthropic and OpenAI data collectors.

  • hdjrudni 2 days ago

    In their defence, the incoming tickets weren't supposed to contain PII in the first place. That means whoever wrote the ticket included too much personal info.

    I don't know what's worse. If they don't use AI, then the bug gets filed as-is and exposed to other humans who shouldn't have access to that info in the first place.

    Maybe a local/on-prem Jev is ideal?

throwawayffffas 2 days ago

Using these decision models as a model router sounds really bad to me the models will most likely be newer than the decision model and thus the decision model will have no information about the capabilities of the model other than the short blurb you feed it.

TacticalCoder 2 days ago

> A probability score of 0.85 still means ~150 wrong answers per 1,000 checks, so plan ahead for false positives and false negatives.

But that's only true if we believe the value of 0.85 in itself is correct!? For all we know it could be 0.75 or 0.9 no?

If were to make calls to something like JEV and know that the probability the answer is correct is, say, 0.85193847098713, wouldn't that be even more amazing than JEV and LLMs themselves!?

I'm not being facetious here: how do we know it's 0.85 and neither 0.75 nor 0.9?

And if 0.85 is "correct", why stop there? Why not 0.85193847098713?

Any explanation is very welcome.

  • thereader12 2 days ago

    Yeah, if I understood correctly, they are using it wrong. The value is Jev's confidence which they trained for higher accuracy but it's still Jev is 85 percent sure not it's an 85% chance of being correct.

lhk931122 2 days ago

If this direction works out, I think service companies like Anthropic or OpenAI would provide it. If there is a lot of demand. Until then you have to set it up yourself locally, which is a hassle. Definitely, for components that act as sub agents in the agents I use personally, this direction actually looks useful for getting answers that rely even a little less on randomness and using them.

  • simianwords 2 days ago

    OpenAI already provides Luna based decision model

simianwords 2 days ago

The fact that Jev can't do reasoning means that it can only be used for very basic decisions.

Jev can't answer this for example

    What is 6^8-3&2 mod 2? 0 or 1? 

You might think it is a contrived example but it is illustrative. It shows that almost most questions are reasoning shaped.

nostrebored 2 days ago

I am so skeptical about these decision model rerankers. Training a reranker has never been easier, and many of them already return calibrated scores.

Decision models are often larger, slower, non colocated… it just seems like a strange tradeoff to make.

npteljes 2 days ago

This reminded me of the Copilot Fusion feature. It tackles a similar problem in a similar way: classify or even split up the task at hand, and trying to use the best qualified model for each step.

HackerThemAll 2 days ago

Let's reflect that statement: barely anything needs an LLM.

fauria 2 days ago

Related: Amazon saves millions by replacing Sonnet 4.6 with Strands Decider - https://news.ycombinator.com/item?id=50436640

  • CharlesW 2 days ago

    As I understand it (and count that I will be proven wrong if not), Strands Decider is an LLM, just a discriminative example instead of a generative one.

bryanhogan 2 days ago

Hearing about use cases for decision models and seeing how they are being explored is quite interesting, but I think the title of this post is poorly chosen.

munishjindal931 1 day ago

but many good things do need LLM. yes its so true that not everything needs that but how the things are moving it feels like everything soon will start needing it

saejox 2 days ago

I have been using structured json outputs since gpt 4 days. Never understood this jev hype

  • acedTrex 2 days ago

    i think the main deal is speed right?

    • triCoder 2 days ago

      No, the main deal is that it CANNOT answer with anything outside the given structured options. LLMs can go off the rails, however small a chance it is. The architecture behind JEV and the like doesn't let it do that.

      • j16sdiz 2 days ago

        > structured json outputs

    • atq2119 2 days ago

      Right. Producing JSON still requires the decode part of inference, which is not as compute efficient. Decider models are in some sense equivalent to producing only a single token. At least in performance terms.

aabdi 2 days ago

Contrarian take. I think everyone should use LLMs and the most expensive possible, unless they know how to run evals (both on line and off line).

Yes its expensive, ugly, etc. But...

I've seen enough IRL that people just create really bad evals that do not reflect real world. Then the product is just bad. Outright broken for the p90 customer.

People don't know statistics. Offline Evals are hard. Sampling the entire population distribution is hard. Sampling the fat tail is harder. Online evals are big ops projects that are giant balls of death.

  • abhgh 2 days ago

    Maybe I should write up a post about this, but about evals ... I have feelings. Yes you're right that writing evals is hard. And it comes down to the simple fact that the thing that you're testing - a LLM-driven feature that presumably suffers from variance and inadequate accuracy - and your measuring device - an LLM judge - suffer from the fundamental problems. But the lure of natural language is such that instead of seeing this pipeline of LLM-driven things as a path of cascading errors, it is easy to view it as one set of problems correcting for another set.

    In this world, the LLM-driven feature is a probabilistic entity, and the judge layered on top of that is another probabilistic entity. The real work is to look at the judge's response and infer the performance of the feature. Good evals make the inference step "pass-through", i.e., the judge outcomes have a 1:1 mapping with feature outcomes. But this kind of reliability often doesn't naturally appear. I think of reliability as being composed of three attributes: (a) consistency: do we get the same result for an eval run on the same llm with the same input (b) inter-llm agreement: do outcomes agree over multiple trials of running an eval over multiple llms with the same input (c) accuracy: does the eval agree with human judgement. You need extra work to guarantee these properties. And even when you can, you won't get 100% reliability, but say high 90s, and you still to make a judgement call as to call a feature "functional".

    • aabdi 2 days ago

      Two things, my problem isn't with the probabilistic nature, its:

      1. with the quality of the data that customers use to eval.

      2. with the lack of rigor/ops/testing around accuracy for deployment and regression.

      I'll give you examples for failures IRL:

      Offline:

      - sample distribution for test data didn't have p99 customers in it, failed in production when it launched for those customers.

      - test data sets were all synthetic, launch to real customers fails because data set was not representative of real.

      Online:

      - yeah we used LLM as a judge in prod/online eval, but the judge sampling failed to measured against our japanese customers, our bilingual customers, customers with media devices etc.

      - the LLM as a judge was used, but there was no measure of human accept rate measures/alarms for various use cases (lightbulbs turning off, music playing, etc) so the lightbulbs started failing.

      - there was no consistent human sampling of online data, so for example samsung TVs stopped working, but no one caught it until 3 months later when some VP was watching his TV and it broke and was accelerated despite it affecting 2-3% of customers.

  • dieselgate 2 days ago

    Can you elaborate on what you mean by Eval? All I can think of is eval() in js...

    I hope you agree people should use the "right tool for the job" meaning a calculator for arithmetic rather than an LLM for everything.

    • dominotw 2 days ago

      its basically assertEquals(x,y) where function is calling llm to check equals

    • aabdi 2 days ago

      Evals are tests of model accuracy. For example here are 100 people's sample houses, go test if the agent can turn off the lightbulbs correctly.

      Right job sure. But, If you're replacing the LLM with a jev like as in the article, or switching with a smaller classifier, then you better have good data/evals otherwise you're reducing accuracy without you even knowing.

  • mrbungie 2 days ago

    If you are not getting feedback from those p90 then your bad evals are the least of your problems.

    Also, unless those p90 customers are bringing a big portion of your income (again, if that happens you'd KNOW that your product is broken for them) I'd not pay for the most expensive LLM to "avoid doing evals".

    • aabdi 2 days ago

      >If you are not getting feedback from those p90 then your bad evals are the least of your problems.

      the feedback is part of your online evals... if you have bad feedback you have bad evals

      > Also, unless those p90 customers are bringing a big portion of your income (again, if that happens you'd KNOW that your product is broken for them) I'd not pay for the most expensive LLM to "avoid doing evals".

      they are but that's besides the point.

      Two ways:

      - logically ROI of accuracy degradation as compared to cost has to be measured, sure.

      - emotionally, morally sub 10% groups in our customer distribution include such things as disabled people, minorities, and the elderly. In my view you should love all your customers.

dieselgate 2 days ago

Completely unrelated to the article itself but I landed on the posthog.com homepage after viewing their open positions and the landing page is cool. It's sort of a desktop UI and different from what I usually see, was expecting some other HR domain to come up.

nandanadileep29 2 days ago

very true, many LLM usecases can be replaced with decision models like these, its wayy cheaper and faster.

vibebingo 2 days ago

> What is the current conservation status of hedgehogs?

A harnessed LLM can look this up using tools.

Maybe LLM at least ti prepare context for decision model.

I guess if using a decision model context prep is key.

codedrift_ 1 day ago

Not everything needs a decision model either. There's so many places people are shoving JEV in like it's a magic determinism layer. If your answer is squishy, then sure, use a squishy system.

> Decision models are up to 444.6x cheaper and 193.6x faster than frontier models

Yes, and two lines of even the worst JavaScript runs circles around making a service call

jakeydus 2 days ago

"Not everything needs an LLM! Use a decision model!"

And what is a decision model exactly? An LLM with instructions to limit responses?

Huh.

  • amelius 2 days ago

    Sounds like a yes/no question.

  • orbital-decay 2 days ago

    Sure it's an LLM, it's just neither generative nor instruction-tuned, like BERT (also an LLM), which gave them a reason to market it as not being one. Though modern "LLMs" are also modeling a lot more than just the language so they're kind of hard to define. Not much point in being pedantic, let them call anything anything and see what sticks.

Rapzid 2 days ago

The approach of using decision models to gate more expensive work.. I mean, yeah.

But using protecting PII as an example use case is a bit bonkers?

  • hdjrudni 2 days ago

    Realistically, how else would you do it? Can't exactly regex "does this contain PII". The alternative is to not attempt to protect the PII.

tchalla 2 days ago

Not everything needs a "decision model" with a "system one" API either but here we are.

jscd 2 days ago

I continue to be astonished that a real company exists called "post hog"

  • Terr_ 2 days ago

    Post hog, ergo proper hog.

cognitif 2 days ago

Wait, I don’t need an LLM to do arithmetic?

purusa0x6c 2 days ago

exactly.. ppl are still using LLMs agents for simple direct workflows..

pj_mukh 2 days ago

tl;dr Not everything needs an LLM, but everything needs a typesafe wrapper around an LLM.

Gotcha

  • acedTrex 2 days ago

    Ridiculous field we are in, people seemingly have forgotten the absolute basics in under a year.

    • LtWorf 2 days ago

      The people doing these things never knew the basics.