Jev Is Not a Decision Model
There's a lot of talk about Jev, and I keep hearing people call it a decision model. TypeSafe describe it that way too:
System One models are a class of AI models built to make fast, structured decisions that software can use directly.
That's from their System One documentation.
To be fair, the rest of their docs don't read that way. I think a lot of people read the headline, kept the word decision, and stopped there.
Honestly, a lot of how I see Jev being used makes me a little uncomfortable.
I couldn't put my finger on why until I used it myself.
I ran a few examples through it and looked at what went in and what came out. That's when it clicked.
Jev gives us an answer, and a number for how confident it is in that answer. That's an opinion.
Then our code does something with it. We might trust it, or run another check first. We might just do whatever Jev said, but even that is a choice we wrote into the code.
Change the threshold from 0.9 to 0.6 and the outcome changes. Jev's answer didn't change. But we decided the threshold needed to change. We made that decision, not Jev. TypeSafe's confidence docs say it well: "Your code encodes the risk tolerance."
So what are we handing over? The opinion. Jev gives it in real time, and very well. But a lot of the demos I've seen, and the content I've read, talk as if Jev can make the decisions that belong to us.
Jev will be wrong sometimes. TypeSafe say so on their System One page: calibration "does not guarantee that an individual answer is correct."
So when Jev gets it wrong, where does the error sit?
With us.
Jev gave us an opinion and told us how confident it was. That's Jev's part. Ours comes after: we wrote the code that decides what that opinion means, and what happens because of it. "We did that because Jev said so" doesn't hold up in production.
We can ask Jev for its opinion. We can't hand it the decision, or the accountability.
So I won't call Jev a decision model. I'll say it gives an opinion, because that word keeps my own thinking straight about whose decision it was when something goes wrong.
That's where I am today. Let me know if you think I'm getting it wrong.