I Tested an AI Model Built to Make Decisions

Usually when a new model catches my attention, I run the same test against specific product tasks. A meeting debrief, release notes, and a pricing comparison. I want to see what it gets right, what it makes up, and whether I would actually use it.

Jev needed a different test.

TypeSafe describes Jev as a model built for fast, structured decisions. You give it information and define the question and possible answers. It returns a decision with probabilities. It can choose a category, score something, or assess whether something is true. It doesn't write you an explanation of its reasoning.

So asking it to write release notes wasn't going to tell me much. I wanted a small, real decision I could understand well enough to check.

Their entity-alignment example caught my eye. Entity alignment is deciding whether records from different sources describe the same thing. Their example uses beer catalogues, with names, breweries, styles, and alcohol content. My version involved people in meeting transcripts.

I have a personal knowledge system, my AI brain. It processes transcripts and connects people, meetings, decisions, and actions in a graph. A graph is a way of storing things along with the relationships between them, so I can ask questions about the work rather than remember which meeting contained the answer.

Names are a problem here. Transcription changes spellings. Someone gets mentioned by first name in one meeting and full name in another. Sometimes two people have similar names. Sometimes two people with the same first name are discussed in the same meeting.

If the system connects a mention to the wrong person, information about one colleague could end up attached to another. That could affect an answer I get later. It seemed like a reasonable place to try a model whose whole job is making a focused decision.

I started with ten pairs of person records. I reviewed them myself, using three choices: same person, different people, or insufficient evidence. I then sent Jev the names alone and the names with selected transcript excerpts. My answers stayed out of the requests over to Jev.

With names alone, Jev deferred on all ten. With the excerpts, it deferred on nine and correctly separated one pair of different people. It didn't resolve any of the four pairs I had identified as the same person.

That was not especially useful.

With this type of model, I needed to understand if it need more context, or is the point that it understands language and can figure this out?

Of course, understanding language doesn't give it the context I have from working with the people involved. A spelling variation might be obvious to me because I know who does that work. The model was getting a name, a meeting header, and sometimes a very short mention.

I put together a reference table from a resourcing map and details I supplied. Then I narrowed the question. Instead of asking whether two potentially messy graph records were the same person, I asked who a particular mention referred to. Each case got an excerpt and two candidate profiles, with options for someone else or insufficient evidence.

This matters because a whole record can already contain mistakes. Deciding who one mention refers to is a smaller job than deciding whether every relationship attached to an existing record belongs to the same person.

I reviewed six cases before I ran the revised test. Jev agreed with five of my six judgments. Two were straightforward examples where the source included a full name, so I wouldn't describe those as impressive detective work. The useful examples were two mentions of the same name. One described an account manager. Another described someone as the engineering team's boss. Jev selected the person I expected in each case.

It also deferred on mention of a name that can be multiple people that I couldn't identify from the supplied excerpt either. That was enough to make the approach interesting. It wasn't enough to say I had solved identity matching.

As I run local models, I felt I needed to also ask could the local LLM also just do this work.

I gave my local Qwen3-30B model the same excerpts, profiles, decision instructions, and choices. The format changed to a chat request asking for a small JSON answer, which is a format software can read directly. I withheld my judgments and Jev's results, and each case ran independently.

Both models selected the same four people correctly. The difference was in the other two cases.

This is a small result, but it shows why counting completed answers would be a poor way to judge this workflow. Qwen filled every slot. That would be convenient right up until the system used one of those unsupported choices to connect information to a person.

Speed turned out to be less of a concern than I expected. The whole local batch took about four and a half seconds, and all six responses were valid JSON. Jev averaged about a quarter of a second per request; local Qwen averaged about seven-tenths. Those timings included different network paths, so this wasn't a controlled speed benchmark. For my background transcript processing, both were quick. The six Jev calls cost about $0.00023.

However, I can see where a fast decision model could fit into a larger workflow. The application would find possible people, gather the relevant evidence, ask Jev to choose, and decide whether to accept that choice or send it for review. The model doesn't assemble that workflow for you. Someone still has to decide what it sees, what the choices mean, and what happens when it can't make the call.

For my own AI brain, I paused here. I haven't established that identity errors are causing enough trouble in the answers I actually use to justify maintaining a people directory, changing extraction, and adding a review process.

The whole point was to understand Jev, and this experiment gave me a concrete way to do that. Jev is fast, cheap, and appears to be accurate. I can see several use cases to choose this model at scale.