Playing Around With System One Models🔗

Jev🔗

I, like much of the software development world, have been struck by the excitement and interest around Typesafe.ai’s Jev. For a certain subset of problems, Jev promises ridiculously low inference costs at ridiculously low latencies. Obviously, for the post-token-maxxing universe, this is something of note.

Jev, for the uninitiated, isn’t a regular generative model. That is, given a bunch of text, it doesn’t in turn produce a bunch of text.

Jev is what Typesafe calls a System One model. That term is derived from the modes of thinking described in Daniel Kahneman’s book Thinking, Fast and Slow.
In it, Kahneman describes System One as :

Fast, automatic, frequent, emotional, stereotypic, unconscious. Examples (in order of complexity) of things system 1 can do

In Jev, you provide a prompt that describes the state and a series of questions you want evaluated about that state. These questions can be a noul (almost like a boolean choice), a choice (a selection between a list of options), and a score (an ordered list from a specified criteria set).

For example, let’s say you were evaluating App Store reviews for your products. You might study a review and ask:

• Should this have been a customer service call? a noul
• What type of issue did they encounter: a broken experience, slow performance, unavailable inventory, or something else? a choice
• What parts of the app were implicated in the poor experience: Home page, Checkout, Profile, Search? a score

In the Jev API, you would posit the customer review and ask it those questions, and receive a set of responses with probablity scores.

System One models are very interesting to me because a good chunk of the AI software that I’ve built is focused on routing, as I assume much of the world is. You don’t need probabilities if your universe is deterministic. But having to make a set of choices and restate your universe of options is something that we do every day, conciously or otherwise.

Laya-TS and System One Models On the Cheap🔗

Paying for something upfront before understanding why it’s better never sits right with me. Maybe that’s just a byproduct of having time on my hands right now, but digging into free tools first is the best way to establish a baseline. You don’t know where a paid service actually excels until you’ve pushed an open one to its limits.

Right as Jev was making the rounds, the CEO of Convai Innovations shared a writeup mentioning that he had tackled this same problem a year ago and open-sourced his software, Laya.

Because these models are so much lighter than standard LLMs, I wanted to see if I could get one running directly in the browser. I started playing around with Laya first, and this weekend Claude and I worked together to convert it to TypeScript and WebGPU and laya-ts was born. Initially, laya-ts is just a straight port of the Laya code to Typescript via WebGPU. I can imagine a good number of interesting things that can be done with low-ms based decisioning running right int he browser.

A day or so later, I also came across Von on Hugging Face. I haven’t had a chance to dig into Von’s weights or internals yet, but seeing that it shipped out of the box with an HTTP server that mimics Jev’s API gave me an idea: I decided to add a similar System One HTTP API to laya-ts as well, so it can be dropped into standard service workflows just as easily as in-browser apps.

Early Experiments🔗

Some concepts lend themselves very easiy to this domain,and the Typesafe team have helpfully enumerated out a few places to get the brain working. There are a few angles I’m working on.

Model and Context Routing🔗

Model routing seems an obtusely obvious scenario for these types of classifiers: given a piece of work, what model and effort level is likely to yield the best outcomes. For the the Local LLM crowd, this becomes even more interesting as you could imagine swapping out models if they were better suited to a task-type ( generating Go code vs tool call usage ).

An equally interesting application is the generation of a persistent knowledge graph over the course of a project that stitches enough context for every given task type. It’s well-known at this point that smaller contexts have a remarkable effect on performance and quality ( and not just at the local llm level )

I imagine a Pi plugin is well suited to this type of ideation.

Browser Use via Typed Decisions🔗

This was the OG idea. Assuming an System One model is good at making decisions, then if we can get good at presenting the right decisions, then every browser and evey device with a modicum of GPU power becomes a lightweight agent capable of getting a few things done.

My initial idea is pairing a “classic” LLM to structure the decisions about the page. It’s a two pronged appraoch: using a text-to-text LLM to study the DOM and ARIA tree or using a VLM to visually identify the screen elements and actions, create a set of choices about what next step to take, lather, rinse, repeat.

I’ve been building this out and the early returns are intruiging: