JevRL: TypeSafe Jev Score as Reinforcement Learning Reward

By BringAI Team · September 22, 2026

What happens when a structured decision model supplies the reward for a reinforcement learning agent? We built JevRL to make that question visible: four classic games, a neural policy that learns from JEV judgments, and saved checkpoints you can actually watch.

Across three training seeds, the final JEV-trained policies achieved 100.0% success on CartPole, 94.7% on MountainCar, 77.3% on Acrobot, and 71.7% on slippery FrozenLake. A frozen collection of 129 official JEV judgments supplied rewards for 1.08 million training steps. The recorded API cost of constructing that collection was $0.002409918, about 0.24 US cents, excluding training compute and development requests.

The numbers above the games are final held-out success rates. Curves show development total score, with mean ± sample standard deviation across three training seeds. Each checkpoint uses 20 development episodes. Human design overlaps JEV in MountainCar and Acrobot. Open the full-size figure.

Watch the games and explore every checkpoint · Get the code · Inspect the experiment records

The experiment also gives a useful qualification: a deterministic Human design control matches the MountainCar and Acrobot results exactly, and matches mean FrozenLake success. Much of the performance comes from the features and rubric we supplied. JevRL demonstrates a working, inspectable reward interface; these results do not establish that JEV outperforms a carefully designed reward function or a tuned RL baseline.

A judge inside the learning loop

The agent chooses an action. The environment advances. We turn that transition into a small structured description and ask JEV to evaluate it against an ordered rubric. A numerical reward derived from the answer goes into the agent's replay buffer, where ordinary Deep Q-Network updates train its policy.

JEV supplies the training reward; the agent chooses every action. The figure's “fast, typesafe, and accurate” phrase describes the intended interface qualities. This experiment does not benchmark service latency or establish calibrated scoring accuracy. The cache audit below shows why that distinction matters.

The policy sees the original Gymnasium observation. JEV sees our compact transition description. The original game reward is logged separately and is never added to the JEV training reward. Evaluation runs on the original environment rewards, without calling a judge.

This separates two questions: does the agent receive a useful learning signal, and does it actually become better at the game?

How does a typed score become a continuous reward?

JEV's Score primitive returns probabilities over ordered descriptive levels. Its built-in score is a probability-weighted position on those levels. JevRL consumes the returned probabilities and maps them onto a task-specific numerical reward scale.

For a transition description z, probabilities p, and designer-selected values v, the adapter computes:

normalized_p[i] = p[i] / sum(p)
reward(z) = sum(normalized_p[i] * v[i])

For CartPole, our six reward values are:

v = [-1.00, 0.10, 0.25, 0.50, 0.75, 1.00]

A real retained answer for an active episode, low pole deviation, and a cart away from the boundary puts probability 0.98 on level 4 and 0.02 on level 5. The resulting reward is:

reward = 0.98 * 0.75 + 0.02 * 1.00 = 0.755

Yes, the resulting reward is continuous. JEV judges which descriptive levels fit; our adapter calculates the weighted numerical reward. It is not asking JEV to invent an arbitrary decimal reward in free text. The numerical values, rubric, and transition features are all explicit human design choices.

The adapter checks the response structure and probability values. It rejects invalid answers rather than silently replacing them with rule-based rewards. It retains the original probabilities and served-model identity. Confidence is logged, but does not rescale the reward in this experiment. A typed response makes the boundary easier to validate; it does not guarantee that the judgment is correct.

Four games, four small descriptions

We use the official Gymnasium environments, including their standard time limits and stochastic slipping in FrozenLake:

• CartPole-v1: balance a pole for 500 steps. The judge sees a failure flag, a band of absolute pole deviation, and whether the cart is near a boundary. Stable states receive more reward than unstable states.
• MountainCar-v0: build momentum to reach the right-hand flag within 200 steps. We provide a discretized change in an engineered mechanical-energy quantity, a hill-height band, and a goal flag.
• Acrobot-v1: swing a two-link pendulum above the target height within 500 steps. We describe tip-height change, absolute height, and goal completion.
• FrozenLake-v1: reach the goal on the slippery 4×4 map within 100 steps. We describe the landed tile and whether shortest safe-path distance improved, worsened, or stayed unchanged.

FrozenLake's distance feature supplies knowledge of the map. MountainCar's energy feature also supplies substantial task knowledge. These abstractions make the scoring problem compact, but they limit what the results can establish about JEV's own reasoning and transfer to new tasks.

129 judgments, reused throughout training

The descriptions have finite combinations: 20 for CartPole, 50 for MountainCar, 50 for Acrobot, and 9 for FrozenLake. We collected one retained answer for each combination from typesafe/jev-1.13-20260917, served by TypeSafe through OpenRouter, then froze those 129 answers before the main training runs.

Every training step looks up the judgment for its exact abstract description. This is reuse of recorded model answers, with the task, rubric, abstraction version, reward values, and provider bound to the cache. A missing entry in frozen mode is an error. The training seeds share one frozen table, so our experiment measures variation in the learner, not variation across independently queried judges.

Some enumerated combinations are physically unreachable. Exhaustive enumeration covers the adapter's input space; it is not a sample of states an agent typically visits.

This design explains the small API bill. No new API calls occur during frozen-cache training or evaluation. Optional live scoring is available for fresh judgments, but it has a different cost profile. These costs cannot be extrapolated to judging raw video frames or arbitrary continuous states.

An experiment you can audit

We trained the same Stable-Baselines3 DQN under three reward conditions:

• Native: the environment's original reward.
• Human design: a deterministic implementation of the same written rubric and numerical scale.
• JEV: the probability-weighted reward from the frozen official judgments.

Each condition uses training seeds 7, 19, and 42. CartPole and FrozenLake receive 60,000 environment steps per run; MountainCar and Acrobot receive 120,000. That produces 36 runs, 3.24 million total training steps, and 360 saved checkpoints. Twelve of those runs, totaling 1.08 million steps, use JEV reward.

The DQN uses two hidden layers of 64 units, learning rate 0.001, discount 0.99, batch size 64, replay capacity 50,000, and 1,000 warm-up steps. It performs one update every four environment steps, updates its target network every 1,000 steps, and decreases exploration from 1.0 to 0.05 over the first half of training. Shared settings control implementation differences; they do not give each reward condition an independently optimized baseline.

We save the untrained policy, checkpoints every 10,000 steps, and the final policy after its last optimizer update. Each checkpoint gets 20 development episodes with reset seeds 10000–10019. Final results use 100 separate test episodes per policy, with reset seeds 100000–100099. We report every final policy, rather than choosing the best development checkpoint.

Total score means the sum of the game's original rewards during an episode, regardless of which reward trained the agent. Success rate measures completion of the task-specific objective. There is no supervised-learning test loss here; DQN's optimization loss is not a replacement for either outcome metric.

What the final policies achieved

The following results are mean ± sample standard deviation across three trained policies, each tested on 100 episodes. The 300 evaluation episodes per condition are not 300 independent training seeds. Success-rate deviations are percentage points. Higher total score is better in all four games, including the games with negative scores.

CartPole

• JEV: total score 500.00 ± 0.00; success 100.0% ± 0.0 pp.
• Human design: total score 474.96 ± 43.13; success 77.0% ± 39.0 pp.
• Native: total score 142.17 ± 27.91; success 0.0% ± 0.0 pp.

The JEV condition reaches the full 500-step objective in every final test episode. The Native condition's zero success does not mean it never learned to balance: its average episode lasts about 142 steps, below our strict 500-step success threshold. CartPole's usual +1-per-step reward is a valid learning signal. The weak result here concerns this shared configuration and budget, not the ability of neural networks to learn from that reward.

One complete, time-compressed rollout: training seed 7, replay seed 10000. A successful replay illustrates behavior; the success rate above comes from separate aggregate evaluation.

MountainCar

• JEV: total score −147.57 ± 9.29; success 94.7% ± 9.2 pp.
• Human design: total score −147.57 ± 9.29; success 94.7% ± 9.2 pp.
• Native: total score −200.00 ± 0.00; success 0.0% ± 0.0 pp.

The JEV and Human design reward tables are numerically identical on every enumerated input. Their paired training runs consequently produce the same measured results. This is evidence that the engineered energy-based reward is useful under this setup; it does not demonstrate a distinct advantage from model judgment.

Acrobot

• JEV: total score −210.91 ± 201.86; success 77.3% ± 39.3 pp.
• Human design: total score −210.91 ± 201.86; success 77.3% ± 39.3 pp.
• Native: total score −394.41 ± 182.89; success 30.0% ± 52.0 pp.

The JEV-trained policies have final success rates of 100%, 32%, and 100% across the three seeds. That spread matters more than a smooth-looking average. Human design matches each paired result, even though a few cached Acrobot rewards differ slightly from the deterministic values.

FrozenLake

• JEV: total score 0.717 ± 0.023; success 71.7% ± 2.3 pp.
• Human design: total score 0.717 ± 0.023; success 71.7% ± 2.3 pp.
• Native: total score 0.690 ± 0.000; success 69.0% ± 0.0 pp.

Total score equals success rate expressed as a fraction because the environment gives 1 for reaching the goal and 0 otherwise. JEV and Human design have the same aggregate mean, while individual-seed outcomes differ. The game retains its stochastic slipping dynamics.

An independently sampled uniform-random policy achieves success rates of 0%, 0%, 3%, and 1% on the four tasks, respectively. The learned JEV policies exceed that reference. The comparison with Native is less conclusive: the CartPole and MountainCar baselines are weak, and no reward condition received its own hyperparameter search.

Successful training does not certify the judge

We compared every cached judgment's most probable level against the deterministic rubric. Agreement is 100% for CartPole, MountainCar, and Acrobot, but only 55.6% for FrozenLake. This is agreement with our rule implementation on an enumerated input set, not an estimate of calibrated accuracy on naturally visited states.

A reachable FrozenLake example makes the issue concrete. For tile=goal, distance_change=closer, JEV puts probability 0.90 on “closer” and 0.10 on “goal.” With our values, that becomes:

JEV reward = 0.90 * 0.04 + 0.10 * 1.00 = 0.136
Human design reward = 1.00

The agent can still learn a useful policy despite this disagreement. That does not make the answer consistent with the rubric's intended terminal-event priority. More explicit event descriptions, separate scoring questions, and independently repeated cache construction are natural next experiments.

CartPole's improvement over Human design also needs care. Soft probabilities coincide with better outcomes in these three runs, but we have not isolated their causal contribution. Repeated caches and a controlled reward-perturbation ablation would be needed to test that explanation.

The actual API cost

The retained cache records the following construction costs:

• CartPole: 20 judgments; $0.000385896.
• MountainCar: 50 judgments; $0.000937440.
• Acrobot: 50 judgments; $0.000933240.
• FrozenLake: 9 judgments; $0.000153342.

Total: 129 calls, 57,379 input tokens, 2,193 output tokens, and $0.002409918. These are recorded usage figures for constructing this particular cache, not a quote for current API pricing or the total cost of the project. CPU training, parser-development diagnostics, and the earlier Key Quest demonstration are excluded.

The public score records retain the answers and usage data. Reusing those frozen answers incurs no additional scoring requests; collecting new live judgments does.

Watch a policy change, or train your own

The interactive demo places four games side by side at 0, 10,000, 30,000, and final training steps. It includes unsuccessful intermediate checkpoints. You can also select a game, reward source, training seed, and saved checkpoint, then inspect its replay and learning curves.

For local training with the recorded official rewards, no API key is needed:

git clone https://github.com/Bring-AI/jev-rl.git
cd jev-rl
uv run --extra classic jev-arcade serve

Open the address printed by the server and choose recorded JEV scores. To run a single CartPole experiment from the terminal:

uv run --extra classic jev-arcade classic \
  --game cartpole --provider jev \
  --steps 60000 --checkpoint-steps 10000 \
  --cache experiments/jevrl-v1/judgments/cartpole.json \
  --frozen-cache --output runs/my-cartpole

Each checkpoint contains a policy/optimizer snapshot and evaluation/replay records. The replay buffer is not saved, so these files support policy inspection and reuse rather than bitwise continuation of the entire training process. The public website serves recorded results; new training runs execute locally.

What this experiment establishes

JevRL makes a typed model judgment usable as an RL reward, and preserves enough evidence to inspect the path from an answer to an agent's behavior. The adapter, score caches, all 36 run records, and interactive replays are available in the open-source project.

Its scope is deliberately narrow. The task features and reward scales are human-supplied, the evaluation uses only three training seeds, all JEV runs share one score table, and the baselines are not separately tuned. The reward replacement also has no general guarantee of preserving the original task's optimal policy. There is no evidence here for general game intelligence or automatic reward discovery.

The next useful tests are stronger tuned baselines, more training seeds, independently collected score tables, and tasks whose rubrics cannot be reproduced so directly by a few rules. Those experiments would help distinguish the value of the typed interface from the value added by the judge itself.

Context and sources

Learning from judgments has a substantial history. Deep reinforcement learning from human preferences learns reward predictors from human comparisons. Reward Design with Language Models studies language models as proxy reward functions. JevRL explores a specific implementation: TypeSafe's typed Score interface, probability-weighted rewards, frozen exact-input reuse, and a visible record of how policies learn.

• TypeSafe Score documentation describes the ordered levels, probabilities, and score fields.
• Stable-Baselines3 provides the DQN implementation used here.
• Protocol, summaries, and all run records document the reported experiment at source snapshot 36f89ce.

JevRL is an independent BringAI project. The application code is MIT-licensed; the repository does not redistribute JEV model weights. This article reports our own experiment and is not a TypeSafe endorsement.