Fine-tuned checkpoint and model card
Browse files- .gitattributes +1 -0
- README.md +61 -0
- backbone_config/config.json +63 -0
- best.safetensors +3 -0
- config.json +197 -0
- state.json +15 -0
- tokenizer/chat_template.jinja +89 -0
- tokenizer/tokenizer.json +3 -0
- tokenizer/tokenizer_config.json +30 -0
- training/calibrate.py +81 -0
- training/common.py +98 -0
- training/nanojev.py +383 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: unknown
|
| 4 |
+
license_link: https://huggingface.co/C-Tianyu/NanoJev
|
| 5 |
+
base_model: C-Tianyu/NanoJev
|
| 6 |
+
library_name: pytorch
|
| 7 |
+
pipeline_tag: text-classification
|
| 8 |
+
tags:
|
| 9 |
+
- decision-model
|
| 10 |
+
- typed-decisions
|
| 11 |
+
- system-one
|
| 12 |
+
- code-review
|
| 13 |
+
- javascript
|
| 14 |
+
- typescript
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# NanoJev Code Review Decisions
|
| 18 |
+
|
| 19 |
+
[C-Tianyu/NanoJev](https://huggingface.co/C-Tianyu/NanoJev) fine-tuned to review JavaScript and TypeScript snippets. It answers
|
| 20 |
+
13 typed questions about a snippet: 10 flaw checks (yes/no), `has_flaw`, `primary_flaw` (a choice over 11 options)
|
| 21 |
+
and `quality` (a 5-level score).
|
| 22 |
+
|
| 23 |
+
## Results
|
| 24 |
+
|
| 25 |
+
Test split of JS/TS Code Review Decisions (`decisions` config): 300 tasks, 1,200 snippets, 15,600 decisions,
|
| 26 |
+
scored with the dataset's `score_decisions.py`. Latency is the median time for one snippet (13 questions) alone on
|
| 27 |
+
one H100 SXM.
|
| 28 |
+
|
| 29 |
+
| Model | Macro F1 @ 0.5 | Accuracy | Log loss | Brier | ECE | p50 ms |
|
| 30 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 31 |
+
| This model | 0.452 | 0.831 | 0.459 | 0.173 | 0.038 | 115.4 |
|
| 32 |
+
| This model + validation temperatures | 0.452 | 0.831 | 0.456 | 0.172 | 0.031 | 115.4 |
|
| 33 |
+
| Base model, zero-shot | 0.237 | 0.192 | 1.413 | 0.559 | 0.510 | 116.0 |
|
| 34 |
+
| TF-IDF + logistic regression | 0.502 | 0.830 | 0.449 | 0.172 | 0.030 | - |
|
| 35 |
+
|
| 36 |
+
Per-flag F1 at 0.5: unclear naming 0.93, noise comments 0.99, dead code 0.41, inconsistent style 0.43, over abstracted 0.00, too clever 0.18, duplicated logic 0.00, error handling problem 0.79, type safety hole 0.53, unsafe input 0.25.
|
| 37 |
+
|
| 38 |
+
## Use
|
| 39 |
+
|
| 40 |
+
This folder is a NanoJev bundle (`best.safetensors`, `config.json`, `backbone_config/`, `tokenizer/`).
|
| 41 |
+
Load it with NanoJev's `DecisionPredictor` from the
|
| 42 |
+
[NanoJev release](https://huggingface.co/C-Tianyu/NanoJev) (revision `unified-games-v1`), or with
|
| 43 |
+
`training/nanojev.py predict --ckpt <folder>`, which also builds the 13 questions.
|
| 44 |
+
|
| 45 |
+
Known weakness: after half an epoch `has_flaw` P(true) never drops below 0.545, so every clean snippet is called
|
| 46 |
+
flawed, and `over_abstracted` and `duplicated_logic` were not learned.
|
| 47 |
+
|
| 48 |
+
## Training
|
| 49 |
+
|
| 50 |
+
- Data: train split of JS/TS Code Review Decisions (2,400 tasks, 9,600 snippets); state = task, language, code.
|
| 51 |
+
- Full fine-tune of the Qwen3-0.6B backbone and choice head for half an epoch, 1,200 steps of 4 snippets, backbone at 2e-5 and head at 2e-4, bf16 autocast, one H100 (40 min). Every option is encoded with the state and question (27 paths per snippet).
|
| 52 |
+
- Best validation loss: 0.453. `training/` holds the script that trained and scored this model.
|
| 53 |
+
|
| 54 |
+
## Limits
|
| 55 |
+
|
| 56 |
+
The labels are synthetic and model-reviewed, so scores measure agreement with the dataset's reviewers, not with
|
| 57 |
+
human reviewers. Snippets are 5 to 80 lines of JS/TS; other languages and whole files are untested.
|
| 58 |
+
|
| 59 |
+
## License
|
| 60 |
+
|
| 61 |
+
Not stated. The NanoJev source is MIT and the Qwen3-0.6B base is Apache 2.0, but the NanoJev release does not state a license for its weights. Settle this with the NanoJev author before making this repo public.
|
backbone_config/config.json
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"Qwen3ForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"bos_token_id": 151643,
|
| 8 |
+
"dtype": "float32",
|
| 9 |
+
"eos_token_id": 151645,
|
| 10 |
+
"head_dim": 128,
|
| 11 |
+
"hidden_act": "silu",
|
| 12 |
+
"hidden_size": 1024,
|
| 13 |
+
"initializer_range": 0.02,
|
| 14 |
+
"intermediate_size": 3072,
|
| 15 |
+
"layer_types": [
|
| 16 |
+
"full_attention",
|
| 17 |
+
"full_attention",
|
| 18 |
+
"full_attention",
|
| 19 |
+
"full_attention",
|
| 20 |
+
"full_attention",
|
| 21 |
+
"full_attention",
|
| 22 |
+
"full_attention",
|
| 23 |
+
"full_attention",
|
| 24 |
+
"full_attention",
|
| 25 |
+
"full_attention",
|
| 26 |
+
"full_attention",
|
| 27 |
+
"full_attention",
|
| 28 |
+
"full_attention",
|
| 29 |
+
"full_attention",
|
| 30 |
+
"full_attention",
|
| 31 |
+
"full_attention",
|
| 32 |
+
"full_attention",
|
| 33 |
+
"full_attention",
|
| 34 |
+
"full_attention",
|
| 35 |
+
"full_attention",
|
| 36 |
+
"full_attention",
|
| 37 |
+
"full_attention",
|
| 38 |
+
"full_attention",
|
| 39 |
+
"full_attention",
|
| 40 |
+
"full_attention",
|
| 41 |
+
"full_attention",
|
| 42 |
+
"full_attention",
|
| 43 |
+
"full_attention"
|
| 44 |
+
],
|
| 45 |
+
"max_position_embeddings": 40960,
|
| 46 |
+
"max_window_layers": 28,
|
| 47 |
+
"model_type": "qwen3",
|
| 48 |
+
"num_attention_heads": 16,
|
| 49 |
+
"num_hidden_layers": 28,
|
| 50 |
+
"num_key_value_heads": 8,
|
| 51 |
+
"pad_token_id": null,
|
| 52 |
+
"rms_norm_eps": 1e-06,
|
| 53 |
+
"rope_parameters": {
|
| 54 |
+
"rope_theta": 1000000,
|
| 55 |
+
"rope_type": "default"
|
| 56 |
+
},
|
| 57 |
+
"sliding_window": null,
|
| 58 |
+
"tie_word_embeddings": true,
|
| 59 |
+
"transformers_version": "5.17.0",
|
| 60 |
+
"use_cache": false,
|
| 61 |
+
"use_sliding_window": false,
|
| 62 |
+
"vocab_size": 151936
|
| 63 |
+
}
|
best.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:25d6e2db15fec76421f6c5f5dabf79903d13bb960f43ad7417399d08e57d663b
|
| 3 |
+
size 2385039280
|
config.json
ADDED
|
@@ -0,0 +1,197 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"input": "/data/rwang/nanojev_sonic_sft_20260920/unified/hard",
|
| 3 |
+
"init_checkpoint": "/tmp/nanojev_sonic_sft_20260920/initial_checkpoint",
|
| 4 |
+
"output_dir": "/data/rwang/nanojev_sonic_sft_20260920/experiment/hard_lr1e5",
|
| 5 |
+
"stage": "sft",
|
| 6 |
+
"loss": "ce",
|
| 7 |
+
"balance": "task",
|
| 8 |
+
"policy_pool_weights": "/data/rwang/nanojev_sonic_sft_20260920/experiment/population_weights.json",
|
| 9 |
+
"retention_fraction": 0.25,
|
| 10 |
+
"reward_samples": 32,
|
| 11 |
+
"steps": 600,
|
| 12 |
+
"head_steps": 0,
|
| 13 |
+
"batch_questions": 24,
|
| 14 |
+
"microbatch_questions": 8,
|
| 15 |
+
"max_microbatch_tokens": 32768,
|
| 16 |
+
"max_length": 8192,
|
| 17 |
+
"eval_every": 100,
|
| 18 |
+
"seed": 17,
|
| 19 |
+
"backbone_lr": 1e-05,
|
| 20 |
+
"head_lr": 0.0001,
|
| 21 |
+
"head_warmup_lr": 0.001,
|
| 22 |
+
"weight_decay": 0.01,
|
| 23 |
+
"precision": "bf16",
|
| 24 |
+
"gradient_checkpointing": true,
|
| 25 |
+
"disable_native_triton": true,
|
| 26 |
+
"td_episodes": null,
|
| 27 |
+
"td_n_step": 3,
|
| 28 |
+
"td_weight": 0.0,
|
| 29 |
+
"target_update_every": 25,
|
| 30 |
+
"validate_only": false,
|
| 31 |
+
"resolved_policy_pool_weights": {
|
| 32 |
+
"maze/policy": 0.3333333333333333,
|
| 33 |
+
"snake/policy": 0.3333333333333333,
|
| 34 |
+
"shooting/policy/basic": 0.16666666666666666,
|
| 35 |
+
"shooting/policy/predict_position": 0.16666666666666666
|
| 36 |
+
},
|
| 37 |
+
"schema_version": "nanojev-unified-games-v1",
|
| 38 |
+
"model": "Qwen/Qwen3-0.6B",
|
| 39 |
+
"set_head": "attention",
|
| 40 |
+
"resolved_model_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
|
| 41 |
+
"initialization": "local DecisionModel warm start; fresh optimizer",
|
| 42 |
+
"init_weights_sha256": "38116340795de1c82369b7fe15819d92d79600a7b4dc7a3cd0d4390cb6782639",
|
| 43 |
+
"data_sha256": {
|
| 44 |
+
"/data/rwang/nanojev_sonic_sft_20260920/unified/hard/manifest.json": "e79f0c3a8559e480db3161781963b1ac7c69be83772eb889208532e4cc5ab80a",
|
| 45 |
+
"/data/rwang/nanojev_sonic_sft_20260920/unified/hard/train.jsonl": "bda8712f89d5db94d80f840f9620f18312c92baca2d8a6df3456218fc807acd0",
|
| 46 |
+
"/data/rwang/nanojev_sonic_sft_20260920/unified/hard/dev.jsonl": "605b40f56e4834b0f234f3328b1e6a8ed41174641986b229e94927a3f05f70bd",
|
| 47 |
+
"/data/rwang/nanojev_sonic_sft_20260920/unified/hard/calibration.jsonl": "6408aa09b9238c724c557c86b68b23f2359140911d84daac7693f9e347fa091b",
|
| 48 |
+
"/data/rwang/nanojev_sonic_sft_20260920/unified/hard/test.jsonl": "2f96cc4e45c67fa375980ff4f355b47716985cf5fb4a1ef034c5e65481b072c9",
|
| 49 |
+
"/data/rwang/nanojev_sonic_sft_20260920/unified/hard/ood.jsonl": "7a778752e0f7699bd709aa6c83bb1ffaaa205568c304909000039b40fb7ff636"
|
| 50 |
+
},
|
| 51 |
+
"implementation_sha256": "a4aedc3b7ad25d875af67943bef6dbc017799f46c4ae0a845fc200e1866fa6cd",
|
| 52 |
+
"continuation_policy_id": null,
|
| 53 |
+
"population_weights": {
|
| 54 |
+
"maze/policy": 0.3333333333333333,
|
| 55 |
+
"snake/policy": 0.3333333333333333,
|
| 56 |
+
"shooting/policy/basic": 0.16666666666666666,
|
| 57 |
+
"shooting/policy/predict_position": 0.16666666666666666
|
| 58 |
+
},
|
| 59 |
+
"policy_pool_weights_sha256": "66afd9cdb9f0479a77896671c201834db1e5889ea081f81aa068dc73359f0449",
|
| 60 |
+
"eligible_by_split_sampling_pool": {
|
| 61 |
+
"calibration/maze/policy": 188,
|
| 62 |
+
"calibration/shooting/policy/basic": 377,
|
| 63 |
+
"calibration/shooting/policy/predict_position": 1006,
|
| 64 |
+
"calibration/snake/policy": 137,
|
| 65 |
+
"dev/maze/policy": 213,
|
| 66 |
+
"dev/shooting/policy/basic": 384,
|
| 67 |
+
"dev/shooting/policy/predict_position": 958,
|
| 68 |
+
"dev/snake/policy": 157,
|
| 69 |
+
"ood/maze/policy": 202,
|
| 70 |
+
"ood/shooting/policy/basic": 589,
|
| 71 |
+
"ood/shooting/policy/predict_position": 1008,
|
| 72 |
+
"ood/snake/policy": 141,
|
| 73 |
+
"test/maze/policy": 211,
|
| 74 |
+
"test/shooting/policy/basic": 756,
|
| 75 |
+
"test/shooting/policy/predict_position": 1413,
|
| 76 |
+
"test/snake/policy": 116,
|
| 77 |
+
"train/maze/policy": 651,
|
| 78 |
+
"train/shooting/policy/basic": 3054,
|
| 79 |
+
"train/shooting/policy/predict_position": 6788,
|
| 80 |
+
"train/snake/policy": 400
|
| 81 |
+
},
|
| 82 |
+
"sampling": "stratified with replacement; per-cell exact population weights; train rows only",
|
| 83 |
+
"parameter_storage": "float32",
|
| 84 |
+
"forward_autocast": "bf16",
|
| 85 |
+
"objective": "Choice CE with explicit API or expert targets plus observed Boolean event loss",
|
| 86 |
+
"selection": "minimum fixed population-weighted dev CE including initial checkpoint; test only after selection",
|
| 87 |
+
"temperature": 1.0,
|
| 88 |
+
"temperature_fitted": false,
|
| 89 |
+
"deps": {
|
| 90 |
+
"torch": "2.14.0",
|
| 91 |
+
"transformers": "5.17.0",
|
| 92 |
+
"safetensors": "0.8.0"
|
| 93 |
+
},
|
| 94 |
+
"gpu": "NVIDIA A100-SXM4-80GB",
|
| 95 |
+
"schema_counts": {
|
| 96 |
+
"records": 18760,
|
| 97 |
+
"outcome_questions": 0,
|
| 98 |
+
"continuation_policy_id": null,
|
| 99 |
+
"questions_by_split_task_role": {
|
| 100 |
+
"calibration/maze/policy": 188,
|
| 101 |
+
"calibration/shooting/policy": 1383,
|
| 102 |
+
"calibration/snake/policy": 138,
|
| 103 |
+
"dev/maze/policy": 215,
|
| 104 |
+
"dev/shooting/policy": 1342,
|
| 105 |
+
"dev/snake/policy": 158,
|
| 106 |
+
"ood/maze/policy": 202,
|
| 107 |
+
"ood/shooting/policy": 1597,
|
| 108 |
+
"ood/snake/policy": 143,
|
| 109 |
+
"test/maze/policy": 211,
|
| 110 |
+
"test/shooting/policy": 2169,
|
| 111 |
+
"test/snake/policy": 116,
|
| 112 |
+
"train/maze/policy": 653,
|
| 113 |
+
"train/shooting/policy": 9842,
|
| 114 |
+
"train/snake/policy": 403
|
| 115 |
+
},
|
| 116 |
+
"eligible_by_split_task_role": {
|
| 117 |
+
"calibration/maze/policy": 188,
|
| 118 |
+
"calibration/shooting/policy": 1383,
|
| 119 |
+
"calibration/snake/policy": 137,
|
| 120 |
+
"dev/maze/policy": 213,
|
| 121 |
+
"dev/shooting/policy": 1342,
|
| 122 |
+
"dev/snake/policy": 157,
|
| 123 |
+
"ood/maze/policy": 202,
|
| 124 |
+
"ood/shooting/policy": 1597,
|
| 125 |
+
"ood/snake/policy": 141,
|
| 126 |
+
"test/maze/policy": 211,
|
| 127 |
+
"test/shooting/policy": 2169,
|
| 128 |
+
"test/snake/policy": 116,
|
| 129 |
+
"train/maze/policy": 651,
|
| 130 |
+
"train/shooting/policy": 9842,
|
| 131 |
+
"train/snake/policy": 400
|
| 132 |
+
},
|
| 133 |
+
"quarantined_policy_questions": 11,
|
| 134 |
+
"policy_questions_by_split_task_target_kind": {
|
| 135 |
+
"calibration/maze/api_policy_distribution": 188,
|
| 136 |
+
"calibration/shooting/expert_action": 1383,
|
| 137 |
+
"calibration/snake/api_policy_distribution": 138,
|
| 138 |
+
"dev/maze/api_policy_distribution": 215,
|
| 139 |
+
"dev/shooting/expert_action": 1342,
|
| 140 |
+
"dev/snake/api_policy_distribution": 158,
|
| 141 |
+
"ood/maze/api_policy_distribution": 202,
|
| 142 |
+
"ood/shooting/expert_action": 1597,
|
| 143 |
+
"ood/snake/api_policy_distribution": 143,
|
| 144 |
+
"test/maze/api_policy_distribution": 211,
|
| 145 |
+
"test/shooting/expert_action": 2169,
|
| 146 |
+
"test/snake/api_policy_distribution": 116,
|
| 147 |
+
"train/maze/api_policy_distribution": 653,
|
| 148 |
+
"train/shooting/expert_action": 9842,
|
| 149 |
+
"train/snake/api_policy_distribution": 403
|
| 150 |
+
},
|
| 151 |
+
"unique_episodes_by_split_task_role": {
|
| 152 |
+
"calibration/maze/policy": 10,
|
| 153 |
+
"calibration/shooting/policy": 128,
|
| 154 |
+
"calibration/snake/policy": 8,
|
| 155 |
+
"dev/maze/policy": 10,
|
| 156 |
+
"dev/shooting/policy": 128,
|
| 157 |
+
"dev/snake/policy": 8,
|
| 158 |
+
"ood/maze/policy": 10,
|
| 159 |
+
"ood/shooting/policy": 256,
|
| 160 |
+
"ood/snake/policy": 8,
|
| 161 |
+
"test/maze/policy": 10,
|
| 162 |
+
"test/shooting/policy": 256,
|
| 163 |
+
"test/snake/policy": 8,
|
| 164 |
+
"train/maze/policy": 32,
|
| 165 |
+
"train/shooting/policy": 1024,
|
| 166 |
+
"train/snake/policy": 28
|
| 167 |
+
},
|
| 168 |
+
"rows_missing_episode_id_by_split_task_role": {},
|
| 169 |
+
"outcome_episode_counts_by_split_task": {},
|
| 170 |
+
"outcome_state_label_counts_by_split_task": {},
|
| 171 |
+
"outcome_question_label_counts_by_split_task": {},
|
| 172 |
+
"episode_audit_note": "Repeated states/questions are not independent episodes. Single-outcome cells are retained without forced balancing. Missing episode IDs are counted explicitly and excluded from episode totals."
|
| 173 |
+
},
|
| 174 |
+
"predictive_samples_are_physical_actions": false,
|
| 175 |
+
"frozen_continuation_policy_is_updated_by_this_script": false,
|
| 176 |
+
"td_enabled": false,
|
| 177 |
+
"td_audit": null,
|
| 178 |
+
"td_implementation_sha256": null,
|
| 179 |
+
"td_contract": {
|
| 180 |
+
"active": false,
|
| 181 |
+
"gamma": 1,
|
| 182 |
+
"n_step_unit": "recorded decision transitions",
|
| 183 |
+
"continuation_policy_id": null,
|
| 184 |
+
"bootstrap_policy": "recorded frozen behavior_probs; never online greedy",
|
| 185 |
+
"target_microbatch_questions": 4,
|
| 186 |
+
"max_microbatch_tokens": 32768,
|
| 187 |
+
"heldout_target": "observed terminal outcome only",
|
| 188 |
+
"outcome_loss": "(1-td_weight)*MC_Brier + td_weight*detached_soft_TD_Brier",
|
| 189 |
+
"initial_target": "deepcopy of initial online model; eval mode; no gradients",
|
| 190 |
+
"prediction_cache": "none; only bootstrap input tokens and indices are cached",
|
| 191 |
+
"bootstrap_cache": null
|
| 192 |
+
},
|
| 193 |
+
"fine_tuned": {
|
| 194 |
+
"from": "C-Tianyu/NanoJev@unified-games-v1",
|
| 195 |
+
"data": "js-ts-code-review-decisions-3000/decisions"
|
| 196 |
+
}
|
| 197 |
+
}
|
state.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"epoch": 0,
|
| 3 |
+
"step_in_epoch": 1200,
|
| 4 |
+
"step": 1200,
|
| 5 |
+
"val_loss": 0.45309440126499306,
|
| 6 |
+
"best_val_loss": 0.45309440126499306,
|
| 7 |
+
"history": [
|
| 8 |
+
{
|
| 9 |
+
"epoch": 0,
|
| 10 |
+
"step": 1200,
|
| 11 |
+
"val_loss": 0.45309440126499306
|
| 12 |
+
}
|
| 13 |
+
],
|
| 14 |
+
"train_layers": -1
|
| 15 |
+
}
|
tokenizer/chat_template.jinja
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- if tools %}
|
| 2 |
+
{{- '<|im_start|>system\n' }}
|
| 3 |
+
{%- if messages[0].role == 'system' %}
|
| 4 |
+
{{- messages[0].content + '\n\n' }}
|
| 5 |
+
{%- endif %}
|
| 6 |
+
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
| 7 |
+
{%- for tool in tools %}
|
| 8 |
+
{{- "\n" }}
|
| 9 |
+
{{- tool | tojson }}
|
| 10 |
+
{%- endfor %}
|
| 11 |
+
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
| 12 |
+
{%- else %}
|
| 13 |
+
{%- if messages[0].role == 'system' %}
|
| 14 |
+
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
| 15 |
+
{%- endif %}
|
| 16 |
+
{%- endif %}
|
| 17 |
+
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
| 18 |
+
{%- for message in messages[::-1] %}
|
| 19 |
+
{%- set index = (messages|length - 1) - loop.index0 %}
|
| 20 |
+
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
|
| 21 |
+
{%- set ns.multi_step_tool = false %}
|
| 22 |
+
{%- set ns.last_query_index = index %}
|
| 23 |
+
{%- endif %}
|
| 24 |
+
{%- endfor %}
|
| 25 |
+
{%- for message in messages %}
|
| 26 |
+
{%- if message.content is string %}
|
| 27 |
+
{%- set content = message.content %}
|
| 28 |
+
{%- else %}
|
| 29 |
+
{%- set content = '' %}
|
| 30 |
+
{%- endif %}
|
| 31 |
+
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
| 32 |
+
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
| 33 |
+
{%- elif message.role == "assistant" %}
|
| 34 |
+
{%- set reasoning_content = '' %}
|
| 35 |
+
{%- if message.reasoning_content is string %}
|
| 36 |
+
{%- set reasoning_content = message.reasoning_content %}
|
| 37 |
+
{%- else %}
|
| 38 |
+
{%- if '</think>' in content %}
|
| 39 |
+
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
|
| 40 |
+
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
| 41 |
+
{%- endif %}
|
| 42 |
+
{%- endif %}
|
| 43 |
+
{%- if loop.index0 > ns.last_query_index %}
|
| 44 |
+
{%- if loop.last or (not loop.last and reasoning_content) %}
|
| 45 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
| 46 |
+
{%- else %}
|
| 47 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 48 |
+
{%- endif %}
|
| 49 |
+
{%- else %}
|
| 50 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 51 |
+
{%- endif %}
|
| 52 |
+
{%- if message.tool_calls %}
|
| 53 |
+
{%- for tool_call in message.tool_calls %}
|
| 54 |
+
{%- if (loop.first and content) or (not loop.first) %}
|
| 55 |
+
{{- '\n' }}
|
| 56 |
+
{%- endif %}
|
| 57 |
+
{%- if tool_call.function %}
|
| 58 |
+
{%- set tool_call = tool_call.function %}
|
| 59 |
+
{%- endif %}
|
| 60 |
+
{{- '<tool_call>\n{"name": "' }}
|
| 61 |
+
{{- tool_call.name }}
|
| 62 |
+
{{- '", "arguments": ' }}
|
| 63 |
+
{%- if tool_call.arguments is string %}
|
| 64 |
+
{{- tool_call.arguments }}
|
| 65 |
+
{%- else %}
|
| 66 |
+
{{- tool_call.arguments | tojson }}
|
| 67 |
+
{%- endif %}
|
| 68 |
+
{{- '}\n</tool_call>' }}
|
| 69 |
+
{%- endfor %}
|
| 70 |
+
{%- endif %}
|
| 71 |
+
{{- '<|im_end|>\n' }}
|
| 72 |
+
{%- elif message.role == "tool" %}
|
| 73 |
+
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
| 74 |
+
{{- '<|im_start|>user' }}
|
| 75 |
+
{%- endif %}
|
| 76 |
+
{{- '\n<tool_response>\n' }}
|
| 77 |
+
{{- content }}
|
| 78 |
+
{{- '\n</tool_response>' }}
|
| 79 |
+
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
| 80 |
+
{{- '<|im_end|>\n' }}
|
| 81 |
+
{%- endif %}
|
| 82 |
+
{%- endif %}
|
| 83 |
+
{%- endfor %}
|
| 84 |
+
{%- if add_generation_prompt %}
|
| 85 |
+
{{- '<|im_start|>assistant\n' }}
|
| 86 |
+
{%- if enable_thinking is defined and enable_thinking is false %}
|
| 87 |
+
{{- '<think>\n\n</think>\n\n' }}
|
| 88 |
+
{%- endif %}
|
| 89 |
+
{%- endif %}
|
tokenizer/tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
|
| 3 |
+
size 11422650
|
tokenizer/tokenizer_config.json
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"backend": "tokenizers",
|
| 4 |
+
"bos_token": null,
|
| 5 |
+
"clean_up_tokenization_spaces": false,
|
| 6 |
+
"eos_token": "<|im_end|>",
|
| 7 |
+
"errors": "replace",
|
| 8 |
+
"extra_special_tokens": [
|
| 9 |
+
"<|im_start|>",
|
| 10 |
+
"<|im_end|>",
|
| 11 |
+
"<|object_ref_start|>",
|
| 12 |
+
"<|object_ref_end|>",
|
| 13 |
+
"<|box_start|>",
|
| 14 |
+
"<|box_end|>",
|
| 15 |
+
"<|quad_start|>",
|
| 16 |
+
"<|quad_end|>",
|
| 17 |
+
"<|vision_start|>",
|
| 18 |
+
"<|vision_end|>",
|
| 19 |
+
"<|vision_pad|>",
|
| 20 |
+
"<|image_pad|>",
|
| 21 |
+
"<|video_pad|>"
|
| 22 |
+
],
|
| 23 |
+
"is_local": true,
|
| 24 |
+
"local_files_only": true,
|
| 25 |
+
"model_max_length": 131072,
|
| 26 |
+
"pad_token": "<|endoftext|>",
|
| 27 |
+
"split_special_tokens": false,
|
| 28 |
+
"tokenizer_class": "Qwen2Tokenizer",
|
| 29 |
+
"unk_token": null
|
| 30 |
+
}
|
training/calibrate.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Temperature scaling for any model's answers: one temperature per question, fitted on validation answers.
|
| 2 |
+
|
| 3 |
+
python calibrate.py --val runs/m/finetuned.validation.jsonl --test runs/m/finetuned.test.jsonl \
|
| 4 |
+
--out runs/m/finetuned-calibrated.test.jsonl
|
| 5 |
+
|
| 6 |
+
Each answer's distribution p becomes softmax(log p / T). T is chosen by validation log loss, so the argmax and the
|
| 7 |
+
noul ranking never change; only confidence moves. Latencies are copied unchanged.
|
| 8 |
+
"""
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
import argparse
|
| 12 |
+
import json
|
| 13 |
+
import math
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
|
| 16 |
+
import numpy as np
|
| 17 |
+
|
| 18 |
+
from common import gold_index, load_split, options_of, to_answer
|
| 19 |
+
|
| 20 |
+
FLOOR = 1e-12
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def read(path: Path) -> dict[str, dict]:
|
| 24 |
+
return {r["id"]: r for r in map(json.loads, path.read_text().splitlines())}
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def distribution(answer: dict, question: dict) -> np.ndarray:
|
| 28 |
+
if question["type"] == "noul":
|
| 29 |
+
return np.array([answer["noul"], 1 - answer["noul"]])
|
| 30 |
+
return np.array([answer["probabilities"][o] for o in options_of(question)])
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def scaled(p: np.ndarray, t: float) -> np.ndarray:
|
| 34 |
+
z = np.log(np.clip(p, FLOOR, 1.0)) / t
|
| 35 |
+
z = np.exp(z - z.max(axis=-1, keepdims=True))
|
| 36 |
+
return z / z.sum(axis=-1, keepdims=True)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def fit(p: np.ndarray, gold: np.ndarray) -> float:
|
| 40 |
+
"""The temperature with the lowest mean log loss, by golden-section search over log T in [-3, 3]."""
|
| 41 |
+
def loss(log_t: float) -> float:
|
| 42 |
+
q = scaled(p, math.exp(log_t))
|
| 43 |
+
return float(-np.log(np.clip(q[np.arange(len(gold)), gold], FLOOR, 1.0)).mean())
|
| 44 |
+
|
| 45 |
+
lo, hi, ratio = -3.0, 3.0, (math.sqrt(5) - 1) / 2
|
| 46 |
+
a, b = hi - ratio * (hi - lo), lo + ratio * (hi - lo)
|
| 47 |
+
for _ in range(60):
|
| 48 |
+
if loss(a) < loss(b):
|
| 49 |
+
hi, b = b, a
|
| 50 |
+
a = hi - ratio * (hi - lo)
|
| 51 |
+
else:
|
| 52 |
+
lo, a = a, b
|
| 53 |
+
b = lo + ratio * (hi - lo)
|
| 54 |
+
return math.exp((lo + hi) / 2)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def main() -> None:
|
| 58 |
+
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
| 59 |
+
parser.add_argument("--val", type=Path, required=True)
|
| 60 |
+
parser.add_argument("--test", type=Path, required=True)
|
| 61 |
+
parser.add_argument("--out", type=Path, required=True)
|
| 62 |
+
args = parser.parse_args()
|
| 63 |
+
val, test = read(args.val), read(args.test)
|
| 64 |
+
cases = [c for c in load_split("validation") if c["id"] in val]
|
| 65 |
+
questions = cases[0]["questions"]
|
| 66 |
+
temperatures = {}
|
| 67 |
+
for name, q in questions.items():
|
| 68 |
+
p = np.stack([distribution(val[c["id"]]["answers"][name], q) for c in cases])
|
| 69 |
+
gold = np.array([gold_index(q, c["gold"][name]) for c in cases])
|
| 70 |
+
temperatures[name] = fit(p, gold)
|
| 71 |
+
with args.out.open("w") as handle:
|
| 72 |
+
for case_id, record in test.items():
|
| 73 |
+
answers = {name: to_answer(q, scaled(distribution(record["answers"][name], q), temperatures[name]).tolist())
|
| 74 |
+
for name, q in questions.items()}
|
| 75 |
+
handle.write(json.dumps({**record, "answers": answers}) + "\n")
|
| 76 |
+
print(json.dumps({"validation_cases": len(cases), "test_cases": len(test),
|
| 77 |
+
"temperatures": {k: round(v, 3) for k, v in temperatures.items()}}, indent=1))
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
if __name__ == "__main__":
|
| 81 |
+
main()
|
training/common.py
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Shared pieces for fitting and answering the `decisions` config of JS/TS Code Review Decisions.
|
| 2 |
+
|
| 3 |
+
Every model script reads cases with `load_split` and writes answers with `PredictionWriter`, in the JSONL shape that
|
| 4 |
+
`score_decisions.py` scores: {"id", "answers": {name: System One answer}, "latency_ms"}.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import json
|
| 9 |
+
import os
|
| 10 |
+
import random
|
| 11 |
+
import time
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import pyarrow.parquet as pq
|
| 15 |
+
|
| 16 |
+
HERE = Path(__file__).resolve().parent
|
| 17 |
+
DATA = Path(os.environ.get("DECISIONS_DATA", HERE.parent / "js-ts-code-review-decisions-3000"))
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def load_split(split: str, limit: int | None = None, seed: int = 0) -> list[dict]:
|
| 21 |
+
"""Cases of one split: id, task_slug, state (task, language, code), questions, and gold label per question.
|
| 22 |
+
`limit` keeps whole tasks (all 4 variants) chosen at random, so a small run still has clean and flawed code."""
|
| 23 |
+
table = pq.read_table(DATA / "decisions" / f"{split}.parquet", columns=["id", "task_slug", "state", "questions", "gold"])
|
| 24 |
+
rows = table.to_pylist()
|
| 25 |
+
if limit is not None and limit < len(rows):
|
| 26 |
+
tasks = sorted({r["task_slug"] for r in rows})
|
| 27 |
+
random.Random(seed).shuffle(tasks)
|
| 28 |
+
keep = set(tasks[: max(1, limit // 4)])
|
| 29 |
+
rows = [r for r in rows if r["task_slug"] in keep]
|
| 30 |
+
return [{"id": r["id"], "task_slug": r["task_slug"], "state": json.loads(r["state"]),
|
| 31 |
+
"questions": json.loads(r["questions"]),
|
| 32 |
+
"gold": {name: g["label"] for name, g in json.loads(r["gold"]).items()}} for r in rows]
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def options_of(question: dict) -> list[str]:
|
| 36 |
+
"""The option labels `score_decisions.py` scores, in order: true/false, choice keys, or level indexes."""
|
| 37 |
+
if question["type"] == "noul":
|
| 38 |
+
return ["true", "false"]
|
| 39 |
+
if question["type"] == "choice":
|
| 40 |
+
return list(question["criteria"])
|
| 41 |
+
return [str(i) for i in range(len(question["criteria"]))]
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def gold_index(question: dict, label: str) -> int:
|
| 45 |
+
return options_of(question).index(label)
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def to_answer(question: dict, probs: list[float]) -> dict:
|
| 49 |
+
"""A System One answer from probabilities in `options_of` order (a noul's are [true, false])."""
|
| 50 |
+
probs = [float(p) for p in probs]
|
| 51 |
+
if question["type"] == "noul":
|
| 52 |
+
return {"type": "noul", "noul": probs[0]}
|
| 53 |
+
return {"type": question["type"], "probabilities": dict(zip(options_of(question), probs))}
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def state_text(state: dict) -> str:
|
| 57 |
+
"""One string for models that take plain text: the task, the language, then the code."""
|
| 58 |
+
return f"Task: {state['task']}\nLanguage: {state['language']}\n\n{state['code']}"
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
class PredictionWriter:
|
| 62 |
+
"""Appends one answered case per line and flushes it, so a stopped run resumes where it left off."""
|
| 63 |
+
|
| 64 |
+
def __init__(self, path: str | Path):
|
| 65 |
+
self.path = Path(path)
|
| 66 |
+
self.path.parent.mkdir(parents=True, exist_ok=True)
|
| 67 |
+
self.done: set[str] = set()
|
| 68 |
+
if self.path.exists():
|
| 69 |
+
for line in self.path.read_text().splitlines():
|
| 70 |
+
try:
|
| 71 |
+
self.done.add(json.loads(line)["id"])
|
| 72 |
+
except (json.JSONDecodeError, KeyError):
|
| 73 |
+
break # a line cut short by a stop; everything before it is kept
|
| 74 |
+
self._handle = self.path.open("a")
|
| 75 |
+
|
| 76 |
+
def pending(self, cases: list[dict]) -> list[dict]:
|
| 77 |
+
return [c for c in cases if c["id"] not in self.done]
|
| 78 |
+
|
| 79 |
+
def write(self, case_id: str, answers: dict, latency_ms: float | None = None) -> None:
|
| 80 |
+
record = {"id": case_id, "answers": answers}
|
| 81 |
+
if latency_ms is not None:
|
| 82 |
+
record["latency_ms"] = round(latency_ms, 3)
|
| 83 |
+
self._handle.write(json.dumps(record) + "\n")
|
| 84 |
+
self._handle.flush()
|
| 85 |
+
self.done.add(case_id)
|
| 86 |
+
|
| 87 |
+
def close(self) -> None:
|
| 88 |
+
self._handle.close()
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def log(path: str | Path | None, **fields) -> None:
|
| 92 |
+
"""Prints a JSON line and appends it to `path` when given (training curves, timings)."""
|
| 93 |
+
line = json.dumps({"time": round(time.time(), 1), **fields})
|
| 94 |
+
print(line, flush=True)
|
| 95 |
+
if path:
|
| 96 |
+
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
| 97 |
+
with open(path, "a") as handle:
|
| 98 |
+
handle.write(line + "\n")
|
training/nanojev.py
ADDED
|
@@ -0,0 +1,383 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""NanoJev (C-Tianyu/NanoJev @ unified-games-v1) on the `decisions` config: answer zero-shot, fine-tune, answer again.
|
| 3 |
+
|
| 4 |
+
NanoJev encodes every candidate path as state + question + candidate in a Qwen3-0.6B backbone and scores the offered
|
| 5 |
+
set with its own head, with no decoding. A noul is posed as a NanoJev boolean (one path, logits [0, z]), `primary_flaw`
|
| 6 |
+
as a choice (11 paths and the attention set head) and `quality` as an ordered score (5 paths). Request format,
|
| 7 |
+
tokenization, forward and probabilities all come from NanoJev's own source, which is hash-checked before it is loaded.
|
| 8 |
+
|
| 9 |
+
python nanojev.py zeroshot --split test --out PRED.jsonl [--limit N] [--device D]
|
| 10 |
+
python nanojev.py train --ckpt DIR [--epochs 2] [--limit N] [--max-steps S] [--device D] [--log PATH]
|
| 11 |
+
python nanojev.py predict --ckpt DIR --split test --out PRED.jsonl [--limit N] [--device D]
|
| 12 |
+
|
| 13 |
+
Set NANOJEV_REPO to use another download directory; a missing release is fetched there (inference bundle only).
|
| 14 |
+
"""
|
| 15 |
+
from __future__ import annotations
|
| 16 |
+
|
| 17 |
+
import argparse
|
| 18 |
+
import contextlib
|
| 19 |
+
import functools
|
| 20 |
+
import hashlib
|
| 21 |
+
import importlib.util
|
| 22 |
+
import json
|
| 23 |
+
import math
|
| 24 |
+
import os
|
| 25 |
+
import random
|
| 26 |
+
import shutil
|
| 27 |
+
import time
|
| 28 |
+
from pathlib import Path
|
| 29 |
+
|
| 30 |
+
import torch
|
| 31 |
+
import torch.nn.functional as F
|
| 32 |
+
|
| 33 |
+
from common import PredictionWriter, load_split, log, options_of, state_text, to_answer
|
| 34 |
+
|
| 35 |
+
REPO_ID, REVISION = "C-Tianyu/NanoJev", "unified-games-v1"
|
| 36 |
+
REPO = Path(os.environ.get("NANOJEV_REPO", Path.home() / ".cache" / "decision-models-scratch" / "nanojev-repo"))
|
| 37 |
+
BUNDLE = ["best.safetensors", "config.json", "backbone_config/*", "tokenizer/*", "source/*", "*.md"]
|
| 38 |
+
SOURCE_SHA256 = {
|
| 39 |
+
"predict_toy_decisions.py": "6eedc49aaf9a81763677e3cff4e87fc4e499bfc5decf0bb4678f452f578dd92b",
|
| 40 |
+
"train_toy_decisions.py": "4f39babd5575e43d7acf3eca357329a93c60041f334bc2bd7424929bf9c0da86",
|
| 41 |
+
}
|
| 42 |
+
STATE_CAP = 1024 # state tokens; the longest train state is 992 tokens, so the cap truncates no train case
|
| 43 |
+
TYPES = {"noul": "boolean", "choice": "choice", "score": "score"}
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
@functools.cache
|
| 47 |
+
def nanojev_source():
|
| 48 |
+
"""NanoJev's `predict_toy_decisions` module, loaded by path after its files match the reviewed hashes."""
|
| 49 |
+
if not (REPO / "best.safetensors").exists():
|
| 50 |
+
from huggingface_hub import snapshot_download
|
| 51 |
+
snapshot_download(REPO_ID, revision=REVISION, local_dir=REPO, allow_patterns=BUNDLE, token=False)
|
| 52 |
+
scripts = REPO / "source" / "scripts"
|
| 53 |
+
for name, digest in SOURCE_SHA256.items():
|
| 54 |
+
if hashlib.sha256((scripts / name).read_bytes()).hexdigest() != digest:
|
| 55 |
+
raise RuntimeError(f"{scripts / name} is not the reviewed NanoJev source; refusing to run it")
|
| 56 |
+
spec = importlib.util.spec_from_file_location("nanojev_predict", scripts / "predict_toy_decisions.py")
|
| 57 |
+
module = importlib.util.module_from_spec(spec)
|
| 58 |
+
spec.loader.exec_module(module)
|
| 59 |
+
return module
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def load_model(root: Path, device: torch.device) -> tuple[torch.nn.Module, object, int]:
|
| 63 |
+
"""Model, tokenizer and path-length limit of one NanoJev bundle (the release or a fine-tuned checkpoint)."""
|
| 64 |
+
nj = nanojev_source()
|
| 65 |
+
if device.type == "cuda":
|
| 66 |
+
predictor = nj.DecisionPredictor(str(root), device_name=str(device), precision="bf16",
|
| 67 |
+
disable_native_triton=True)
|
| 68 |
+
return predictor.model, predictor.tokenizer, predictor.limit
|
| 69 |
+
# DecisionPredictor refuses non-CUDA devices, so on CPU these are its own loading steps without that check.
|
| 70 |
+
from safetensors.torch import load_file
|
| 71 |
+
from torch._native import triton_utils
|
| 72 |
+
from transformers import AutoConfig, AutoModel, AutoTokenizer
|
| 73 |
+
triton_utils.deregister_op_overrides()
|
| 74 |
+
_, paths = nj.local_checkpoint_files(root)
|
| 75 |
+
run = nj.read_json(paths["run_config"])
|
| 76 |
+
tokenizer = AutoTokenizer.from_pretrained(str(paths["tokenizer"]), local_files_only=True)
|
| 77 |
+
if tokenizer.pad_token_id is None:
|
| 78 |
+
tokenizer.pad_token = tokenizer.eos_token
|
| 79 |
+
config = AutoConfig.from_pretrained(str(paths["body_config"]), local_files_only=True)
|
| 80 |
+
config.use_cache = False
|
| 81 |
+
body = AutoModel.from_config(config, attn_implementation="sdpa").float()
|
| 82 |
+
model = nj.load_decision_model_class()(body, run["set_head"])
|
| 83 |
+
model.load_state_dict(load_file(str(paths["weights"])), strict=True)
|
| 84 |
+
return model.to(device).eval(), tokenizer, run.get("max_length", 512)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def autocast(device: torch.device):
|
| 88 |
+
"""BF16 autocast over FP32 weights on CUDA, as NanoJev was trained; plain FP32 on CPU."""
|
| 89 |
+
return torch.autocast("cuda", dtype=torch.bfloat16) if device.type == "cuda" else contextlib.nullcontext()
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def capped_state(case: dict, tokenizer, cap: int) -> tuple[str, bool]:
|
| 93 |
+
"""`common.state_text`, cut to `cap` tokens with a visible marker when longer."""
|
| 94 |
+
text = state_text(case["state"])
|
| 95 |
+
ids = tokenizer.encode(text, add_special_tokens=False)
|
| 96 |
+
if len(ids) <= cap:
|
| 97 |
+
return text, False
|
| 98 |
+
return tokenizer.decode(ids[:cap]) + "\n[code truncated]", True
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def request(cases: list[dict], states: dict[str, str]) -> dict:
|
| 102 |
+
"""NanoJev's request: one state per case with its 13 questions in NanoJev's boolean, choice and score types."""
|
| 103 |
+
return {"states": [{"id": case["id"], "state": states[case["id"]],
|
| 104 |
+
"questions": {name: {"type": TYPES[q["type"]], "instructions": q["instructions"],
|
| 105 |
+
"criteria": q["criteria"]} for name, q in case["questions"].items()}}
|
| 106 |
+
for case in cases]}
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def examples_for(cases: list[dict], states: dict[str, str], tokenizer, limit: int, gold: bool = True) -> list[dict]:
|
| 110 |
+
"""NanoJev's tokenized questions (all candidate paths each), with the gold candidate index for training."""
|
| 111 |
+
examples = nanojev_source().prepare_examples(request(cases, states), tokenizer, limit)
|
| 112 |
+
golds = {case["id"]: case["gold"] for case in cases} if gold else {}
|
| 113 |
+
for ex in examples if gold else []:
|
| 114 |
+
ex["gold_index"] = ex["candidate_ids"].index(golds[ex["state_id"]][ex["qid"]])
|
| 115 |
+
return examples
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def answers_of(case: dict, examples: list[dict], logits: torch.Tensor) -> dict:
|
| 119 |
+
"""System One answers in `options_of` order from NanoJev's per-question softmax over its candidates."""
|
| 120 |
+
answers = {}
|
| 121 |
+
for ex, values in zip(examples, logits):
|
| 122 |
+
probs = values[: len(ex["candidate_ids"])].float().softmax(-1).cpu().tolist()
|
| 123 |
+
result = nanojev_source().answer_from_probabilities(ex, probs)
|
| 124 |
+
question = case["questions"][ex["qid"]]
|
| 125 |
+
if question["type"] == "noul":
|
| 126 |
+
vector = [result["p_true"], 1 - result["p_true"]]
|
| 127 |
+
else:
|
| 128 |
+
vector = [result["probabilities"][o] for o in options_of(question)]
|
| 129 |
+
answers[ex["qid"]] = to_answer(question, vector)
|
| 130 |
+
return answers
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def microbatches(examples: list[dict], budget: int) -> list[list[dict]]:
|
| 134 |
+
"""Whole questions sorted by length and grouped so paths x longest path stays within `budget` tokens."""
|
| 135 |
+
groups, group, paths, width = [], [], 0, 0
|
| 136 |
+
for ex in sorted(examples, key=lambda e: max(map(len, e["leaf_tokens"]))):
|
| 137 |
+
n, size = len(ex["leaf_tokens"]), max(map(len, ex["leaf_tokens"]))
|
| 138 |
+
if group and (paths + n) * max(width, size) > budget:
|
| 139 |
+
groups.append(group)
|
| 140 |
+
group, paths, width = [], 0, 0
|
| 141 |
+
group.append(ex)
|
| 142 |
+
paths, width = paths + n, max(width, size)
|
| 143 |
+
return groups + [group] if group else groups
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def padded_tokens(group: list[dict]) -> int:
|
| 147 |
+
return sum(len(ex["leaf_tokens"]) for ex in group) * max(len(p) for ex in group for p in ex["leaf_tokens"])
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
def group_loss(model: torch.nn.Module, group: list[dict], pad: int, device: torch.device) -> torch.Tensor:
|
| 151 |
+
"""Summed cross entropy of each question's gold candidate; padded candidates sit at -1e9 and get no mass."""
|
| 152 |
+
with autocast(device):
|
| 153 |
+
logits, _ = model(group, pad)
|
| 154 |
+
gold = torch.tensor([ex["gold_index"] for ex in group], device=logits.device)
|
| 155 |
+
return F.cross_entropy(logits.float(), gold, reduction="sum")
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def bundle_of(ckpt: Path) -> Path:
|
| 159 |
+
"""The bundle `predict` loads from a checkpoint directory: final, else the last saved step, else DIR itself."""
|
| 160 |
+
for path in (ckpt / "final", ckpt / "last", ckpt / "last.old"):
|
| 161 |
+
if (path / "best.safetensors").exists():
|
| 162 |
+
return path
|
| 163 |
+
return ckpt
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
def answer_split(root: Path, args: argparse.Namespace) -> None:
|
| 167 |
+
"""Answers each pending case alone (all 13 questions in one forward) and records its wall-clock latency."""
|
| 168 |
+
device = torch.device(args.device)
|
| 169 |
+
model, tokenizer, limit = load_model(root, device)
|
| 170 |
+
writer = PredictionWriter(args.out)
|
| 171 |
+
cases = writer.pending(load_split(args.split, args.limit))
|
| 172 |
+
log(None, event="answer", split=args.split, bundle=str(root), device=str(device), pending=len(cases),
|
| 173 |
+
done=len(writer.done), state_cap=args.state_cap)
|
| 174 |
+
truncated, latencies = 0, []
|
| 175 |
+
with torch.inference_mode():
|
| 176 |
+
for i, case in enumerate(cases, 1):
|
| 177 |
+
start = time.perf_counter()
|
| 178 |
+
text, cut = capped_state(case, tokenizer, args.state_cap)
|
| 179 |
+
examples = examples_for([case], {case["id"]: text}, tokenizer, limit, gold=False)
|
| 180 |
+
with autocast(device):
|
| 181 |
+
logits, _ = model(examples, tokenizer.pad_token_id)
|
| 182 |
+
answers = answers_of(case, examples, logits)
|
| 183 |
+
if device.type == "cuda":
|
| 184 |
+
torch.cuda.synchronize()
|
| 185 |
+
elif device.type == "mps":
|
| 186 |
+
torch.mps.synchronize()
|
| 187 |
+
latencies.append((time.perf_counter() - start) * 1000)
|
| 188 |
+
writer.write(case["id"], answers, latencies[-1])
|
| 189 |
+
truncated += cut
|
| 190 |
+
if i % 50 == 0 or i == len(cases):
|
| 191 |
+
log(None, event="progress", answered=i, of=len(cases), last_ms=round(latencies[-1], 1))
|
| 192 |
+
writer.close()
|
| 193 |
+
if latencies:
|
| 194 |
+
ordered = sorted(latencies)
|
| 195 |
+
log(None, event="answered", cases=len(latencies), truncated=truncated, out=str(args.out),
|
| 196 |
+
median_ms=round(ordered[len(ordered) // 2], 1), mean_ms=round(sum(ordered) / len(ordered), 1))
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
def save_bundle(model: torch.nn.Module, dest: Path, state: dict, optimizer=None) -> None:
|
| 200 |
+
"""A NanoJev bundle that DecisionPredictor loads, plus state.json: written to DIR.tmp, then swapped in by rename."""
|
| 201 |
+
from safetensors.torch import save_file
|
| 202 |
+
tmp, old = dest.with_name(dest.name + ".tmp"), dest.with_name(dest.name + ".old")
|
| 203 |
+
shutil.rmtree(tmp, ignore_errors=True)
|
| 204 |
+
tmp.mkdir(parents=True)
|
| 205 |
+
for name in ("backbone_config", "tokenizer"):
|
| 206 |
+
shutil.copytree(REPO / name, tmp / name)
|
| 207 |
+
run = json.loads((REPO / "config.json").read_text())
|
| 208 |
+
run["fine_tuned"] = {"from": f"{REPO_ID}@{REVISION}", "data": "js-ts-code-review-decisions-3000/decisions"}
|
| 209 |
+
(tmp / "config.json").write_text(json.dumps(run, indent=1))
|
| 210 |
+
save_file({k: v.detach().cpu().contiguous() for k, v in model.state_dict().items()}, str(tmp / "best.safetensors"))
|
| 211 |
+
if optimizer is not None:
|
| 212 |
+
torch.save(optimizer.state_dict(), tmp / "optimizer.pt")
|
| 213 |
+
(tmp / "state.json").write_text(json.dumps(state, indent=1))
|
| 214 |
+
shutil.rmtree(old, ignore_errors=True)
|
| 215 |
+
if dest.exists():
|
| 216 |
+
dest.rename(old)
|
| 217 |
+
tmp.rename(dest)
|
| 218 |
+
shutil.rmtree(old, ignore_errors=True)
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def prepare_states(cases: list[dict], tokenizer, cap: int, limit: int) -> tuple[dict[str, str], int, int]:
|
| 222 |
+
"""Capped state text per case, how many were truncated, and the unpadded candidate-path tokens of one pass."""
|
| 223 |
+
states, truncated, tokens = {}, 0, 0
|
| 224 |
+
for case in cases:
|
| 225 |
+
states[case["id"]], cut = capped_state(case, tokenizer, cap)
|
| 226 |
+
truncated += cut
|
| 227 |
+
examples = examples_for([case], states, tokenizer, limit)
|
| 228 |
+
tokens += sum(len(path) for ex in examples for path in ex["leaf_tokens"])
|
| 229 |
+
return states, truncated, tokens
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
def validation_loss(model: torch.nn.Module, cases: list[dict], states: dict[str, str], tokenizer, limit: int,
|
| 233 |
+
device: torch.device, budget: int) -> float:
|
| 234 |
+
"""Mean per-question cross entropy over the validation cases."""
|
| 235 |
+
model.eval()
|
| 236 |
+
total, count = 0.0, 0
|
| 237 |
+
with torch.no_grad():
|
| 238 |
+
for i in range(0, len(cases), 4):
|
| 239 |
+
for group in microbatches(examples_for(cases[i:i + 4], states, tokenizer, limit), budget):
|
| 240 |
+
total += float(group_loss(model, group, tokenizer.pad_token_id, device))
|
| 241 |
+
count += len(group)
|
| 242 |
+
model.train()
|
| 243 |
+
return total / count
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
def set_trainable(model: torch.nn.Module, layers: int) -> list[torch.nn.Parameter]:
|
| 247 |
+
"""Trains the head plus the last `layers` decoder layers (and final norm); all layers means a full fine-tune."""
|
| 248 |
+
blocks = model.backbone.layers
|
| 249 |
+
full = layers >= len(blocks)
|
| 250 |
+
for param in model.backbone.parameters():
|
| 251 |
+
param.requires_grad_(full)
|
| 252 |
+
if not full:
|
| 253 |
+
for module in [*blocks[len(blocks) - layers:], model.backbone.norm] if layers else []:
|
| 254 |
+
module.requires_grad_(True)
|
| 255 |
+
return [p for p in model.parameters() if p.requires_grad]
|
| 256 |
+
|
| 257 |
+
|
| 258 |
+
def lr_scale(step: int, total: int, warmup: int) -> float:
|
| 259 |
+
"""Linear warmup over `warmup` steps, then linear decay to zero at `total`."""
|
| 260 |
+
return min(1.0, (step + 1) / warmup) * max(0.0, min(1.0, (total - step) / max(1, total - warmup)))
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
def train(args: argparse.Namespace) -> None:
|
| 264 |
+
"""Supervised cross entropy over every question's options; resumes from DIR/last and keeps the best as DIR/final."""
|
| 265 |
+
device, ckpt = torch.device(args.device), Path(args.ckpt)
|
| 266 |
+
last = next((p for p in (ckpt / "last", ckpt / "last.old") if (p / "state.json").exists()), None)
|
| 267 |
+
state = json.loads((last / "state.json").read_text()) if last else {
|
| 268 |
+
"epoch": 0, "step_in_epoch": 0, "step": 0, "val_loss": None, "best_val_loss": None, "history": [],
|
| 269 |
+
"train_layers": args.train_layers if args.train_layers is not None else (-1 if device.type == "cuda" else 2)}
|
| 270 |
+
train_cases = load_split("train", args.limit)
|
| 271 |
+
per_epoch = math.ceil(len(train_cases) / args.batch_cases)
|
| 272 |
+
total = args.epochs * per_epoch
|
| 273 |
+
stop = min(total, args.max_steps or total)
|
| 274 |
+
if state["step"] >= stop:
|
| 275 |
+
log(args.log, event="done", step=state["step"], stop=stop, best_val_loss=state["best_val_loss"],
|
| 276 |
+
final=str(ckpt / "final"))
|
| 277 |
+
return
|
| 278 |
+
model, tokenizer, limit = load_model(last or REPO, device)
|
| 279 |
+
val_cases = load_split("validation", min(args.val_cases, args.limit or args.val_cases))
|
| 280 |
+
started = time.perf_counter()
|
| 281 |
+
states, train_cut, epoch_tokens = prepare_states(train_cases, tokenizer, args.state_cap, limit)
|
| 282 |
+
val_states, val_cut, _ = prepare_states(val_cases, tokenizer, args.state_cap, limit)
|
| 283 |
+
states.update(val_states)
|
| 284 |
+
layers = len(model.backbone.layers) if state["train_layers"] < 0 else state["train_layers"]
|
| 285 |
+
params = set_trainable(model, layers)
|
| 286 |
+
head = {id(p) for n, p in model.named_parameters() if not n.startswith("backbone.")}
|
| 287 |
+
optimizer = torch.optim.AdamW([{"params": [p for p in params if id(p) not in head], "lr": args.lr},
|
| 288 |
+
{"params": [p for p in params if id(p) in head], "lr": args.head_lr}],
|
| 289 |
+
weight_decay=0.01)
|
| 290 |
+
if last:
|
| 291 |
+
optimizer.load_state_dict(torch.load(last / "optimizer.pt", map_location=device))
|
| 292 |
+
model.backbone.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
|
| 293 |
+
model.train()
|
| 294 |
+
warmup = max(1, min(args.warmup, total // 10))
|
| 295 |
+
log(args.log, event="start", resume_from=str(last) if last else None, step=state["step"], stop=stop,
|
| 296 |
+
steps_per_epoch=per_epoch, train_cases=len(train_cases), val_cases=len(val_cases), state_cap=args.state_cap,
|
| 297 |
+
train_truncated=train_cut, val_truncated=val_cut, unpadded_tokens_per_epoch=epoch_tokens,
|
| 298 |
+
train_layers=layers, trainable_params=sum(p.numel() for p in params), warmup=warmup,
|
| 299 |
+
prepare_s=round(time.perf_counter() - started, 1), device=str(device))
|
| 300 |
+
for epoch in range(state["epoch"], args.epochs):
|
| 301 |
+
order = list(range(len(train_cases)))
|
| 302 |
+
random.Random(args.seed + epoch).shuffle(order)
|
| 303 |
+
for b in range(state["step_in_epoch"], per_epoch):
|
| 304 |
+
tick = time.perf_counter()
|
| 305 |
+
batch = [train_cases[i] for i in order[b * args.batch_cases:(b + 1) * args.batch_cases]]
|
| 306 |
+
examples = examples_for(batch, states, tokenizer, limit)
|
| 307 |
+
scale = lr_scale(state["step"], stop, warmup)
|
| 308 |
+
for group, base in zip(optimizer.param_groups, (args.lr, args.head_lr)):
|
| 309 |
+
group["lr"] = base * scale
|
| 310 |
+
optimizer.zero_grad(set_to_none=True)
|
| 311 |
+
loss_sum, tokens = 0.0, 0
|
| 312 |
+
for group in microbatches(examples, args.micro_tokens):
|
| 313 |
+
loss = group_loss(model, group, tokenizer.pad_token_id, device) / len(examples)
|
| 314 |
+
loss.backward()
|
| 315 |
+
loss_sum, tokens = loss_sum + float(loss.detach()), tokens + padded_tokens(group)
|
| 316 |
+
grad_norm = torch.nn.utils.clip_grad_norm_(params, 1.0)
|
| 317 |
+
optimizer.step()
|
| 318 |
+
state.update(step=state["step"] + 1, step_in_epoch=b + 1)
|
| 319 |
+
seconds = time.perf_counter() - tick
|
| 320 |
+
if state["step"] % args.log_every == 0 or state["step"] in (1, stop):
|
| 321 |
+
log(args.log, event="step", epoch=epoch, step=state["step"], loss=round(loss_sum, 5),
|
| 322 |
+
lr=args.lr * scale, grad_norm=round(float(grad_norm), 4), padded_tokens=tokens,
|
| 323 |
+
tokens_per_s=round(tokens / seconds), step_s=round(seconds, 2))
|
| 324 |
+
if state["step"] >= stop:
|
| 325 |
+
break
|
| 326 |
+
if state["step"] % args.save_every == 0 and b + 1 < per_epoch:
|
| 327 |
+
save_bundle(model, ckpt / "last", state, optimizer)
|
| 328 |
+
log(args.log, event="saved", step=state["step"], checkpoint=str(ckpt / "last"))
|
| 329 |
+
if state["step_in_epoch"] >= per_epoch:
|
| 330 |
+
state.update(epoch=epoch + 1, step_in_epoch=0)
|
| 331 |
+
state["val_loss"] = validation_loss(model, val_cases, states, tokenizer, limit, device, args.micro_tokens)
|
| 332 |
+
state["history"].append({"epoch": state["epoch"], "step": state["step"], "val_loss": state["val_loss"]})
|
| 333 |
+
best = state["best_val_loss"] is None or state["val_loss"] < state["best_val_loss"]
|
| 334 |
+
if best:
|
| 335 |
+
state["best_val_loss"] = state["val_loss"]
|
| 336 |
+
save_bundle(model, ckpt / "final", state)
|
| 337 |
+
save_bundle(model, ckpt / "last", state, optimizer)
|
| 338 |
+
log(args.log, event="validated", epoch=state["epoch"], step=state["step"], val_loss=round(state["val_loss"], 5),
|
| 339 |
+
best=best, elapsed_s=round(time.perf_counter() - started, 1))
|
| 340 |
+
if state["step"] >= stop:
|
| 341 |
+
break
|
| 342 |
+
log(args.log, event="finished", step=state["step"], best_val_loss=state["best_val_loss"], final=str(ckpt / "final"))
|
| 343 |
+
|
| 344 |
+
|
| 345 |
+
def main() -> None:
|
| 346 |
+
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 347 |
+
commands = parser.add_subparsers(dest="command", required=True)
|
| 348 |
+
for name in ("zeroshot", "train", "predict"):
|
| 349 |
+
command = commands.add_parser(name)
|
| 350 |
+
command.add_argument("--limit", type=int, help="whole tasks covering about N cases")
|
| 351 |
+
command.add_argument("--device", default="cuda:0" if torch.cuda.is_available() else "cpu")
|
| 352 |
+
command.add_argument("--state-cap", type=int, default=STATE_CAP, help="state tokens kept per case")
|
| 353 |
+
if name != "train":
|
| 354 |
+
command.add_argument("--split", choices=["validation", "test"], required=True)
|
| 355 |
+
command.add_argument("--out", required=True)
|
| 356 |
+
if name != "zeroshot":
|
| 357 |
+
command.add_argument("--ckpt", required=True)
|
| 358 |
+
fit = commands.choices["train"]
|
| 359 |
+
fit.add_argument("--epochs", type=int, default=2)
|
| 360 |
+
fit.add_argument("--max-steps", type=int, help="stop after this many optimizer steps in total")
|
| 361 |
+
fit.add_argument("--log", help="also append the JSON progress lines here")
|
| 362 |
+
fit.add_argument("--batch-cases", type=int, default=4, help="cases (13 questions each) per optimizer step")
|
| 363 |
+
fit.add_argument("--micro-tokens", type=int, help="padded tokens per forward (default 32768 CUDA, 8192 CPU)")
|
| 364 |
+
fit.add_argument("--lr", type=float, default=2e-5, help="backbone learning rate")
|
| 365 |
+
fit.add_argument("--head-lr", type=float, default=2e-4, help="head learning rate (NanoJev used 10x backbone)")
|
| 366 |
+
fit.add_argument("--warmup", type=int, default=100)
|
| 367 |
+
fit.add_argument("--train-layers", type=int, help="last decoder layers to train (default all on CUDA, 2 on CPU)")
|
| 368 |
+
fit.add_argument("--val-cases", type=int, default=300)
|
| 369 |
+
fit.add_argument("--save-every", type=int, default=500)
|
| 370 |
+
fit.add_argument("--log-every", type=int, default=20)
|
| 371 |
+
fit.add_argument("--seed", type=int, default=0)
|
| 372 |
+
args = parser.parse_args()
|
| 373 |
+
if args.command == "zeroshot":
|
| 374 |
+
answer_split(REPO, args)
|
| 375 |
+
elif args.command == "predict":
|
| 376 |
+
answer_split(bundle_of(Path(args.ckpt)), args)
|
| 377 |
+
else:
|
| 378 |
+
args.micro_tokens = args.micro_tokens or (32768 if args.device.startswith("cuda") else 8192)
|
| 379 |
+
train(args)
|
| 380 |
+
|
| 381 |
+
|
| 382 |
+
if __name__ == "__main__":
|
| 383 |
+
main()
|