Sion
Collection
Sion model family โข 4 items โข Updated
Sion 1.5 base
Sion 1.5 base is an experimental encoder-decoder foundation model for Korean, Japanese, and English, trained from scratch with the sion_translate pipeline. It is the first stage of the Sion 1.5 family and the starting point of the translation models:
| Stage | Model | Purpose |
|---|---|---|
| Foundation | alice-noa-chan/sion-1.5-base (this model) | Monolingual span reconstruction |
| Translation SFT | alice-noa-chan/sion_translate-1.5 | Six-direction ko/ja/en translation |
| Translation MRT | alice-noa-chan/sion_translate-1.5-mrt | Minimum-risk post-training of the SFT model |
All three are in the Sion collection.
This is not a translation model. It has never learned translation directions, and the translation runtime refuses to load it. It is published as a research artifact and as a starting point for fine-tuning. As a standalone fill-in model it is weaker than comparable public base models (see Evaluation).
Model details
| Architecture | Encoder-decoder Transformer, 18 encoder + 9 decoder layers, d_model 832, 8 heads (4 KV heads), d_ff 2304, RoPE, QK-norm, tied embeddings |
| Parameters | 273.5M |
| Tokenizer | SentencePiece, 48,000 pieces (sion.model, SHA-256 89303ab8โฆbc5d8) |
| Objective | Span-corruption denoising: 15% of tokens masked in spans of mean length 3; input <denoise_xx> corrupted text </s>, target is the full original text |
| Extra data | 5% structured reasoning samples |
| Language sampling | ฮฑ = 0.7, giving weights ko 65.0%, ja 27.3%, en 7.6% |
| Training | 3 epochs, batch 32/GPU, learning rate 3e-4, warm-up 2,000 steps; selected step 1,768,546 |
| Best validation loss | 1.265 (label-smoothed, 0.1) |
Training data
Monolingual text only, deduplicated, 8โ4,000 characters per document:
- Korean: Korean Wikipedia, NIKL corpora (newspaper, online posts, spoken dialogue, parliamentary records, non-published text), a community-text corpus, technical/UI text, and short stories.
- Japanese: Common Crawl (ABEJA), FineWeb2-Edu, Japanese Wikipedia, Aozora Bunko, e-Gov, Diet records, JParaCrawl (Japanese side), and open2ch.
- English: FineWeb-Edu (curated subject subsets).
- Reasoning: small Korean, Japanese, and English reasoning sets (5% of samples).
Users are responsible for reviewing the terms of the upstream corpora for their use case.
Files
| File | Notes |
|---|---|
model.pt |
FP32 native weights |
model_bf16.pt |
BF16 native weights |
sion.model |
SentencePiece tokenizer |
token_features.npz |
Per-token script features read by the model |
export_manifest.json |
Artifact hashes and lineage |
reconstruct.py |
Command-line span filling example |
A Transformers export exists in the training output but is not published: its remote code refuses to load this checkpoint because the foundation stage has no candidate-refinement release evidence.
Usage
python -m pip install "sion-translate @ git+https://github.com/gaon12/sion_translate.git@ccccf5742909efffb8571a9cabfd9b70977e1ec4"
hf download alice-noa-chan/sion-1.5-base --local-dir sion-1.5-base
python sion-1.5-base/reconstruct.py --model-dir sion-1.5-base --language ko --text "์ค๋์ ๋ ์จ๊ฐ <mask> ์ฐ์ฑ
ํ๊ธฐ ์ข๋ค."
# ์ค๋์ ๋ ์จ๊ฐ ์ข ์ฐ์ฑ
ํ๊ธฐ ์ข๋ค.
Mark each span to fill with <mask> and select the language with --language. The model
regenerates the whole text, not only the masked spans.
Evaluation
The model was evaluated on its own training objective with the training settings (15% density, mean span 3, beam 4 for generation). Two text sources were used, 50 samples per language each:
- FLORES-200 devtest: sentences sampled with seed 4242.
- Held-out training validation split: the exact split used during training (manifest
be5c672aโฆ), so it is in-domain for this model and unseen by the comparison model.
The comparison model is google/mt5-small (300M parameters, also span-corruption pre-trained), run with the same masking procedure on its own tokenizer. Hidden-token accuracy is the teacher-forced top-1 accuracy on masked tokens. ฮ chrF++ compares the reconstruction with the corrupted text that has the masked spans simply deleted; a positive value means the filled text is closer to the original.
| Data | Model | Hidden-token accuracy (ko / ja / en) | ฮ chrF++ (ko / ja / en) | Outputs with leaked Korean (ja / en) |
|---|---|---|---|---|
| FLORES-200 | Sion 1.5 base | 20.9% / 17.7% / 20.1% | โ12.0 / โ17.5 / โ7.5 | 42/50 / 23/50 |
| FLORES-200 | mT5-small | 39.8% / 34.1% / 43.9% | +4.4 / +2.9 / +4.2 | 0/50 / 0/50 |
| Held-out (in-domain) | Sion 1.5 base | 21.2% / 22.0% / 25.6% | โ1.5 / โ26.2 / โ9.8 | 18/50 / 40/50 |
| Held-out (in-domain) | mT5-small | 39.7% / 29.1% / 48.0% | +3.6 / +2.0 / +3.8 | 0/50 / 0/50 |
Tokenizers differ (48k vs 250k pieces), so hidden-token accuracy is indicative only; ฮ chrF++ compares surface text and does not depend on the tokenizer.
Why the validation loss looked healthy. On the held-out split the teacher-forced loss is 1.16 nats per token, consistent with the recorded validation loss. About 85% of target tokens are visible in the input and are copied almost for free, while the masked 15% carry a loss of 5โ7 nats per token. The average therefore hides weak span prediction.
Downstream. The SFT model fine-tuned from this checkpoint reaches parity with Papago on KoreanโJapanese FLORES-200 samples (see the translation model card). Whether this base is a better or worse fine-tuning starting point than other public models has not been measured.
Limitations
- Weak masked-span prediction compared with mT5-small, both in and out of domain.
- Frequent Korean fragments in Japanese and English output, consistent with the Korean-heavy sampling weights.
- Reconstructions can alter visible text, not only the masked spans.
- Not instruction-tuned, not a chat model, and not a translation model.
AI-Hub dataset attribution for 1.5
The following AI-Hub datasets supplied Korean-Japanese and Korean-English text for training the shared tokenizer used by this foundation model. We acknowledge AI-Hub (aihub.or.kr) and the dataset creators. This table credits the individually identified AI-Hub sources; it is not a complete inventory of all training sources. The language pairs below describe the identified source files, not every language available in the upstream datasets.
Additional Korean-English tokenizer text came from a combined source containing 14 AI-Hub corpora alongside other sources. The available combined-source records retain its aggregate provenance but do not identify each AI-Hub dataset ID; the table above lists the individually verified datasets.
This attribution concerns tokenizer training. Foundation parameter training used the monolingual and reasoning sources described in Training data.
AI-Hub's official FAQ permits commercial use of learned models with dataset-name and AI-Hub attribution; it restricts redistribution of original or merely edited source data.
License
MIT for the original code and this model release. This does not grant rights to the upstream training corpora or third-party software.