NextGenU/translator-french-shingazidja

NLLB French → shiNgazidja (fine-tuned)

This model is a fine-tuned version of facebook/nllb-200-distilled-600M for French → shiNgazidja (zdj) translation.

Note on language code: NLLB-200 does not include a native token for shiNgazidja, a Comorian language. During fine-tuning, the closest available NLLB language tag, swh_Latn (Swahili), was reused as a stand-in token to represent shiNgazidja. This is purely an internal tokenizer artifact — the model's actual output language is shiNgazidja (zdj), not Swahili. Keep this in mind if you inspect the code below: every reference to sw / swh_Latn refers to shiNgazidja.

The model also uses custom control tokens prepended to each input (target language, source language, domain, dialect) that were introduced during fine-tuning. You must reproduce the same prefix format at inference time, or translation quality will degrade significantly.

Model details

• Base model: facebook/nllb-200-distilled-600M
• Source language: French — fra_Latn
• Target language: shiNgazidja — internally tagged swh_Latn (proxy for zdj, no native NLLB code available)
• Direction: src2tgt (French → shiNgazidja)
• Fine-tuning framework: 🤗 Transformers

Input format

Every input sentence must be prefixed like this:

<to_sw> <src_fr> <dom_unknown> <dialect_default> your sentence here

• <to_sw> / <src_fr> — target/source short language codes used at training time (sw is the internal proxy tag for shiNgazidja, see note above)
• <dom_unknown> / <dialect_default> — the model was trained with optional domain/dialect tags (e.g. <dom_dictionary>, <dom_picture_book>); this README fixes both to their default values since no domain/dialect metadata is used at inference here

The build_prefix helper below reproduces this exactly as used during training.

Installation

pip install -q "transformers>=4.56,<4.57" sentencepiece sacremoses sacrebleu datasets pandas numpy tqdm protobuf

Inference

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer

MODEL_ID = "NextGenU/translator-french-shiNgazidja"  # replace with your repo id

SOURCE_LANG, TARGET_LANG = "fr", "sw"          # "sw" = internal proxy tag for shiNgazidja (zdj)
SOURCE_LID, TARGET_LID = "fra_Latn", "swh_Latn"  # swh_Latn = internal proxy for shiNgazidja
DIRECTION = "src2tgt"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


# --- Prefix builder (must match training) -------------------------------
# Domain and dialect are fixed to their default values here since this
# deployment does not use per-sentence domain/dialect metadata.

def build_prefix(direction=DIRECTION) -> str:
    src_short = SOURCE_LANG if direction == "src2tgt" else TARGET_LANG
    tgt_short = TARGET_LANG if direction == "src2tgt" else SOURCE_LANG
    return f"<to_{tgt_short}> <src_{src_short}> <dom_unknown> <dialect_default>"


# --- Load model -----------------------------------------------------------

tokenizer = NllbTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).to(device).eval()

# NLLB decoder starts on EOS
model.config.decoder_start_token_id = tokenizer.eos_token_id
if getattr(model, "generation_config", None) is not None:
    model.generation_config.decoder_start_token_id = tokenizer.eos_token_id


def ensure_token(token: str) -> int:
    tid = tokenizer.convert_tokens_to_ids(token)
    if tid == tokenizer.unk_token_id or tokenizer.convert_ids_to_tokens(tid) != token:
        raise ValueError(f"{token} is not a known token in the tokenizer")
    return int(tid)


# --- Single sentence translation ------------------------------------------

@torch.no_grad()
def translate(text, max_new_tokens=128, num_beams=4):
    prefix = build_prefix()
    prompt = f"{prefix} {text}"

    tokenizer.src_lang = SOURCE_LID
    enc = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=256).to(device)

    out = model.generate(
        **enc,
        forced_bos_token_id=ensure_token(TARGET_LID),
        decoder_start_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
        max_new_tokens=max_new_tokens,
        num_beams=num_beams,
        no_repeat_ngram_size=3,
        repetition_penalty=1.15,
        length_penalty=1.0,
    )
    return tokenizer.batch_decode(out, skip_special_tokens=True)[0]


# --- Batch translation ------------------------------------------------------

@torch.no_grad()
def translate_batch(texts, batch_size=16,
                     max_length=256, max_new_tokens=128, num_beams=4):
    prefix = build_prefix()
    forced_id = ensure_token(TARGET_LID)
    outputs = []

    for start in range(0, len(texts), batch_size):
        batch = [f"{prefix} {str(t)}" for t in texts[start:start + batch_size]]
        tokenizer.src_lang = SOURCE_LID
        enc = tokenizer(
            batch, return_tensors="pt", padding=True, truncation=True,
            max_length=max_length, return_token_type_ids=False,
        ).to(device)

        gen = model.generate(
            **enc,
            forced_bos_token_id=forced_id,
            decoder_start_token_id=tokenizer.eos_token_id,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.pad_token_id,
            max_new_tokens=max_new_tokens,
            num_beams=num_beams,
            no_repeat_ngram_size=3,
            repetition_penalty=1.15,
            length_penalty=1.0,
        )
        outputs.extend(tokenizer.batch_decode(gen, skip_special_tokens=True))

    return outputs


# --- Example ----------------------------------------------------------------

if __name__ == "__main__":
    print(translate("Demain matin nous irons là-bas."))
    print(translate_batch(["Je pense qu'il va pleuvoir.", "Les enfants n'ont pas encore mangé."]))

Notes

• Set MODEL_ID to the actual repository id once uploaded.
• Domain and dialect are fixed to <dom_unknown> / <dialect_default>.
• Translation quality depends on prefix consistency — omitting the prefix or using a different format than training will produce degraded output.
• Beam search (num_beams=4) and repetition_penalty=1.15 are the defaults used during evaluation; adjust for your speed/quality trade-off.
• All sw / swh_Latn references in this model and code are an internal proxy for shiNgazidja (zdj) — see the note above.

Limitations

• This model is fine-tuned for French → shiNgazidja only (src2tgt direction). A tgt2src (shiNgazidja → French) direction requires either a separately fine-tuned checkpoint or the tags/prefix swapped accordingly, if the base model supports it.
• Because shiNgazidja has no native NLLB language token, the model relies entirely on the fine-tuning data to associate the swh_Latn proxy token with shiNgazidja vocabulary and grammar — any residual bias toward actual Swahili is possible, especially on out-of-domain text.
• As with any NMT model, performance may vary across domains not well represented in the fine-tuning data.