palashsharma048/fourkites-intent-classifier

Logistics intent classifier (fine-tuned multilingual MiniLM-L12)

Routes a short user message from a supply-chain / logistics chat agent to one of 12 intents, and to unknown when the message does not look like any intent it was trained on.

Labels

• 0: ai_agent_performance
• 1: appointment_manager
• 2: chitchat
• 3: customer_support
• 4: document_processing
• 5: knowledge_base
• 6: orders
• 7: other
• 8: shipment_information.analytics
• 9: shipment_information.disruptions
• 10: shipment_information.realtime_query
• 11: yard_management

Training data

• 500 synthetic, labelled chat messages; 50 locked away as a final lockbox, 450 used for development.
• 12 intents (27–50 rows each); ~13% non-English (es, fr, de, zh, nl, it, ar); informal chat (typos, slang, reference IDs such as LD-10234, PO-76540).
• Group-aware stratified split (near-duplicate messages kept together; stratified by intent × English/non-English): 322 train / 65 validation / 63 test.

Training

• Base model sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (118M parameters, 50+ languages) with a 12-way classification head.
• Word-embedding layer frozen (protects cross-lingual alignment; 21M trainable parameters).
• AdamW, learning rate 4e-5, batch size 16, weight decay 0.01, 10% linear warm-up then linear decay.
• Class-weighted cross-entropy with label smoothing 0.1; max sequence length 48 tokens.
• Early stopping on validation macro-F1 (best epoch 14); seed 42.
• Regularisation (dropout 0.2, frozen lower layers) and machine-translation augmentation were tested with 5-fold cross-validation and did not improve validation macro-F1, so they are not used.

Metrics (primary metric: macro-F1)

Data	macro-F1	accuracy
5-fold cross-validation, training folds	1.000	1.000
5-fold cross-validation, validation folds (387 rows)	0.859 ± 0.045	0.876 ± 0.043
Test set (63 rows)	0.942	0.937
Lockbox (50 rows)	0.910	0.920
Challenge set (72 longer / noisier / multilingual messages)	0.860	0.861

For comparison (cross-validated macro-F1): TF-IDF + Linear SVM 0.752, frozen multilingual embeddings + logistic regression 0.860.

Non-English: 0.875 accuracy out-of-fold (English: 0.876), 9/9 on test, 6/7 on the lockbox.

Unknown detection (intents yard_management + document_processing removed from training; Ensemble score chosen beforehand on a different held-out pair; threshold keeps 95% of business-intent validation messages): AUROC 0.838 · retention 0.956 · rejection recall 0.444. At 90% retention the rejection recall is 0.736.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "palashsharma048/fourkites-intent-classifier"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

logits = model(**tok(["where is LD-12345 right now?"], return_tensors="pt")).logits
print(model.config.id2label[logits.argmax(-1).item()])   # shipment_information.realtime_query

The unknown rule

config.json holds unknown_score (Ensemble), unknown_threshold and fallback_intents (other, chitchat). The Ensemble score is the mean percentile (against the stored validation reference scores) of: the max softmax probability, the mean cosine similarity to the 5 nearest training messages in this model's mean-pooled, L2-normalised last-layer embeddings, and the same kNN similarity using the frozen paraphrase-multilingual-MiniLM-L12-v2 sentence encoder. All required statistics are in router_stats.npz. If the score is below the threshold and the predicted intent is not a fallback intent, route the message to unknown.

Intended use and limitations

• First routing step of a logistics chat agent; not intended for other domains.
• Trained on a small synthetic dataset; real traffic will differ. Re-validate on real, labelled traffic before production use.
• The model fits its training data perfectly (train macro-F1 1.00 vs 0.86 cross-validated). The gap could not be reduced by regularisation or augmentation; the remaining errors are messages that sit between two intents.
• Known confusion zones: customer_support tickets that mention another intent's object (APT-…, DOC-…), and yard_management vs shipment_information.realtime_query.
• Non-English results rest on few real examples per language.
• Unknown detection catches only part of genuinely new intents; monitor the unknown rate and the confidence distribution in production to detect drift.