SmolSynformer: SmolLM2 as Syntax-aware transformer

SmolSynformer is trained on various instructions including GEC, paraphrase identification and universal dependency generation. Code and math are not included. This model is overfitted on in-context learning and does sometimes generate follow up questions and answers.

Inference with transformer

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig, pipeline

test_model = "Bachstelze/smolSynformer"
model = AutoModelForCausalLM.from_pretrained(test_model)
tokenizer = AutoTokenizer.from_pretrained(test_model)
config = AutoConfig.from_pretrained(test_model)

prompt_pipeline = pipeline("text-generation", model=test_model, tokenizer=tokenizer, max_new_tokens=250)
print(prompt_pipeline("Why is syntax relevant for language modeling and instruction following?\n"))

Example answer:

Syntax is relevant for language modeling and instruction following because it provides a structured and organized way to represent and analyze language. It allows for the creation of rules and patterns that govern how language is used, which can be used to train models to recognize and generate language. Additionally, syntax can be used to identify and classify different types of language, such as grammatical or idiomatic language.