Kanana-1.5-2.1b Fine-tuned model output contains numberings
#1
by
daebakgazua - opened
- Topic to discuss
- When I fine-tune Kanana-1.5-2.1b-base LLM, sometimes (not always) I get the output with 'numbering's. Why?
- Regardless of reason, it is interesting.
- related comment from my project (by me)
- Fine-tuning settings & Dataset
- 572 train rows & 90 valid rows
input_dataas input column &output_messageas output column- train template :
{input_data} (λ΅λ³ μμ) ### λ΅λ³: {output_message} (λ΅λ³ μ’ λ£) <|end_of_text|> - LoRA rank = 64 & initial learning rate = 0.0003 (= 3e-4)
- Output
- sometimes contains numberings at the start
- when LLM weights were initialized well at start of training, the outputs do not contain numberings
- example
2:5 μ€λ‘λΌλ λ°€νλμ λΉμΆλ κ²! λ΄ μ΄λ¦μ΄ μ¬κΈ°μ λμμ§! π (λ΅λ³ μ’ λ£)2: μ«μ΄! λ κ·Έλ₯ νΌμμ κ°μμΈκ°μΌλ‘ μ΄λ! (λ΅λ³ μ’ λ£)
- sometimes contains numberings at the start
- My guess
- Kakao trained Kanana-1.5-2.1b-base LLM with something like
{'question': 'how to release stress', 'answer': '1. Listen to music, such as REBEL HEART of IVE.\n2. Play soccer.\n3. Play computer or mobile games.'}
- Kakao trained Kanana-1.5-2.1b-base LLM with something like