Instructions to use AppleMind-AI/AppleMind-1.0-Mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AppleMind-AI/AppleMind-1.0-Mini with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AppleMind-AI/AppleMind-1.0-Mini")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("AppleMind-AI/AppleMind-1.0-Mini")
model = AutoModelForCausalLM.from_pretrained("AppleMind-AI/AppleMind-1.0-Mini", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AppleMind-AI/AppleMind-1.0-Mini with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AppleMind-AI/AppleMind-1.0-Mini"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AppleMind-AI/AppleMind-1.0-Mini",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AppleMind-AI/AppleMind-1.0-Mini

SGLang

How to use AppleMind-AI/AppleMind-1.0-Mini with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AppleMind-AI/AppleMind-1.0-Mini" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AppleMind-AI/AppleMind-1.0-Mini",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AppleMind-AI/AppleMind-1.0-Mini" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AppleMind-AI/AppleMind-1.0-Mini",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AppleMind-AI/AppleMind-1.0-Mini with Docker Model Runner:
```
docker model run hf.co/AppleMind-AI/AppleMind-1.0-Mini
```

GGUFGuy commited on Aug 13

Commit

02508d4

verified ·

1 Parent(s): edd1e6e

Create LOGS.md

Browse files

Files changed (1) hide show

LOGS.md +993 -0

LOGS.md ADDED Viewed

	@@ -0,0 +1,993 @@

+ Requirement already satisfied: transformers in /usr/local/lib/python3.12/dist-packages (5.13.1)
+Collecting transformers
+  Downloading transformers-5.15.0-py3-none-any.whl.metadata (32 kB)
+Requirement already satisfied: datasets in /usr/local/lib/python3.12/dist-packages (4.0.0)
+Collecting datasets
+  Downloading datasets-5.0.1-py3-none-any.whl.metadata (23 kB)
+Requirement already satisfied: accelerate in /usr/local/lib/python3.12/dist-packages (1.14.0)
+Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.12/dist-packages (1.23.0)
+Collecting huggingface_hub
+  Downloading huggingface_hub-1.27.0-py3-none-any.whl.metadata (16 kB)
+Requirement already satisfied: hf_xet in /usr/local/lib/python3.12/dist-packages (1.5.1)
+Collecting hf_xet
+  Downloading hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (4.9 kB)
+Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.12/dist-packages (from transformers) (2.0.2)
+Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (26.2)
+Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from transformers) (6.0.3)
+Requirement already satisfied: regex>=2025.10.22 in /usr/local/lib/python3.12/dist-packages (from transformers) (2025.11.3)
+Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.22.2)
+Requirement already satisfied: typer in /usr/local/lib/python3.12/dist-packages (from transformers) (0.26.8)
+Requirement already satisfied: safetensors>=0.8.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.8.0)
+Requirement already satisfied: tqdm>=4.60 in /usr/local/lib/python3.12/dist-packages (from transformers) (4.67.3)
+Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from datasets) (3.29.7)
+Collecting pyarrow>=21.0.0 (from datasets)
+  Downloading pyarrow-25.0.1-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
+Requirement already satisfied: dill<0.4.2,>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.3.8)
+Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from datasets) (2.2.2)
+Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.12/dist-packages (from datasets) (2.32.4)
+Requirement already satisfied: httpx<1.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.28.1)
+Requirement already satisfied: xxhash in /usr/local/lib/python3.12/dist-packages (from datasets) (3.8.1)
+Requirement already satisfied: multiprocess<0.70.20 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.70.16)
+Requirement already satisfied: fsspec<=2026.6.0,>=2023.1.0 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (2025.3.0)
+Requirement already satisfied: psutil in /usr/local/lib/python3.12/dist-packages (from accelerate) (5.9.5)
+Requirement already satisfied: torch>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from accelerate) (2.11.0+cu128)
+Requirement already satisfied: click<9.0.0,>=8.4.2 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (8.4.2)
+Requirement already satisfied: typing-extensions>=4.1.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (4.16.0)
+Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (3.14.1)
+Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (4.14.2)
+Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (2026.6.17)
+Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (1.0.9)
+Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (3.18)
+Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1.0.0->datasets) (0.16.0)
+Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests>=2.32.2->datasets) (3.4.9)
+Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests>=2.32.2->datasets) (2.5.0)
+Requirement already satisfied: setuptools<82 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (75.2.0)
+Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (1.14.0)
+Requirement already satisfied: networkx>=2.5.1 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.6.1)
+Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.1.6)
+Requirement already satisfied: cuda-toolkit==12.8.1 in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.1)
+Requirement already satisfied: cuda-bindings<13,>=12.9.4 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (12.9.7)
+Requirement already satisfied: nvidia-cudnn-cu12==9.19.0.56 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (9.19.0.56)
+Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (0.7.1)
+Requirement already satisfied: nvidia-nccl-cu12==2.28.9 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (2.28.9)
+Requirement already satisfied: nvidia-nvshmem-cu12==3.4.5 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.4.5)
+Requirement already satisfied: triton==3.6.0 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.6.0)
+Requirement already satisfied: nvidia-cublas-cu12==12.8.4.1.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.4.1)
+Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.90)
+Requirement already satisfied: nvidia-cufft-cu12==11.3.3.83.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (11.3.3.83)
+Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (1.13.1.3)
+Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.90)
+Requirement already satisfied: nvidia-curand-cu12==10.3.9.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (10.3.9.90)
+Requirement already satisfied: nvidia-cusolver-cu12==11.7.3.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (11.7.3.90)
+Requirement already satisfied: nvidia-cusparse-cu12==12.5.8.93.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.5.8.93)
+Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.93.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.93)
+Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.93.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.93)
+Requirement already satisfied: nvidia-nvtx-cu12==12.8.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.90)
+Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2.9.0.post0)
+Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)
+Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2026.3)
+Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer->transformers) (1.5.4)
+Requirement already satisfied: rich>=13.8.0 in /usr/local/lib/python3.12/dist-packages (from typer->transformers) (13.9.4)
+Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from typer->transformers) (0.0.4)
+Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (2.7.1)
+Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.4.0)
+Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (26.1.0)
+Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.8.0)
+Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (6.7.1)
+Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (0.5.2)
+Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.24.2)
+Requirement already satisfied: cuda-pathfinder~=1.1 in /usr/local/lib/python3.12/dist-packages (from cuda-bindings<13,>=12.9.4->torch>=2.0.0->accelerate) (1.5.6)
+Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)
+Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer->transformers) (4.2.0)
+Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer->transformers) (2.20.0)
+Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch>=2.0.0->accelerate) (1.3.0)
+Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->torch>=2.0.0->accelerate) (3.0.3)
+Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers) (0.1.2)
+Downloading transformers-5.15.0-py3-none-any.whl (11.7 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 133.0 MB/s eta 0:00:00
+Downloading datasets-5.0.1-py3-none-any.whl (559 kB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 559.1/559.1 kB 48.0 MB/s eta 0:00:00
+Downloading huggingface_hub-1.27.0-py3-none-any.whl (784 kB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 784.9/784.9 kB 61.8 MB/s eta 0:00:00
+Downloading hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.5 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 132.9 MB/s eta 0:00:00
+Downloading pyarrow-25.0.1-cp312-cp312-manylinux_2_28_x86_64.whl (50.1 MB)
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.1/50.1 MB 49.6 MB/s eta 0:00:00
+Installing collected packages: pyarrow, hf_xet, huggingface_hub, datasets, transformers
+  Attempting uninstall: pyarrow
+    Found existing installation: pyarrow 18.1.0
+    Uninstalling pyarrow-18.1.0:
+      Successfully uninstalled pyarrow-18.1.0
+  Attempting uninstall: hf_xet
+    Found existing installation: hf-xet 1.5.1
+    Uninstalling hf-xet-1.5.1:
+      Successfully uninstalled hf-xet-1.5.1
+  Attempting uninstall: huggingface_hub
+    Found existing installation: huggingface_hub 1.23.0
+    Uninstalling huggingface_hub-1.23.0:
+      Successfully uninstalled huggingface_hub-1.23.0
+  Attempting uninstall: datasets
+    Found existing installation: datasets 4.0.0
+    Uninstalling datasets-4.0.0:
+      Successfully uninstalled datasets-4.0.0
+  Attempting uninstall: transformers
+    Found existing installation: transformers 5.13.1
+    Uninstalling transformers-5.13.1:
+      Successfully uninstalled transformers-5.13.1
+Successfully installed datasets-5.0.1 hf_xet-1.6.0 huggingface_hub-1.27.0 pyarrow-25.0.1 transformers-5.15.0
+To log in, open this URL and enter the code:https://hf.co/oauth/device1JHU-54EFWaiting for authorization...Login successful. Logged in as GGUFGuy (token: oauth-GGUFGuy).This token will be refreshed automatically when it expires.
+======================================================================
+AppleMind 1.0 Mini
+Colab L4 High-RAM
+300M TOKEN PRETRAINING
+======================================================================
+Device: cuda
+GPU: NVIDIA L4
+VRAM: 22.03 GB
+CUDA: 12.8
+Total training tokens: 300,000,000
+Target ratio: ~300:1
+======================================================================
+DATASET MIXTURE
+======================================================================
+FineWeb-Edu sample-10BT : 100,000,000
+FineWeb-HQ              : 100,000,000
+SmolLM-Corpus           : 100,000,000
+TOTAL                   : 300,000,000
+TinyStories: REMOVED
+WikiText: REMOVED
+Local datasets: REMOVED
+Dataset cycling: DISABLED
+======================================================================
+LOADING GPT-2 TOKENIZER
+======================================================================
+tokenizer_config.json: 100% 26.0/26.0 [00:00<00:00, 3.04kB/s]vocab.json: 100% 1.04M/1.04M [00:00<00:00, 15.7MB/s]merges.txt: 100% 456k/456k [00:00<00:00, 5.83MB/s]tokenizer.json: 100% 1.36M/1.36M [00:00<00:00, 28.9MB/s]
+Vocabulary: 50,260
+Special tokens added: 3
+======================================================================
+CREATING APPLEMIND
+======================================================================
+Parameters: 1,020,480
+Trainable: 1,020,480
+Target tokens: 300,000,000
+Target token/parameter ratio: 293.98:1
+======================================================================
+SETTING UP HUGGING FACE
+======================================================================
+Logged in as:
+GGUFGuy
+Checkpoint repository ready:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints
+Final model repository ready:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini
+Precision: BF16
+======================================================================
+TRAINING CONFIGURATION
+======================================================================
+Parameters: 1,020,480
+Training tokens: 300,000,000
+Tokens/parameter: 293.98:1
+Micro batch size: 64
+Gradient accumulation: 8
+Effective batch: 512
+Context length: 256
+Tokens/micro batch: 16,384
+Tokens/optimizer step: 131,072
+Optimizer steps: 2,289
+Warmup steps: 114
+FineWeb-Edu: 100M
+FineWeb-HQ:  100M
+SmolLM:      100M
+Checkpoint repository:
+AppleMind-AI/AppleMind-1.0-Mini-Checkpoints
+Final repository:
+AppleMind-AI/AppleMind-1.0-Mini
+======================================================================
+STARTING 300M-TOKEN PRETRAINING
+======================================================================
+Opening FineWeb-Edu sample-10BT...
+README.md: 100% 26.4k/26.4k [00:00<00:00, 2.88MB/s]Resolving data files: 100% 2410/2410 [00:00<00:00, 23346.46it/s]
+======================================================================
+STREAMING FineWeb-Edu
+======================================================================
+Target tokens: 100,000,000
+Progress state: 0
+[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
+FineWeb-Edu: 1,000,131 / 100,000,000 tokens
+FineWeb-Edu: 2,001,441 / 100,000,000 tokens
+FineWeb-Edu: 3,001,677 / 100,000,000 tokens
+FineWeb-Edu: 4,003,530 / 100,000,000 tokens
+FineWeb-Edu: 5,003,864 / 100,000,000 tokens
+FineWeb-Edu: 6,014,497 / 100,000,000 tokens
+FineWeb-Edu: 7,014,690 / 100,000,000 tokens
+FineWeb-Edu: 8,014,892 / 100,000,000 tokens
+FineWeb-Edu: 9,015,759 / 100,000,000 tokens
+FineWeb-Edu: 10,016,701 / 100,000,000 tokens
+FineWeb-Edu: 11,042,613 / 100,000,000 tokens
+FineWeb-Edu: 12,043,456 / 100,000,000 tokens
+FineWeb-Edu: 13,044,297 / 100,000,000 tokens
+Step 100/2,289 | Loss 10.8201 | LR 2.632e-05 | 100,991 tok/s | 13,107,200 tokens | 4.37% | 0.04h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-100...
+Writing model shards: 100% 1/1 [00:00<00:00, 19.95it/s]
+======================================================================
+UPLOADING CHECKPOINT 100
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-100
+FineWeb-Edu: 14,051,468 / 100,000,000 tokens
+FineWeb-Edu: 15,057,858 / 100,000,000 tokens
+FineWeb-Edu: 16,058,350 / 100,000,000 tokens
+FineWeb-Edu: 17,058,910 / 100,000,000 tokens
+FineWeb-Edu: 18,059,120 / 100,000,000 tokens
+FineWeb-Edu: 19,059,214 / 100,000,000 tokens
+FineWeb-Edu: 20,060,034 / 100,000,000 tokens
+FineWeb-Edu: 21,061,034 / 100,000,000 tokens
+FineWeb-Edu: 22,061,243 / 100,000,000 tokens
+FineWeb-Edu: 23,061,857 / 100,000,000 tokens
+FineWeb-Edu: 24,068,376 / 100,000,000 tokens
+FineWeb-Edu: 25,068,451 / 100,000,000 tokens
+FineWeb-Edu: 26,069,133 / 100,000,000 tokens
+Step 200/2,289 | Loss 10.7533 | LR 2.990e-05 | 103,793 tok/s | 26,214,400 tokens | 8.74% | 0.07h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-200...
+Writing model shards: 100% 1/1 [00:00<00:00, 11.00it/s]
+======================================================================
+UPLOADING CHECKPOINT 200
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-200
+FineWeb-Edu: 27,075,071 / 100,000,000 tokens
+FineWeb-Edu: 28,080,714 / 100,000,000 tokens
+FineWeb-Edu: 29,080,972 / 100,000,000 tokens
+FineWeb-Edu: 30,085,136 / 100,000,000 tokens
+FineWeb-Edu: 31,085,689 / 100,000,000 tokens
+FineWeb-Edu: 32,086,415 / 100,000,000 tokens
+FineWeb-Edu: 33,095,884 / 100,000,000 tokens
+FineWeb-Edu: 34,096,837 / 100,000,000 tokens
+FineWeb-Edu: 35,097,143 / 100,000,000 tokens
+FineWeb-Edu: 36,097,343 / 100,000,000 tokens
+FineWeb-Edu: 37,104,064 / 100,000,000 tokens
+FineWeb-Edu: 38,104,207 / 100,000,000 tokens
+FineWeb-Edu: 39,105,073 / 100,000,000 tokens
+Step 300/2,289 | Loss 10.6609 | LR 2.952e-05 | 104,146 tok/s | 39,321,600 tokens | 13.11% | 0.11h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-300...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
+======================================================================
+UPLOADING CHECKPOINT 300
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-300
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-100
+FineWeb-Edu: 40,107,907 / 100,000,000 tokens
+FineWeb-Edu: 41,108,319 / 100,000,000 tokens
+FineWeb-Edu: 42,108,687 / 100,000,000 tokens
+FineWeb-Edu: 43,108,744 / 100,000,000 tokens
+FineWeb-Edu: 44,109,584 / 100,000,000 tokens
+FineWeb-Edu: 45,109,641 / 100,000,000 tokens
+FineWeb-Edu: 46,110,333 / 100,000,000 tokens
+FineWeb-Edu: 47,110,722 / 100,000,000 tokens
+FineWeb-Edu: 48,111,125 / 100,000,000 tokens
+FineWeb-Edu: 49,111,530 / 100,000,000 tokens
+FineWeb-Edu: 50,113,218 / 100,000,000 tokens
+FineWeb-Edu: 51,113,865 / 100,000,000 tokens
+FineWeb-Edu: 52,114,128 / 100,000,000 tokens
+Step 400/2,289 | Loss 10.5818 | LR 2.887e-05 | 104,108 tok/s | 52,428,800 tokens | 17.48% | 0.14h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-400...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.91it/s]
+======================================================================
+UPLOADING CHECKPOINT 400
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-400
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-200
+FineWeb-Edu: 53,114,351 / 100,000,000 tokens
+FineWeb-Edu: 54,115,268 / 100,000,000 tokens
+FineWeb-Edu: 55,115,867 / 100,000,000 tokens
+FineWeb-Edu: 56,118,747 / 100,000,000 tokens
+FineWeb-Edu: 57,118,792 / 100,000,000 tokens
+FineWeb-Edu: 58,129,696 / 100,000,000 tokens
+FineWeb-Edu: 59,129,772 / 100,000,000 tokens
+FineWeb-Edu: 60,130,463 / 100,000,000 tokens
+FineWeb-Edu: 61,130,846 / 100,000,000 tokens
+FineWeb-Edu: 62,131,974 / 100,000,000 tokens
+FineWeb-Edu: 63,145,958 / 100,000,000 tokens
+FineWeb-Edu: 64,146,051 / 100,000,000 tokens
+FineWeb-Edu: 65,146,257 / 100,000,000 tokens
+Step 500/2,289 | Loss 10.5081 | LR 2.797e-05 | 103,733 tok/s | 65,536,000 tokens | 21.85% | 0.18h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-500...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.95it/s]
+======================================================================
+UPLOADING CHECKPOINT 500
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-500
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-300
+FineWeb-Edu: 66,147,176 / 100,000,000 tokens
+FineWeb-Edu: 67,153,903 / 100,000,000 tokens
+FineWeb-Edu: 68,163,652 / 100,000,000 tokens
+FineWeb-Edu: 69,164,783 / 100,000,000 tokens
+FineWeb-Edu: 70,166,355 / 100,000,000 tokens
+FineWeb-Edu: 71,166,545 / 100,000,000 tokens
+FineWeb-Edu: 72,167,733 / 100,000,000 tokens
+FineWeb-Edu: 73,168,881 / 100,000,000 tokens
+FineWeb-Edu: 74,175,780 / 100,000,000 tokens
+FineWeb-Edu: 75,176,435 / 100,000,000 tokens
+FineWeb-Edu: 76,177,413 / 100,000,000 tokens
+FineWeb-Edu: 77,177,431 / 100,000,000 tokens
+FineWeb-Edu: 78,179,729 / 100,000,000 tokens
+Step 600/2,289 | Loss 10.4357 | LR 2.682e-05 | 104,154 tok/s | 78,643,200 tokens | 26.21% | 0.21h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-600...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.96it/s]
+======================================================================
+UPLOADING CHECKPOINT 600
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-600
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-400
+FineWeb-Edu: 79,187,841 / 100,000,000 tokens
+FineWeb-Edu: 80,187,924 / 100,000,000 tokens
+FineWeb-Edu: 81,188,888 / 100,000,000 tokens
+FineWeb-Edu: 82,189,255 / 100,000,000 tokens
+FineWeb-Edu: 83,189,443 / 100,000,000 tokens
+FineWeb-Edu: 84,192,547 / 100,000,000 tokens
+FineWeb-Edu: 85,192,626 / 100,000,000 tokens
+FineWeb-Edu: 86,193,916 / 100,000,000 tokens
+FineWeb-Edu: 87,194,129 / 100,000,000 tokens
+FineWeb-Edu: 88,196,386 / 100,000,000 tokens
+FineWeb-Edu: 89,203,193 / 100,000,000 tokens
+FineWeb-Edu: 90,203,499 / 100,000,000 tokens
+FineWeb-Edu: 91,204,016 / 100,000,000 tokens
+Step 700/2,289 | Loss 10.3676 | LR 2.546e-05 | 104,272 tok/s | 91,750,400 tokens | 30.58% | 0.25h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-700...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.96it/s]
+======================================================================
+UPLOADING CHECKPOINT 700
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-700
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-500
+FineWeb-Edu: 92,205,445 / 100,000,000 tokens
+FineWeb-Edu: 93,206,406 / 100,000,000 tokens
+FineWeb-Edu: 94,215,721 / 100,000,000 tokens
+FineWeb-Edu: 95,216,609 / 100,000,000 tokens
+FineWeb-Edu: 96,218,844 / 100,000,000 tokens
+FineWeb-Edu: 97,219,420 / 100,000,000 tokens
+FineWeb-Edu: 98,219,961 / 100,000,000 tokens
+FineWeb-Edu: 99,220,378 / 100,000,000 tokens
+FineWeb-Edu complete: 100,000,000 tokens
+Opening FineWeb-HQ...
+README.md: 100% 4.65k/4.65k [00:00<00:00, 385kB/s]Resolving data files: 100% 9246/9246 [00:00<00:00, 25238.04it/s]
+======================================================================
+STREAMING FineWeb-HQ
+======================================================================
+Target tokens: 100,000,000
+Progress state: 0
+FineWeb-HQ: 1,000,237 / 100,000,000 tokens
+FineWeb-HQ: 2,000,353 / 100,000,000 tokens
+FineWeb-HQ: 3,000,648 / 100,000,000 tokens
+FineWeb-HQ: 4,001,578 / 100,000,000 tokens
+Step 800/2,289 | Loss 10.3038 | LR 2.391e-05 | 98,189 tok/s | 104,857,600 tokens | 34.95% | 0.28h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-800...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.89it/s]
+======================================================================
+UPLOADING CHECKPOINT 800
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-800
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-600
+FineWeb-HQ: 5,003,165 / 100,000,000 tokens
+FineWeb-HQ: 6,003,192 / 100,000,000 tokens
+FineWeb-HQ: 7,003,245 / 100,000,000 tokens
+FineWeb-HQ: 8,005,095 / 100,000,000 tokens
+FineWeb-HQ: 9,005,500 / 100,000,000 tokens
+FineWeb-HQ: 10,005,995 / 100,000,000 tokens
+FineWeb-HQ: 11,006,275 / 100,000,000 tokens
+FineWeb-HQ: 12,006,703 / 100,000,000 tokens
+FineWeb-HQ: 13,008,348 / 100,000,000 tokens
+FineWeb-HQ: 14,039,910 / 100,000,000 tokens
+FineWeb-HQ: 15,040,042 / 100,000,000 tokens
+FineWeb-HQ: 16,040,179 / 100,000,000 tokens
+FineWeb-HQ: 17,040,530 / 100,000,000 tokens
+Step 900/2,289 | Loss 10.2507 | LR 2.221e-05 | 103,602 tok/s | 117,964,800 tokens | 39.32% | 0.32h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-900...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
+======================================================================
+UPLOADING CHECKPOINT 900
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-900
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-700
+FineWeb-HQ: 18,045,133 / 100,000,000 tokens
+FineWeb-HQ: 19,053,155 / 100,000,000 tokens
+FineWeb-HQ: 20,053,155 / 100,000,000 tokens
+FineWeb-HQ: 21,053,631 / 100,000,000 tokens
+FineWeb-HQ: 22,061,664 / 100,000,000 tokens
+FineWeb-HQ: 23,061,940 / 100,000,000 tokens
+FineWeb-HQ: 24,062,978 / 100,000,000 tokens
+FineWeb-HQ: 25,063,229 / 100,000,000 tokens
+FineWeb-HQ: 26,073,081 / 100,000,000 tokens
+FineWeb-HQ: 27,073,220 / 100,000,000 tokens
+FineWeb-HQ: 28,073,681 / 100,000,000 tokens
+FineWeb-HQ: 29,073,966 / 100,000,000 tokens
+FineWeb-HQ: 30,074,272 / 100,000,000 tokens
+Step 1,000/2,289 | Loss 10.1952 | LR 2.039e-05 | 103,234 tok/s | 131,072,000 tokens | 43.69% | 0.35h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1000...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.99it/s]
+======================================================================
+UPLOADING CHECKPOINT 1000
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1000
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-800
+FineWeb-HQ: 31,075,196 / 100,000,000 tokens
+FineWeb-HQ: 32,075,679 / 100,000,000 tokens
+FineWeb-HQ: 33,077,041 / 100,000,000 tokens
+FineWeb-HQ: 34,077,283 / 100,000,000 tokens
+FineWeb-HQ: 35,077,442 / 100,000,000 tokens
+FineWeb-HQ: 36,078,144 / 100,000,000 tokens
+FineWeb-HQ: 37,078,751 / 100,000,000 tokens
+FineWeb-HQ: 38,080,143 / 100,000,000 tokens
+FineWeb-HQ: 39,082,841 / 100,000,000 tokens
+FineWeb-HQ: 40,083,517 / 100,000,000 tokens
+FineWeb-HQ: 41,084,347 / 100,000,000 tokens
+FineWeb-HQ: 42,087,132 / 100,000,000 tokens
+FineWeb-HQ: 43,087,302 / 100,000,000 tokens
+FineWeb-HQ: 44,087,920 / 100,000,000 tokens
+Step 1,100/2,289 | Loss 10.1467 | LR 1.849e-05 | 103,958 tok/s | 144,179,200 tokens | 48.06% | 0.39h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1100...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.97it/s]
+======================================================================
+UPLOADING CHECKPOINT 1100
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1100
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-900
+FineWeb-HQ: 45,091,588 / 100,000,000 tokens
+FineWeb-HQ: 46,091,666 / 100,000,000 tokens
+FineWeb-HQ: 47,091,765 / 100,000,000 tokens
+FineWeb-HQ: 48,093,667 / 100,000,000 tokens
+FineWeb-HQ: 49,095,400 / 100,000,000 tokens
+FineWeb-HQ: 50,100,158 / 100,000,000 tokens
+FineWeb-HQ: 51,103,033 / 100,000,000 tokens
+FineWeb-HQ: 52,103,833 / 100,000,000 tokens
+FineWeb-HQ: 53,103,842 / 100,000,000 tokens
+FineWeb-HQ: 54,105,589 / 100,000,000 tokens
+FineWeb-HQ: 55,106,976 / 100,000,000 tokens
+FineWeb-HQ: 56,108,077 / 100,000,000 tokens
+FineWeb-HQ: 57,108,658 / 100,000,000 tokens
+Step 1,200/2,289 | Loss 10.0994 | LR 1.655e-05 | 103,466 tok/s | 157,286,400 tokens | 52.43% | 0.42h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1200...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
+======================================================================
+UPLOADING CHECKPOINT 1200
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1200
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1000
+FineWeb-HQ: 58,108,705 / 100,000,000 tokens
+FineWeb-HQ: 59,109,817 / 100,000,000 tokens
+FineWeb-HQ: 60,115,751 / 100,000,000 tokens
+FineWeb-HQ: 61,116,056 / 100,000,000 tokens
+FineWeb-HQ: 62,116,477 / 100,000,000 tokens
+FineWeb-HQ: 63,116,951 / 100,000,000 tokens
+FineWeb-HQ: 64,118,012 / 100,000,000 tokens
+FineWeb-HQ: 65,122,172 / 100,000,000 tokens
+FineWeb-HQ: 66,122,389 / 100,000,000 tokens
+FineWeb-HQ: 67,124,930 / 100,000,000 tokens
+FineWeb-HQ: 68,124,933 / 100,000,000 tokens
+FineWeb-HQ: 69,126,235 / 100,000,000 tokens
+FineWeb-HQ: 70,127,674 / 100,000,000 tokens
+Step 1,300/2,289 | Loss 10.0603 | LR 1.461e-05 | 103,354 tok/s | 170,393,600 tokens | 56.80% | 0.46h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1300...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.95it/s]
+======================================================================
+UPLOADING CHECKPOINT 1300
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1300
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1100
+FineWeb-HQ: 71,128,320 / 100,000,000 tokens
+FineWeb-HQ: 72,130,508 / 100,000,000 tokens
+FineWeb-HQ: 73,131,425 / 100,000,000 tokens
+FineWeb-HQ: 74,145,259 / 100,000,000 tokens
+FineWeb-HQ: 75,168,512 / 100,000,000 tokens
+FineWeb-HQ: 76,169,236 / 100,000,000 tokens
+FineWeb-HQ: 77,170,751 / 100,000,000 tokens
+FineWeb-HQ: 78,171,712 / 100,000,000 tokens
+FineWeb-HQ: 79,171,903 / 100,000,000 tokens
+FineWeb-HQ: 80,172,653 / 100,000,000 tokens
+FineWeb-HQ: 81,180,767 / 100,000,000 tokens
+FineWeb-HQ: 82,209,091 / 100,000,000 tokens
+FineWeb-HQ: 83,210,028 / 100,000,000 tokens
+Step 1,400/2,289 | Loss 10.0248 | LR 1.270e-05 | 103,746 tok/s | 183,500,800 tokens | 61.17% | 0.49h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1400...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.83it/s]
+======================================================================
+UPLOADING CHECKPOINT 1400
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1400
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1200
+FineWeb-HQ: 84,221,481 / 100,000,000 tokens
+FineWeb-HQ: 85,221,902 / 100,000,000 tokens
+FineWeb-HQ: 86,221,927 / 100,000,000 tokens
+FineWeb-HQ: 87,224,781 / 100,000,000 tokens
+FineWeb-HQ: 88,226,957 / 100,000,000 tokens
+FineWeb-HQ: 89,227,905 / 100,000,000 tokens
+FineWeb-HQ: 90,230,257 / 100,000,000 tokens
+FineWeb-HQ: 91,234,535 / 100,000,000 tokens
+FineWeb-HQ: 92,235,135 / 100,000,000 tokens
+FineWeb-HQ: 93,235,228 / 100,000,000 tokens
+FineWeb-HQ: 94,235,284 / 100,000,000 tokens
+FineWeb-HQ: 95,242,150 / 100,000,000 tokens
+FineWeb-HQ: 96,254,650 / 100,000,000 tokens
+Step 1,500/2,289 | Loss 9.9967 | LR 1.088e-05 | 103,650 tok/s | 196,608,000 tokens | 65.54% | 0.53h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1500...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.86it/s]
+======================================================================
+UPLOADING CHECKPOINT 1500
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1500
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1300
+FineWeb-HQ: 97,258,984 / 100,000,000 tokens
+FineWeb-HQ: 98,259,041 / 100,000,000 tokens
+FineWeb-HQ: 99,263,648 / 100,000,000 tokens
+FineWeb-HQ complete: 100,000,000 tokens
+Opening SmolLM-Corpus cosmopedia-v2...
+README.md: 100% 7.05k/7.05k [00:00<00:00, 847kB/s]Resolving data files: 100% 104/104 [00:00<00:00, 6458.13it/s]Resolving data files: 100% 104/104 [00:00<00:00, 7784.83it/s]
+======================================================================
+STREAMING SmolLM-Corpus
+======================================================================
+Target tokens: 100,000,000
+Progress state: 0
+SmolLM-Corpus: 1,000,389 / 100,000,000 tokens
+SmolLM-Corpus: 2,000,612 / 100,000,000 tokens
+SmolLM-Corpus: 3,000,937 / 100,000,000 tokens
+SmolLM-Corpus: 4,001,355 / 100,000,000 tokens
+SmolLM-Corpus: 5,001,629 / 100,000,000 tokens
+SmolLM-Corpus: 6,002,147 / 100,000,000 tokens
+SmolLM-Corpus: 7,002,160 / 100,000,000 tokens
+SmolLM-Corpus: 8,002,233 / 100,000,000 tokens
+SmolLM-Corpus: 9,002,382 / 100,000,000 tokens
+Step 1,600/2,289 | Loss 10.0072 | LR 9.168e-06 | 101,065 tok/s | 209,715,200 tokens | 69.91% | 0.57h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1600...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
+======================================================================
+UPLOADING CHECKPOINT 1600
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1600
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1400
+SmolLM-Corpus: 10,002,523 / 100,000,000 tokens
+SmolLM-Corpus: 11,002,661 / 100,000,000 tokens
+SmolLM-Corpus: 12,003,098 / 100,000,000 tokens
+SmolLM-Corpus: 13,003,147 / 100,000,000 tokens
+SmolLM-Corpus: 14,004,159 / 100,000,000 tokens
+SmolLM-Corpus: 15,004,183 / 100,000,000 tokens
+SmolLM-Corpus: 16,004,330 / 100,000,000 tokens
+SmolLM-Corpus: 17,005,124 / 100,000,000 tokens
+SmolLM-Corpus: 18,005,586 / 100,000,000 tokens
+SmolLM-Corpus: 19,006,013 / 100,000,000 tokens
+SmolLM-Corpus: 20,006,056 / 100,000,000 tokens
+SmolLM-Corpus: 21,006,417 / 100,000,000 tokens
+SmolLM-Corpus: 22,006,642 / 100,000,000 tokens
+Step 1,700/2,289 | Loss 9.9950 | LR 7.613e-06 | 104,388 tok/s | 222,822,400 tokens | 74.27% | 0.60h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1700...
+Writing model shards: 100% 1/1 [00:00<00:00, 11.01it/s]
+======================================================================
+UPLOADING CHECKPOINT 1700
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1700
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1500
+SmolLM-Corpus: 23,007,410 / 100,000,000 tokens
+SmolLM-Corpus: 24,007,513 / 100,000,000 tokens
+SmolLM-Corpus: 25,008,024 / 100,000,000 tokens
+SmolLM-Corpus: 26,008,152 / 100,000,000 tokens
+SmolLM-Corpus: 27,008,820 / 100,000,000 tokens
+SmolLM-Corpus: 28,009,655 / 100,000,000 tokens
+SmolLM-Corpus: 29,009,777 / 100,000,000 tokens
+SmolLM-Corpus: 30,010,160 / 100,000,000 tokens
+SmolLM-Corpus: 31,010,762 / 100,000,000 tokens
+SmolLM-Corpus: 32,010,887 / 100,000,000 tokens
+SmolLM-Corpus: 33,011,339 / 100,000,000 tokens
+SmolLM-Corpus: 34,011,816 / 100,000,000 tokens
+SmolLM-Corpus: 35,012,785 / 100,000,000 tokens
+Step 1,800/2,289 | Loss 9.9749 | LR 6.242e-06 | 104,425 tok/s | 235,929,600 tokens | 78.64% | 0.63h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1800...
+Writing model shards: 100% 1/1 [00:00<00:00, 11.00it/s]
+======================================================================
+UPLOADING CHECKPOINT 1800
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1800
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1600
+SmolLM-Corpus: 36,013,511 / 100,000,000 tokens
+SmolLM-Corpus: 37,013,686 / 100,000,000 tokens
+SmolLM-Corpus: 38,015,046 / 100,000,000 tokens
+SmolLM-Corpus: 39,015,520 / 100,000,000 tokens
+SmolLM-Corpus: 40,016,051 / 100,000,000 tokens
+SmolLM-Corpus: 41,016,215 / 100,000,000 tokens
+SmolLM-Corpus: 42,016,528 / 100,000,000 tokens
+SmolLM-Corpus: 43,016,537 / 100,000,000 tokens
+SmolLM-Corpus: 44,016,544 / 100,000,000 tokens
+SmolLM-Corpus: 45,017,221 / 100,000,000 tokens
+SmolLM-Corpus: 46,017,703 / 100,000,000 tokens
+SmolLM-Corpus: 47,017,830 / 100,000,000 tokens
+SmolLM-Corpus: 48,018,530 / 100,000,000 tokens
+SmolLM-Corpus: 49,019,055 / 100,000,000 tokens
+Step 1,900/2,289 | Loss 9.9581 | LR 5.086e-06 | 104,428 tok/s | 249,036,800 tokens | 83.01% | 0.67h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-1900...
+Writing model shards: 100% 1/1 [00:00<00:00, 11.07it/s]
+======================================================================
+UPLOADING CHECKPOINT 1900
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1900
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1700
+SmolLM-Corpus: 50,019,088 / 100,000,000 tokens
+SmolLM-Corpus: 51,019,166 / 100,000,000 tokens
+SmolLM-Corpus: 52,019,182 / 100,000,000 tokens
+SmolLM-Corpus: 53,019,341 / 100,000,000 tokens
+SmolLM-Corpus: 54,019,557 / 100,000,000 tokens
+SmolLM-Corpus: 55,019,622 / 100,000,000 tokens
+SmolLM-Corpus: 56,020,576 / 100,000,000 tokens
+SmolLM-Corpus: 57,021,061 / 100,000,000 tokens
+SmolLM-Corpus: 58,021,198 / 100,000,000 tokens
+SmolLM-Corpus: 59,022,144 / 100,000,000 tokens
+SmolLM-Corpus: 60,022,635 / 100,000,000 tokens
+SmolLM-Corpus: 61,022,949 / 100,000,000 tokens
+SmolLM-Corpus: 62,023,318 / 100,000,000 tokens
+Step 2,000/2,289 | Loss 9.9444 | LR 4.167e-06 | 104,400 tok/s | 262,144,000 tokens | 87.38% | 0.70h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-2000...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.99it/s]
+======================================================================
+UPLOADING CHECKPOINT 2000
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-2000
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1800
+SmolLM-Corpus: 63,024,246 / 100,000,000 tokens
+SmolLM-Corpus: 64,024,602 / 100,000,000 tokens
+SmolLM-Corpus: 65,025,944 / 100,000,000 tokens
+SmolLM-Corpus: 66,027,471 / 100,000,000 tokens
+SmolLM-Corpus: 67,027,613 / 100,000,000 tokens
+SmolLM-Corpus: 68,028,290 / 100,000,000 tokens
+SmolLM-Corpus: 69,028,776 / 100,000,000 tokens
+SmolLM-Corpus: 70,028,797 / 100,000,000 tokens
+SmolLM-Corpus: 71,030,241 / 100,000,000 tokens
+SmolLM-Corpus: 72,030,477 / 100,000,000 tokens
+SmolLM-Corpus: 73,030,534 / 100,000,000 tokens
+SmolLM-Corpus: 74,030,705 / 100,000,000 tokens
+SmolLM-Corpus: 75,030,996 / 100,000,000 tokens
+Step 2,100/2,289 | Loss 9.9332 | LR 3.505e-06 | 104,456 tok/s | 275,251,200 tokens | 91.75% | 0.74h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-2100...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.96it/s]
+======================================================================
+UPLOADING CHECKPOINT 2100
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-2100
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1900
+SmolLM-Corpus: 76,031,414 / 100,000,000 tokens
+SmolLM-Corpus: 77,031,466 / 100,000,000 tokens
+SmolLM-Corpus: 78,032,243 / 100,000,000 tokens
+SmolLM-Corpus: 79,032,688 / 100,000,000 tokens
+SmolLM-Corpus: 80,032,933 / 100,000,000 tokens
+SmolLM-Corpus: 81,033,099 / 100,000,000 tokens
+SmolLM-Corpus: 82,034,041 / 100,000,000 tokens
+SmolLM-Corpus: 83,034,272 / 100,000,000 tokens
+SmolLM-Corpus: 84,035,283 / 100,000,000 tokens
+SmolLM-Corpus: 85,035,460 / 100,000,000 tokens
+SmolLM-Corpus: 86,035,589 / 100,000,000 tokens
+SmolLM-Corpus: 87,036,259 / 100,000,000 tokens
+SmolLM-Corpus: 88,036,685 / 100,000,000 tokens
+Step 2,200/2,289 | Loss 9.9239 | LR 3.114e-06 | 104,304 tok/s | 288,358,400 tokens | 96.12% | 0.77h
+VRAM: 1.57 GB allocated / 13.91 GB reserved
+Saving checkpoint-2200...
+Writing model shards: 100% 1/1 [00:00<00:00, 10.98it/s]
+======================================================================
+UPLOADING CHECKPOINT 2200
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+Checkpoint uploaded to:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-2200
+Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-2000
+SmolLM-Corpus: 89,036,736 / 100,000,000 tokens
+SmolLM-Corpus: 90,037,952 / 100,000,000 tokens
+SmolLM-Corpus: 91,038,162 / 100,000,000 tokens
+SmolLM-Corpus: 92,038,464 / 100,000,000 tokens
+SmolLM-Corpus: 93,038,712 / 100,000,000 tokens
+SmolLM-Corpus: 94,038,967 / 100,000,000 tokens
+SmolLM-Corpus: 95,039,446 / 100,000,000 tokens
+SmolLM-Corpus: 96,039,542 / 100,000,000 tokens
+SmolLM-Corpus: 97,040,355 / 100,000,000 tokens
+SmolLM-Corpus: 98,041,440 / 100,000,000 tokens
+SmolLM-Corpus: 99,041,658 / 100,000,000 tokens
+SmolLM-Corpus complete: 100,000,000 tokens
+======================================================================
+DATASET STREAM EXHAUSTED
+======================================================================
+Final dataset batch was incomplete.
+Stopping without an incomplete optimizer step.
+======================================================================
+TRAINING LOOP FINISHED
+======================================================================
+Completed optimizer steps: 2,288
+Configured optimizer steps: 2,289
+Dataset progress:
+  FineWeb-Edu: 100,000,000 / 100,000,000
+  FineWeb-HQ:  100,000,000 / 100,000,000
+  SmolLM:      100,000,000 / 100,000,000
+======================================================================
+PRETRAINING FINISHED
+======================================================================
+Final steps: 2,288
+Final tokens: 299,892,736
+Target tokens: 300,000,000
+Actual tokens/parameter: 293.87:1
+======================================================================
+SAVING FINAL MODEL
+======================================================================
+Writing model shards: 100% 1/1 [00:00<00:00, 21.09it/s]
+======================================================================
+UPLOADING COMPLETE MODEL TO HUGGING FACE
+======================================================================
+No files have been modified since last commit. Skipping to prevent empty commit.
+WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.
+FINAL MODEL UPLOAD SUCCESSFUL!
+Final model:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini
+======================================================================
+CHAT TEMPLATE TEST
+======================================================================
+<|system|>
+You are AppleMind, a helpful AI assistant.
+<|user|>
+Hello!
+<|assistant|>
+======================================================================
+GENERATION TEST
+======================================================================
+Prompt: Once upon a time
+------------------------------------------------------------
+Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked
+ its G during work used after several There and at there
+b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many
+------------------------------------------------------------
+Prompt: The little boy
+------------------------------------------------------------
+The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many
+------------------------------------------------------------
+Prompt: In the forest
+------------------------------------------------------------
+In the forest often
+ form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used
+ to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but
+------------------------------------------------------------
+======================================================================
+APPLE MIND 1.0 MINI COMPLETE
+======================================================================
+Parameters: 1,020,480
+Tokens: 299,892,736
+Tokens/parameter: 293.87:1
+Dataset mixture:
+  FineWeb-Edu: 100M
+  FineWeb-HQ:  100M
+  SmolLM:      100M
+Checkpoint repository:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints
+Final model repository:
+https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini
+AppleMind 1.0 Mini complete!