linlianjiang/ffgs

feedforward — resume bundle (step 100000 / 154000)

A training run stopped at optimizer step 100000 of 154000. This repo holds the checkpoint and the code to continue it. Two things are not here and you must supply them yourself — see below.

Files

File	Size	Goes to
checkpoint_last.pt	12.9 GiB	feedforward/outputs/full/checkpoint_last.pt
feedforward_code.tar.gz	523 KiB	unpacks to feedforward/
SHA256	—	checksums (the Hub already verifies integrity on download)

hf download linlianjiang/ffgs --local-dir ./bundle

What you must add

1. The pretrained VGGT-Omega backbone — weights/vggt_omega_1b_512.pt, the official 1B fp32 checkpoint. It is deliberately not in this repo: the HuggingFace source repo is gated, so you request access and download it under your own agreement. Verify you got the identical file:

sha256  c02da418b18bb01d0392598d3f6147366bcde1bb70fd08a5e3bf7925b0667934
size    4576706117 bytes

Do not substitute facebook/VGGT-1B — different architecture, the keys will not load (dense_head here vs. depth_head/point_head/track_head there). Do not substitute the bf16 release either; it shifts the distillation numerics 100k steps into a run.

2. RealEstate10K in the pixelSplat shard layout (~510 G): <root>/train/*.torch, <root>/test/*.torch, <root>/test/index.json.

Setup

mkdir -p feedforward && tar -xzf feedforward_code.tar.gz -C feedforward
cd feedforward
mkdir -p outputs/full weights
mv ../checkpoint_last.pt outputs/full/
cp /wherever/you/downloaded/vggt_omega_1b_512.pt weights/

pip install -r requirements.txt      # install the torch build matching your driver first

Three things that will bite you

1. weights/vggt_omega_1b_512.pt is not optional, even to resume. model.load_pretrained() at train.py:405 runs unconditionally before the resume block at train.py:438, and asserts 0 missing / 0 unexpected. Without the file the job dies within seconds — even though the checkpoint is about to overwrite those weights anyway. The frozen teacher then loads it a second time (train.py:417), because DISTILL_TEACHER=1 and it is used at every step.

2. checkpoint_last.pt must sit inside $OUT_DIR (outputs/full by default). train.py:438 checks that one path and nothing else; anywhere else and the run starts fresh from step 0 without complaining. Confirm RESUMED from step 100000 appears in the log before walking away.

3. Throughput drops here. Step 100000 is exactly where the curriculum's second phase opens (ANISO_AFTER=100000 SHAPE_AFTER=100000) and the cosine schedule restarts. Expect ~3 it/s rather than the ~17 it/s of the first 100k steps. All 54k remaining steps are in the expensive phases — budget wall-clock accordingly.

Running

Smoke test first. It hard-fails on things like gsplat being unable to rasterize, which otherwise surfaces as a silent ~15 dB quality drop rather than an exception:

DATA_ROOT=/path/to/re10k scripts/smoke.sh

Then confirm the ARM= line reads curriculum=[aniso@100000 shapeK3@100000->K5@130000 app@130000] with restarts=100000,130000, and launch:

GPUS=4 DATA_ROOT=/path/to/re10k scripts/train_full.sh

docs/TRAINING.md inside the tarball has the full curriculum, multi-GPU notes, protocols and the traps hit along the way. Read it before committing a long allocation.

logs/ holds the launchers from the originating cluster (SLURM, 4x H100). Their partition names, account and module paths are site-specific — reference, not scripts to run as-is.

Note: scripts/train_full.sh differs from git HEAD by one change (torchrun → $PY -m torch.distributed.run), which fixes a job that died on torchrun: not found after the recipe check had already passed. That is intentional; keep it.