kungchuking/E2E_SCSI

[ECE1508 Final Project] Joint Learning of Exposure Patterns and Stereo Depth from Coded Snapshots

This project introduces a novel, end-to-end learning approach that jointly addresses two traditionally separate computer vision challenges: Snapshot Compressed Image (SCI) decoding and dynamic stereo depth estimation. The framework is an adaptation of the DynamicStereo repository and was trained using the DynamicReplica dataset.

Dataset

The DynamicReplica dataset consists of 145200 stereo frames (524 videos) with humans and animals in motion.

Download the Dynamic Replica dataset

Due to the enormous size of the original dataset, we created the links_lite.json file to enable quick testing by downloading just a small portion of the dataset.

python ./scripts/download_dynamic_replica.py --link_list_file links_lite.json --download_folder ./dynamic_replica_data --download_splits test train valid real

To download the full dataset, please visit the original site created by Meta.

Installation

To set up and run the project, please follow these steps.

Setup the root for all source files:

git clone https://github.com/kungchuking/E2E_SCSI.git
cd dynamic_stereo

Create a conda env:

conda create -n dynamicstereo python=3.8
conda activate dynamicstereo

Install requirements

pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
pip install -r requirements.txt

Evaluation

To download the pre-trained model weights (checkpoints), please follow the instructions below.

Command Line Download

You can use the following commands to create the required directory and download the primary checkpoint directly from the Hugging Face repository:

mkdir dynamicstereo_sf_dr
wget -O dynamicstereo_sf_dr/model_dynamic-stereo_050895.pth "https://huggingface.co/kungchuking/E2E_SCSI/resolve/main/dynamicstereo_sf_dr/model_dynamic-stereo_050895.pth"

Manual Download

Alternatively, you can manually download the checkpoints by clicking the link. Ensure the downloaded file is placed in the required path: ./dynamicstereo_sf_dr/.

Evaluation Notebook

For detailed instructions on how to evaluate the model, please refer to the dedicated evaluation notebook.

Evaluation and Validation

To execute the final evaluation on the DynamicReplica test set, navigate to the evaluationdirectory and run the following Python script:

cd evaluation
python evaluate.py

Training

Hardware and Memory Requirements

Training the model requires a minimum of a 50GB GPU.

• Memory Adjustment: If your GPU memory is limited, you may decrease the image_size and/or the sample_len parameters.
• Resolution Note: The chosen image_size of 480x640 corresponds to the native resolution of the custom-designed coded-exposure camera used for our research.
• Compression Impact: Reducing the sample_length will inherently decrease the effective compression ratio for the Snapshot Compressed Imaging (SCI) process. Before starting training, you must download the Dynamic Replica dataset.

Execution

If you are running on a Linux machine, use the provided shell script for training:

./train.csh

For other operating systems, you can open the ./train.csh file and manually copy and execute the instruction.

License

Portions of the project are available under separate license terms: DynamicStereo is licensed under CC-BY-NC, RAFT-Stereo is licensed under the MIT license, LoFTR and CREStereo are licensed under the Apache 2.0 license.