Separating Knowledge and Perception with Procedural Data

Welcome to the repository for "Separating Knowledge with Procedural Data"! Here are the instructions for reproducing all trainings and evaluations in the paper.

[Project page] [Paper]

Creating environment

conda create -n separating_knowledge python=3.9.20
conda activate separating_knowledge
pip install torch==2.4.1 torchaudio==2.4.1 torchvision==0.19.1
pip install importlib-metadata
pip install opencv-python
pip install timm
pip install matplotlib
pip install pandas
pip install pykeops
pip install seaborn
pip install medmnist
pip install pycocotools
pip install scikit-learn

Downloading and generating data

Follow each subsection instructions, starting each subsection in the main directory.

Downloading ImageNet, Places

See their respective websites for instructions ImageNet and Places, then symlink to ./data.

Downloading Stylegan, Shaders, Shaders Mixup

cd download_data_scripts
./download_stylegan.sh YOUR_DATASETS_FOLDER_HERE
./download_shaders.sh YOUR_DATASETS_FOLDER_HERE

Setting up data symlinks for training scripts

mkdir data
cd data

mkdir imagenet
mkdir places
mkdir shaders_mixup
mkdir shaders
mkdir stylegan

ln -s PATH_TO_IMAGENET/train imagenet/train
ln -s PATH_TO_IMAGENET/val imagenet/val
ln -s PATH_TO_PLACES/train places/train
ln -s PATH_TO_SHADERS_MIXUP/train shaders_mixup/train
ln -s PATH_TO_SHADERS/train shaders/train
ln -s PATH_TO_STYLEGAN/train stylegan/train

Creating Shaders KML and Shaders KML Mixup

cd data_generation

# Shaders KML
./shaders_kml.sh PATH_TO_DATASET_FOLDER
mkdir ../data/shaders_kml
ln -s PATH_TO_SHADERS_KML/train ../data/shaders_kml/train

# Shaders KML Mixup
./shaders_kml_mixup.sh PATH_TO_DATASET_FOLDER
mkdir ../data/shaders_kml_mixup
ln -s PATH_TO_SHADERS_KML_MIXUP/train ../data/shaders_kml_mixup/train

Download CUB200, Flowers102, Food101

See their respective websites for instructions CUB, Flowers102, and Food, then symlink to ./data.

Download MedicalMNIST

See the website MedicalMNIST for instructions, download in 224x244 resolution and then symlink to ./data.

Download COCO, Ade20k, Pascal-VOC

See their respective websites for instructions COCO, Ade20k, and Pascal, then symlink to ./data.

Training + ImageNet-1K evaluation

cd dino
./scripts/train_imagenet.sh
./scripts/train_places.sh
./scripts/train_shaders_kml_mixup.sh
./scripts/train_shaders_kml.sh
./scripts/train_shaders_mixup.sh
./scripts/train_shaders.sh
./scripts/train_stylegan.sh

Evaluation

Fine-grained dataset evaluation

cd dino

# ENCODER_NAME=imagenet, shaders_kml_mixup, etc.
./scripts/evals/eval_knn_cub.sh ENCODER_NAME # ENCODER_NAME=imagenet, shaders_kml_mixup, etc.
./scripts/evals/eval_knn_flowers.sh ENCODER_NAME # ENCODER_NAME=imagenet, shaders_kml_mixup, etc.
./scripts/evals/eval_knn_food.sh ENCODER_NAME # ENCODER_NAME=imagenet, shaders_kml_mixup, etc.

Medical evaluation

cd dino

# ENCODER_NAME=imagenet, shaders_kml_mixup, etc.
# DATASET_NAME=bloodmnist, breastmnist, dermamnist, octmnist, organamnist, organcmnist, organsmnist, pathmnist, pneumoniamnist, tissuemnist
./scripts/evals/eval_knn_medicalmnist.sh ENCODER_NAME DATASET_NAME 

Segmentation evaluation

Dump COCO image features

cd dino

# ENCODER_NAME=imagenet, shaders_kml_mixup, etc.
python dump_coco_features.py --pretrained_weights ./encoders/ENCODER_NAME/checkpoint.pth

Go to notebooks

Open notebooks notebook_figures/figures_segmentation_zeroshot.ipynb, notebook_figures/figures_segmentation_incontext.ipynb, and notebook_figures/figures_segmentation_knn.ipynb.