AFUN: Towards an Affordance Foundation Model for Functionality Understanding

Where + How for Functionality Understanding

Zhaoning Wang^1,*  ·  Yi Zhong^1,*  ·  Jiawei Fu²  ·  Henrik I. Christensen²  ·  Jun Gao^1,3

¹University of Michigan    ²UC San Diego    ³NVIDIA     ^*Equal contribution

TL;DR

RGB + text query + depth ──► AFUN model ──► mask + 3D motion curve

From a single RGB observation and a natural-language task description, AFUN predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). It pairs frozen pretrained vision-language features with lightweight trainable modules for joint mask + motion prediction.

Currently, this repository provides inference-only code: load a trained AFUN checkpoint and run it on your own images. Depth is handled automatically — Depth-Anything-3 generates metric depth from a single RGB image, or lingbot-depth refines a noisy sensor (e.g. RealSense) depth map if you provide one.

Contents

• 📋 TODO
• 🔧 Installation
• 🚀 Quick Start
• 🙏 Acknowledgements
• 📚 Citation

---

📋 TODO

• Inference code & demo
• Training Code
• Dataset, Evaluation, and Benchmark
• Robot Demonstration Code
• Claude Agent md Files

---

🔧 Installation

1. Clone (with submodules)

git clone --recursive https://github.com/EricWang12/AFUN.git
cd AFUN
# (already cloned non-recursively? run: git submodule update --init --recursive)

2. Create the environment

bash install.sh
conda activate afun

NOTE: install.sh defaults to Python 3.11 and CUDA 13 (cu130). For a different CUDA builds, edit the file directly.

3. Get the checkpoint

The AFUN checkpoint afun.pt is hosted on Google Drive. Fetch it automatically — this also pre-caches the Hugging Face backbone weights (e.g. Qwen3-VL-8B, SAM3, Depth-Anything-3, lingbot, Sonata):

python scripts/prefetch_weights.py

Or download it by hand from the Google Drive link and place it at:

checkpoints/afun.pt

The backbone weights (Qwen3-VL-8B, SAM3, Depth-Anything-3, lingbot, Sonata) download automatically from Hugging Face on first run (cached); prefetch_weights.py fetches those too.

---

🚀 Quick Start

Run the demo with an RGB image, a text prompt, and (optionally) a sensor-depth folder:

python demo.py --rgb <path_to_rgb_image> --query <text_prompt> --depth-dir <path_to_depth_map>

--depth-dir would be a folder containing two files:

<depth-dir>/
├── depth.npy    # raw sensor depth — float32, millimeters, shape (H, W) matching the RGB
└── cam_K.txt    # 3×3 camera-intrinsics matrix

By default lingbot refines this depth; pass --no-refine to skip refinement and use the raw sensor depth as-is. Omit --depth-dir entirely and Depth-Anything-3 estimates depth from the RGB instead.

NOTE: Raw sensor depth without refinement generally does not work well with our model, as it is not trained on such data.

Example 1 — sensor depth (lingbot refines the bundled depth via --depth-dir):

python demo.py --rgb       examples/robot_toaster_oven/rgb.png \
               --query     "Open the toaster oven door" \
               --depth-dir examples/robot_toaster_oven

Example 2 — Depth-Anything-3 (no --depth-dir; DA3 estimates depth from the RGB):

python demo.py --rgb examples/kitchen_fridge/rgb.png --query "Open the refrigerator door"

Output (per run, under outputs/):

File	Contents
pred_seg.png	RGB + predicted affordance mask (red) + motion trajectory (yellow→blue, yellow = contact)
pred.npz	mask, motion parameters, intrinsics, confidence
pred_3d.html	interactive 3D point cloud + motion curve (omit with --no-3d)
meta.json / summary.csv / montage.png	run metadata (single / batch)

---

Run all bundled examples (model loads once, then loops over examples/):

python run_examples.py            # add --no-3d to skip the interactive 3D HTML

🙏 Acknowledgements

This project builds on Qwen3-VL, SAM3, Depth-Anything-3, lingbot-depth, and Sonata. We thank the authors for releasing their code and models.

---

📚 Citation

@article{wang2026afun,
  title   = {{AFUN}: Towards an Affordance Foundation Model for Functionality Understanding},
  author  = {Wang, Zhaoning and Zhong, Yi and Fu, Jiawei and Christensen, Henrik I. and Gao, Jun},
  journal = {arXiv preprint arXiv:2606.02551},
  year    = {2026},
}