README.md

metadata

title: DSSD Demo
emoji: 🚀
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.3.0
app_file: app.py
pinned: false
license: apache-2.0

🚀 Dynamic Self-Speculative Decoding (DSSD) Demo

This demo showcases early exit inference with true speculative decoding. Tokens are drafted from intermediate layers when the model is confident, then verified by the full model. The final output matches full-model greedy decoding. This CPU-hosted implementation may run slower than standard decoding.

Features

• Speculative Decoding: Uses early exit heads to draft tokens, then verifies them with the full model.
• Streaming Output: Watch the generation process live, including drafting and verification statuses.
• Model Comparison: Compare performance and output between DSSD and the full model side-by-side.
• Color-coded Visualization: Each token is colored based on which head/layer generated it.

How it works

1. Draft Phase: The model tries to predict the next token(s) using early exit heads placed at intermediate layers.
2. Verification Phase: The full model checks the drafted tokens in a single forward pass.
3. Acceptance: Matching tokens are kept. The first mismatch is corrected, and the process restarts.

Models

• Qwen 3 0.6B: Using 4 auxiliary heads at layers 5, 11, 16, and 22.
• Llama 3 8B: Using 3 auxiliary heads at layers 8, 16, and 24. This option appears only with CUDA and access to the gated Meta Llama 3 base model; it is unavailable on the CPU-hosted Space.

Quick Start (Local)

pip install -r requirements.txt
python app.py