Instructions to use deepseek-ai/DeepSeek-R1-Distill-Qwen-32B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-R1-Distill-Qwen-32B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use deepseek-ai/DeepSeek-R1-Distill-Qwen-32B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

SGLang

How to use deepseek-ai/DeepSeek-R1-Distill-Qwen-32B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-R1-Distill-Qwen-32B with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
```

[RESOLVED] Model is not outputting the <think> token at the beginning.

#37

by bsvaz - opened Feb 14, 2025

Discussion

bsvaz

Feb 14, 2025

•

edited Feb 19, 2025

Neither this model nor the distill 1.5B model outputs the opening thinking token <think> before starting to think, but they do output the closing token </think>.

Edit 1: I found the solution in the model card, here is:
"Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting <think>\n\n</think>) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with <think>\n at the beginning of every output."

But I still have a concern about this: Is there a way to enforce this while using the HF Inference API?

Edit 2: I found the solution:

If you want to interact with the model similar to the way you interact using model.generate() you should make a direct HTTP request to the Inference API like this:

API_URL = "https://huggingface.co/static-proxy/api-inference.huggingface.co/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"  

headers = {"Authorization": f"Bearer {hf_api_key}"}  

# Construct the EXACT input string 
formatted_input = '<｜begin▁of▁sentence｜>' + '<｜User｜>' + prompt + '<｜Assistant｜>' + '<think>\n'  

payload = { "inputs": formatted_input, "parameters": { "do_sample": False, "temperature": 0.6 } } 
response = requests.post(API_URL, headers=headers, json=payload)  

print(response.json())```

NekroniX588

Feb 15, 2025

•

edited Feb 15, 2025

Hi. Thanks for the message. Can you please show how you pass the token to the model?

Is this way correct?
chat_completion = client.chat.completions.create( messages=[ {"role": "assistant", "content": "<think>\n"}, { "role": "user", "content": f"""what is happiness?""", } ], model="default", temperature=0.35, top_p=0.9 ) print(chat_completion.choices[0].message.content)

bsvaz

Feb 15, 2025

Hi. Thanks for the message. Can you please show how you pass the token to the model?

Is this way correct?
chat_completion = client.chat.completions.create( messages=[ {"role": "assistant", "content": "<think>\n"}, { "role": "user", "content": f"""what is happiness?""", } ], model="default", temperature=0.35, top_p=0.9 ) print(chat_completion.choices[0].message.content)

With this, you are using the Inference API. The model works by completing a single string, so the messages input is processed into one single string (which also includes special tokens).
What I did was to download the model and do the inference, here is the code

model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B")

inputs = tokenizer('<｜User｜>' + prompt + '<｜Assistant｜>' + '<think>\n', return_tensors="pt")

# Generate text
model.eval()
with torch.no_grad():
    outputs = model.generate(**inputs.to(device), max_new_tokens=500)

# Decode the generated text
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)

I tried this way you told me now, but I'm not sure if it considered the <think> because it was not outputted in the message.

yiwan

Feb 19, 2025

Has this issue been resolved?

bsvaz

Feb 19, 2025

Has this issue been resolved?

Yes, I'll edit the post with the solution so other people with the same question can know.

bsvaz changed discussion title from Model is not outputting the <think> token at the beginning. to [RESOLVED] Model is not outputting the <think> token at the beginning. Feb 19, 2025

sagawang

Mar 13, 2025

•

edited Mar 13, 2025

edit "tokenizer_config.json"，last line，delete "<think>\\n".

{{'<｜Assistant｜><think>\\n'}} change to {{'<｜Assistant｜>'}}

After performing these operations, reloading the model will output content that starts with "<think>".

Ziger

Jul 24, 2025

edit "tokenizer_config.json"，last line，delete "<think>\\n".

{{'<｜Assistant｜><think>\\n'}} change to {{'<｜Assistant｜>'}}

After performing these operations, reloading the model will output content that starts with "<think>".

that's the problem in the model card

"Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting \n\n) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with \n at the beginning of every output."

actually, if you ask DeepSeek-R1-Distill-Qwen-32B with modified tokenizer_config.json, it would skip thinking.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment