Instructions to use jadechoghari/Ferret-UI-Llama8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jadechoghari/Ferret-UI-Llama8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="jadechoghari/Ferret-UI-Llama8b", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("jadechoghari/Ferret-UI-Llama8b", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jadechoghari/Ferret-UI-Llama8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jadechoghari/Ferret-UI-Llama8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jadechoghari/Ferret-UI-Llama8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/jadechoghari/Ferret-UI-Llama8b

SGLang

How to use jadechoghari/Ferret-UI-Llama8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jadechoghari/Ferret-UI-Llama8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jadechoghari/Ferret-UI-Llama8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jadechoghari/Ferret-UI-Llama8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jadechoghari/Ferret-UI-Llama8b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use jadechoghari/Ferret-UI-Llama8b with Docker Model Runner:
```
docker model run hf.co/jadechoghari/Ferret-UI-Llama8b
```

how to run it in mac?

by chenliangjing - opened Nov 19, 2024

Discussion

chenliangjing

Nov 19, 2024

Dear Ferret-UI-Llama8b repository authors,

I am writing to inquire about running the model_UI code on a Mac computer in a CPU-only environment. I have reviewed the code and made some modifications to ensure it can run without CUDA support, but I would appreciate your guidance on the best approach.

Specifically, I have made the following changes to the code:

Disabled CUDA support by setting torch.cuda.is_available = lambda: False and setting the CUDA_VISIBLE_DEVICES and TORCH_DEVICE environment variables.
Set the data_type argument to use either torch.float16, torch.bfloat16, or torch.float32 depending on the user's preference, in order to leverage mixed precision on the CPU.
Modified the image preprocessing function to use a custom image_process_func that resizes the images without center cropping, as the original code assumes CUDA availability.
Ensured that any region masks are converted to the appropriate data type before being used in the model.

These changes should allow the model_UI code to run on a Mac in a CPU-only environment. However, I would appreciate if you could provide any additional guidance or considerations for running the code in this configuration. For example, are there any specific requirements or recommendations for the CPU hardware, or any other optimizations that could be made to improve the performance on a CPU-only system?

Thank you in advance for your assistance. I look forward to your response and to continuing to work with your excellent Ferret-UI-Llama8b project.

Best regards,
chenliangjing

jadechoghari

Owner Jan 8, 2025

Sorry for the late reply, @chenliangjing —this issue slipped past me. I usually reply faster if I’m tagged directly. :)
Sure! The hardware/specifications are similar to those of Llama 8b, which are well-documented (you can look them up). If you’re referring to the Gemma models, they’re also similar to the Gemma 2b specifications.
Hope this helps!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment