Instructions to use ifable/gemma-2-Ifable-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ifable/gemma-2-Ifable-9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ifable/gemma-2-Ifable-9B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ifable/gemma-2-Ifable-9B")
model = AutoModelForCausalLM.from_pretrained("ifable/gemma-2-Ifable-9B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ifable/gemma-2-Ifable-9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ifable/gemma-2-Ifable-9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ifable/gemma-2-Ifable-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ifable/gemma-2-Ifable-9B

SGLang

How to use ifable/gemma-2-Ifable-9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ifable/gemma-2-Ifable-9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ifable/gemma-2-Ifable-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ifable/gemma-2-Ifable-9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ifable/gemma-2-Ifable-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ifable/gemma-2-Ifable-9B with Docker Model Runner:
```
docker model run hf.co/ifable/gemma-2-Ifable-9B
```

gemma-2-Ifable-9B / trainer_log.jsonl

linzaiyun

Upload folder using huggingface_hub

eea2579 verified almost 2 years ago

Raw

History Blame Contribute Delete

9.55 kB

	{"current_steps": 1, "total_steps": 35, "loss": 6.8307, "accuracy": 0.5, "learning_rate": 8e-08, "epoch": 0.028021015761821366, "percentage": 2.86, "elapsed_time": "0:00:53", "remaining_time": "0:30:04", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 2, "total_steps": 35, "loss": 7.3627, "accuracy": 0.5, "learning_rate": 2.6e-07, "epoch": 0.05604203152364273, "percentage": 5.71, "elapsed_time": "0:01:44", "remaining_time": "0:28:51", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 3, "total_steps": 35, "loss": 7.025, "accuracy": 0.6875, "learning_rate": 4.4e-07, "epoch": 0.0840630472854641, "percentage": 8.57, "elapsed_time": "0:02:25", "remaining_time": "0:25:54", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 4, "total_steps": 35, "loss": 6.4674, "accuracy": 0.3125, "learning_rate": 6.2e-07, "epoch": 0.11208406304728546, "percentage": 11.43, "elapsed_time": "0:03:05", "remaining_time": "0:23:59", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 5, "total_steps": 35, "loss": 7.1364, "accuracy": 0.625, "learning_rate": 7.981529564210822e-07, "epoch": 0.14010507880910683, "percentage": 14.29, "elapsed_time": "0:03:49", "remaining_time": "0:22:59", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 6, "total_steps": 35, "loss": 5.6445, "accuracy": 0.6875, "learning_rate": 7.926307788508979e-07, "epoch": 0.1681260945709282, "percentage": 17.14, "elapsed_time": "0:04:31", "remaining_time": "0:21:53", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 7, "total_steps": 35, "loss": 6.5965, "accuracy": 0.375, "learning_rate": 7.834901323040175e-07, "epoch": 0.19614711033274956, "percentage": 20.0, "elapsed_time": "0:05:14", "remaining_time": "0:20:56", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 8, "total_steps": 35, "loss": 6.3454, "accuracy": 0.6875, "learning_rate": 7.70824812183283e-07, "epoch": 0.22416812609457093, "percentage": 22.86, "elapsed_time": "0:05:55", "remaining_time": "0:19:58", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 9, "total_steps": 35, "loss": 5.2256, "accuracy": 0.5, "learning_rate": 7.547647818120495e-07, "epoch": 0.2521891418563923, "percentage": 25.71, "elapsed_time": "0:06:40", "remaining_time": "0:19:16", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 10, "total_steps": 35, "loss": 4.8325, "accuracy": 0.5625, "learning_rate": 7.354748388346194e-07, "epoch": 0.28021015761821366, "percentage": 28.57, "elapsed_time": "0:07:19", "remaining_time": "0:18:18", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 11, "total_steps": 35, "loss": 4.8686, "accuracy": 0.875, "learning_rate": 7.131529241694047e-07, "epoch": 0.30823117338003503, "percentage": 31.43, "elapsed_time": "0:07:58", "remaining_time": "0:17:24", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 12, "total_steps": 35, "loss": 4.1241, "accuracy": 0.625, "learning_rate": 6.880280908672471e-07, "epoch": 0.3362521891418564, "percentage": 34.29, "elapsed_time": "0:08:42", "remaining_time": "0:16:40", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 13, "total_steps": 35, "loss": 3.8904, "accuracy": 0.75, "learning_rate": 6.603581537171586e-07, "epoch": 0.36427320490367776, "percentage": 37.14, "elapsed_time": "0:09:20", "remaining_time": "0:15:48", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 14, "total_steps": 35, "loss": 2.6976, "accuracy": 0.75, "learning_rate": 6.304270437177064e-07, "epoch": 0.3922942206654991, "percentage": 40.0, "elapsed_time": "0:10:04", "remaining_time": "0:15:07", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 15, "total_steps": 35, "loss": 1.2788, "accuracy": 0.9375, "learning_rate": 5.985418945607484e-07, "epoch": 0.4203152364273205, "percentage": 42.86, "elapsed_time": "0:10:51", "remaining_time": "0:14:28", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 16, "total_steps": 35, "loss": 1.4116, "accuracy": 0.875, "learning_rate": 5.650298910241353e-07, "epoch": 0.44833625218914186, "percentage": 45.71, "elapsed_time": "0:11:32", "remaining_time": "0:13:42", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 17, "total_steps": 35, "loss": 1.6013, "accuracy": 0.9375, "learning_rate": 5.302349116131393e-07, "epoch": 0.4763572679509632, "percentage": 48.57, "elapsed_time": "0:12:09", "remaining_time": "0:12:52", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 18, "total_steps": 35, "loss": 1.8319, "accuracy": 0.875, "learning_rate": 4.945139999016476e-07, "epoch": 0.5043782837127846, "percentage": 51.43, "elapsed_time": "0:12:43", "remaining_time": "0:12:01", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 19, "total_steps": 35, "loss": 1.7772, "accuracy": 0.8125, "learning_rate": 4.5823370078193663e-07, "epoch": 0.532399299474606, "percentage": 54.29, "elapsed_time": "0:13:23", "remaining_time": "0:11:16", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 20, "total_steps": 35, "loss": 1.6103, "accuracy": 0.8125, "learning_rate": 4.217662992180634e-07, "epoch": 0.5604203152364273, "percentage": 57.14, "elapsed_time": "0:14:03", "remaining_time": "0:10:32", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 21, "total_steps": 35, "loss": 1.4708, "accuracy": 0.9375, "learning_rate": 3.8548600009835237e-07, "epoch": 0.5884413309982487, "percentage": 60.0, "elapsed_time": "0:14:44", "remaining_time": "0:09:49", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 22, "total_steps": 35, "loss": 1.1655, "accuracy": 1.0, "learning_rate": 3.4976508838686066e-07, "epoch": 0.6164623467600701, "percentage": 62.86, "elapsed_time": "0:15:32", "remaining_time": "0:09:11", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 23, "total_steps": 35, "loss": 1.2242, "accuracy": 0.9375, "learning_rate": 3.149701089758648e-07, "epoch": 0.6444833625218914, "percentage": 65.71, "elapsed_time": "0:16:13", "remaining_time": "0:08:28", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 24, "total_steps": 35, "loss": 1.4752, "accuracy": 0.9375, "learning_rate": 2.8145810543925163e-07, "epoch": 0.6725043782837128, "percentage": 68.57, "elapsed_time": "0:16:57", "remaining_time": "0:07:46", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 25, "total_steps": 35, "loss": 1.4732, "accuracy": 1.0, "learning_rate": 2.495729562822935e-07, "epoch": 0.7005253940455342, "percentage": 71.43, "elapsed_time": "0:17:39", "remaining_time": "0:07:03", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 26, "total_steps": 35, "loss": 1.2593, "accuracy": 0.8125, "learning_rate": 2.196418462828415e-07, "epoch": 0.7285464098073555, "percentage": 74.29, "elapsed_time": "0:18:24", "remaining_time": "0:06:22", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 27, "total_steps": 35, "loss": 1.1459, "accuracy": 0.8125, "learning_rate": 1.9197190913275294e-07, "epoch": 0.7565674255691769, "percentage": 77.14, "elapsed_time": "0:19:07", "remaining_time": "0:05:40", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 28, "total_steps": 35, "loss": 1.3002, "accuracy": 0.9375, "learning_rate": 1.6684707583059529e-07, "epoch": 0.7845884413309983, "percentage": 80.0, "elapsed_time": "0:19:48", "remaining_time": "0:04:57", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 29, "total_steps": 35, "loss": 0.8862, "accuracy": 0.9375, "learning_rate": 1.4452516116538054e-07, "epoch": 0.8126094570928196, "percentage": 82.86, "elapsed_time": "0:20:27", "remaining_time": "0:04:14", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 30, "total_steps": 35, "loss": 1.3672, "accuracy": 1.0, "learning_rate": 1.2523521818795044e-07, "epoch": 0.840630472854641, "percentage": 85.71, "elapsed_time": "0:21:14", "remaining_time": "0:03:32", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 31, "total_steps": 35, "loss": 1.3672, "accuracy": 0.875, "learning_rate": 1.0917518781671699e-07, "epoch": 0.8686514886164624, "percentage": 88.57, "elapsed_time": "0:21:58", "remaining_time": "0:02:50", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 32, "total_steps": 35, "loss": 1.6485, "accuracy": 0.9375, "learning_rate": 9.650986769598242e-08, "epoch": 0.8966725043782837, "percentage": 91.43, "elapsed_time": "0:22:44", "remaining_time": "0:02:07", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 33, "total_steps": 35, "loss": 1.3242, "accuracy": 0.9375, "learning_rate": 8.736922114910199e-08, "epoch": 0.9246935201401051, "percentage": 94.29, "elapsed_time": "0:23:28", "remaining_time": "0:01:25", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 34, "total_steps": 35, "loss": 0.8932, "accuracy": 0.9375, "learning_rate": 8.184704357891779e-08, "epoch": 0.9527145359019265, "percentage": 97.14, "elapsed_time": "0:24:10", "remaining_time": "0:00:42", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 35, "total_steps": 35, "loss": 1.4444, "accuracy": 0.9375, "learning_rate": 8e-08, "epoch": 0.9807355516637478, "percentage": 100.0, "elapsed_time": "0:24:50", "remaining_time": "0:00:00", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 35, "total_steps": 35, "eval_loss": 1.0162526369094849, "epoch": 0.9807355516637478, "percentage": 100.0, "elapsed_time": "0:26:02", "remaining_time": "0:00:00", "throughput": "0.00", "total_tokens": 0}
	{"current_steps": 35, "total_steps": 35, "epoch": 0.9807355516637478, "percentage": 100.0, "elapsed_time": "0:27:09", "remaining_time": "0:00:00", "throughput": "0.00", "total_tokens": 0}