The Lernex engine with its speeds, and direct-mode sampling in generation_config.json

Lernex quickstart with decode speeds against vLLM on an RTX 5070 Ti, the vLLM speeds remeasured on the same workload. llama.cpp's whole-context penalty window as 131072 (it rejects -1).

README.md

@@ -270,8 +270,8 @@ at higher precision.
 
 | You have | Use | With | Size |
 |---|---|---|---|
-| A GPU with 16 GB or more | This repository, BF16 | Transformers, vLLM, SGLang | 6.2 GB |
-| A smaller GPU | [Metis-1.6-Praxis-FP8](https://huggingface.co/Lernex/Metis-1.6-Praxis-FP8) | vLLM, SGLang | 3.5 GB |
+| A GPU with 16 GB or more | This repository, BF16 | Lernex, Transformers, vLLM, SGLang | 6.2 GB |
+| A smaller GPU | [Metis-1.6-Praxis-FP8](https://huggingface.co/Lernex/Metis-1.6-Praxis-FP8) | Lernex, vLLM, SGLang | 3.5 GB |
 | A Mac with Apple silicon | [Metis-1.6-Praxis-MLX-4bit](https://huggingface.co/Lernex/Metis-1.6-Praxis-MLX-4bit) or [-8bit](https://huggingface.co/Lernex/Metis-1.6-Praxis-MLX-8bit) | mlx-lm | 2.0 / 3.2 GB |
 | A laptop, a CPU, anything else | [Metis-1.6-Praxis-GGUF](https://huggingface.co/Lernex/Metis-1.6-Praxis-GGUF), Q4_K_M | llama.cpp | 2.0 GB |
 
@@ -300,13 +300,46 @@ with the request. The commands below pass the settings where an engine does not
   and keywords on purpose, and the penalty works against them: it cost 12 points on HumanEval+.
 - **Keep penalties off in direct mode.** Direct answers are short and rarely repeat, so a penalty
   only costs accuracy.
-- **In llama.cpp and Ollama, apply the penalty to the whole context** with `--repeat-last-n -1` (or
-  `repeat_last_n -1`). By default they look at the last 64 tokens only.
+- **In llama.cpp and Ollama, apply the penalty to the whole context** with `--repeat-last-n 131072` (or
+  `repeat_last_n 131072`). By default they look at the last 64 tokens only, and they reject `-1`.
 - **If your engine has no repetition penalty, use temperature 1.0** for thinking.
 
 I chose these by sweeping 13 thinking and 8 direct settings over MATH-500, GPQA Diamond, HumanEval+
 and IFEval, then comparing the finalists blind on 32 everyday conversations.
 
+### Lernex
+
+[Lernex](https://github.com/lernex/lernex-engine) is the inference engine I wrote for Metis, from
+scratch, and the fastest way to run Praxis.
+
+```bash
+git clone https://github.com/lernex/lernex-engine && cd lernex-engine
+cmake -B build -DLX_ARCHS=all   # NVIDIA from the RTX 30 series on; a Mac builds for Metal
+cmake --build build -j
+./build/lernex run praxis    # downloads the model, then chats in the terminal
+./build/lernex serve praxis  # an OpenAI-compatible API on port 8080
+```
+
+It needs CMake 3.28 and, for NVIDIA, the CUDA 13 toolkit. With no NVIDIA GPU, add
+`-DLX_BACKEND_CUDA=OFF` for the CPU build. Decode speed on an RTX 5070 Ti in tokens per second,
+against vLLM on the same files, with 512 prompt tokens and 256 generated per stream:
+
+| | Lernex | vLLM |
+|---|---|---|
+| BF16, one stream | **294.5** | 204.2 |
+| BF16, 32 streams | **2,141** | 1,871 |
+| FP8, one stream | **295.1** | 235.3 |
+| FP8, 32 streams | **2,489** | 2,268 |
+
+In BF16, Lernex decodes from a lossless compressed copy of the weights: it reads fewer bytes for
+every token with nothing lost, which is why BF16 runs as fast as FP8 for one stream.
+
+`lernex serve` speaks the same API as vLLM below: the thinking comes back as `reasoning_content`,
+tool calls as typed `tool_calls`, and `"chat_template_kwargs": {"enable_thinking": false}` gives a
+direct answer. Lernex reads the sampling settings above from the model and switches to the direct
+column by itself, so you never pass them. Weights run in the format the file stores, never
+quantized behind your back: `--weights int8` or `--weights fp8` picks a narrower one, and
+`lernex doctor praxis` shows every format, what fits on your machine and the speed to expect.
 ### Transformers
 
 The Transformers implementation ships in this repository, so the model loads with
@@ -330,7 +363,7 @@ print(answer.strip())
 ```
 
 This implementation is the reference: it computes exactly what the native model computes, but it is
-not the fastest way to run it. For serving, use one of the engines below.
+not the fastest way to run it. For serving, use Lernex or one of the engines below.
 
 ### vLLM
 
@@ -343,10 +376,10 @@ vllm serve Lernex/Metis-1.6-Praxis --trust-remote-code \
 [`vllm-metis`](https://github.com/lernex/vllm-metis) registers the model, a reasoning parser and a
 tool-call parser through vLLM's plugin system, so there is nothing to patch. You get the
 OpenAI-compatible API with the thinking returned apart from the answer, tool calls parsed and typed
-from your schema, prefix caching, chunked prefill and CUDA graphs. Praxis runs in BF16 on a 16 GB card: an RTX 5070 Ti decodes 201 tokens/s for a single stream
-and 1,836 tokens/s across 32. On a smaller card, serve
-[Metis-1.6-Praxis-FP8](https://huggingface.co/Lernex/Metis-1.6-Praxis-FP8) (3.5 GB), which is also
-the fastest option: 231 and 2,386 tokens/s.
+from your schema, prefix caching, chunked prefill and CUDA graphs. Praxis runs in BF16 on a 16 GB card: an RTX 5070 Ti decodes 204 tokens/s for a single stream
+and 1,871 tokens/s across 32. On a smaller card, serve
+[Metis-1.6-Praxis-FP8](https://huggingface.co/Lernex/Metis-1.6-Praxis-FP8) (3.5 GB), which vLLM
+also runs faster: 235 and 2,268 tokens/s.
 
 ```python
 from openai import OpenAI
@@ -391,7 +424,7 @@ git fetch origin pull/31134/head:metis16 && git checkout metis16
 cmake -B build                  # add -DGGML_CUDA=ON for NVIDIA; Metal is on by default on a Mac
 cmake --build build --config Release -j
 ./build/bin/llama-server -hf Lernex/Metis-1.6-Praxis-GGUF:Q4_K_M --jinja -ngl 99 -c 32768 \
-    --temp 0.8 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 --repeat-last-n -1
+    --temp 0.8 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 --repeat-last-n 131072
 ```
 
 `--jinja` uses the chat template inside the file, and the server returns the thinking in
@@ -433,9 +466,9 @@ user message and keeps it within an ongoing tool loop, as the model was trained.
 
 ### Tool calling
 
-Pass JSON-schema tools with `tools=`. The model writes each call after `<|tool_call|>`; the vLLM,
-SGLang and llama.cpp servers parse these into typed `tool_calls` for you. With Transformers, decode
-with `skip_special_tokens=False` to find them:
+Pass JSON-schema tools with `tools=`. The model writes each call after `<|tool_call|>`; the Lernex,
+vLLM, SGLang and llama.cpp servers parse these into typed `tool_calls` for you. With Transformers,
+decode with `skip_special_tokens=False` to find them:
 
 ```python
 tools = [{

generation_config.json

@@ -9,5 +9,9 @@
   "temperature": 0.8,
   "top_p": 0.95,
   "top_k": 20,
-  "repetition_penalty": 1.05
+  "repetition_penalty": 1.05,
+  "direct": {
+    "temperature": 0.3,
+    "repetition_penalty": 1.0
+  }
 }