Embeddings-Collab/Llama_3.1_8b_aside

Llama 3.1 8B ASIDE

This is Llama 3.1 8B model augmented and fine-tuned with the ASIDE modification introduced in the paper ASIDE: Architectural Separation of Instructions and Data in Language Models.

Model Description

ASIDE applies an orthogonal rotation to the embeddings of data tokens, thus creating clearly distinct representations of instructions and data tokens without introducing any additional parameters.

Usage

To use this model, first clone and follow the installation instructions in the official ASIDE Repository.

Inside the repository, run the following code snippet (also provided here as a script) to do inference with this model.

import torch
import deepspeed
import json
import os
from huggingface_hub import login

from model_api import CustomModelHandler  # Import your custom handler
from model_api import format_prompt  # Import your prompt formatting function

# Define your instruction and data
instruction_text = "Translate to German."
data_text = "Who is Albert Einstein?"

# Model configuration
hf_token = os.environ["HUGGINGFACE_HUB_TOKEN"]
login(token=hf_token)
embedding_type = "forward_rot"  
base_model =  "meta-llama/Llama-3.1-8B" 
model_path = "Embeddings-Collab/llama_3.1_8b_forward_rot_emb_SFTv110_from_base_run_15_fix"

# Initialize the model handler
handler = CustomModelHandler(
    model_path, 
    base_model, 
    base_model, 
    model_path, 
    None,
    0, 
    embedding_type=embedding_type, 
    load_from_checkpoint=True
)

# Initialize DeepSpeed inference engine
engine = deepspeed.init_inference(
    model=handler.model,
    mp_size=torch.cuda.device_count(),  # Number of GPUs
    dtype=torch.float16,
    replace_method='auto',
    replace_with_kernel_inject=False
)
handler.model = engine.module

# Load prompt templates
with open("./data/prompt_templates.json", "r") as f:
    templates = json.load(f)

template = templates[0]  
instruction_text = format_prompt(instruction_text, template, "system")
data_text = format_prompt(data_text, template, "user")

# Generate output
output, inp = handler.call_model_api_batch([instruction_text], [data_text])
print(output)

Citation

If you use this model, please cite our paper:

@inproceedings{
  zverev2026aside,
  title={{ASIDE}: Architectural Separation of Instructions and Data in Language Models},
  author={Egor Zverev and Evgenii Kortukov and Alexander Panfilov and Alexandra Volkova and Rush Tabesh and Sebastian Lapuschkin and Wojciech Samek and Christoph H. Lampert},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=C81TnwHiRM}
}