KT313
/

Llama-3-8B-RAG-v1-exl2-6.0bpw

@@ -1,101 +1,45 @@
----
-license: llama3
-datasets:
-- glaiveai/RAG-v1
-language:
-- en
-tags:
-- RAG
----
-# Llama-3-8B-RAG-v1
-This model is a fine-tuned version of Llama-3-8B for Retrieval-Augmented Generation (RAG) tasks on the Glaive-RAG-v1 dataset.
-## Model Details
-* **Developed by:** [Glaive](https://glaive.ai)
-* **Model type:** Llama-3 finetuned for RAG tasks
-## How to Get Started with the Model
-Use the code below to get started with the model:
-NOTE: Try to use the same system prompt and document formatting as the example provided below, this is the same format that was used to finetune the model.
 ```python
-from transformers import AutoTokenizer, pipeline
-model_name = "glaiveai/Llama-3-8B-RAG-v1"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-# Example user query
-user_query = """Document:0
-Title: Financial Compliance and Company Statements
-Text: This document provides a detailed overview of the financial compliance measures and statements issued by corporations. It discusses various aspects of financial reporting, the importance of accurate financial disclosures, and the role of chief executive officers in certifying financial documents. The document elaborates on the legal frameworks that govern corporate financial disclosures, including the Securities Exchange Act of 1934, which mandates that public companies must provide periodic reports detailing their financial status. These reports are crucial for investors, regulators, and other stakeholders who rely on them to make informed decisions. The document also highlights the penalties for non-compliance with financial reporting requirements, which can include fines and criminal charges for executives who misreport financial information.
-Document:1
-Title: Certification of Financial Reports by CEO of Black Knight, Inc.
-Text: Pursuant to 18 U.S.C. §1350, the Chief Executive Officer of Black Knight, Inc., a corporation registered in Delaware, has officially certified the periodic financial report containing the company's financial statements. The certification confirms that these reports comply fully with the requirements of Section 13(a) or 15(d) of the Securities Exchange Act of 1934. The CEO, Anthony M. Jabbour, asserts that the information contained within these reports fairly presents, in all material respects, the financial condition and results of operations of Black Knight, Inc. The certification is part of the official documents filed with the Securities and Exchange Commission and serves as a testament to the accuracy and fairness of the financial disclosures made by the company. This certification is crucial as it assures stakeholders of the reliability of the financial statements provided by Black Knight, Inc.
-Answer Mode: Grounded
-Question: How does the CEO of Black Knight, Inc. ensure compliance with the Securities Exchange Act of 1934, and what are the implications of the certification provided?"""
-# Prepare chat template
-chat = [
-   {"role": "system", "content": "You are a conversational AI assistant that is provided a list of documents and a user query to answer based on information from the documents. The user also provides an answer mode which can be 'Grounded' or 'Mixed'. For answer mode Grounded only respond with exact facts from documents, for answer mode Mixed answer using facts from documents and your own knowledge. Cite all facts from the documents using <co: doc_id></co> tags."},
-   {"role": "user", "content": user_query}
-]
-prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
-# Set up pipeline and generate response
-pipe = pipeline('text-generation', model=model_name, tokenizer=model_name, device=0)
-output = pipe(
-    prompt,
-    max_length=1000,
-    num_return_sequences=1,
-    top_k=50,
-    top_p=0.95,
-    temperature=0.5,
-    do_sample=True,
-    return_full_text=False
 )
-print(output[0]['generated_text'])
-```
-## Output of above code block
 ```
-Cited Documents: 1
-Answer: <co:1>The CEO of Black Knight, Inc., Anthony M. Jabbour, ensures compliance with the Securities Exchange Act of 1934 by certifying that the periodic financial report and financial statements comply fully with the requirements of Section 13(a) or 15(d) of the Act. This certification confirms that the information in the reports fairly presents, in all material respects, the financial condition and results of operations of Black Knight, Inc.</co> This certification is crucial as it assures stakeholders of the reliability of the financial statements provided by the company, thereby maintaining investor confidence and adherence to legal financial reporting standards.
-In this output:
-The model cites Document 1 as the source of its information.
-It provides a grounded answer based on the content of Document 1, as requested in the "Answer Mode: Grounded" instruction.
-The answer explains how the CEO ensures compliance and the implications of the certification.
-The citation is marked with <co:1></co> tags, indicating that this information comes directly from Document 1.
-```
-## Answer mode and citations
-The model can be instructed to answer only using grounded information from the documents, or to combine it's internal knowledge as well by using Answer Mode: Grounded/Mixed in the user prompt.
-Citations are generated at the beginning of each answer, and also present within the text in <co: doc_id> </co> blocks.
-## Future work
-We are working on the following improvements for the next version of the dataset:
-- Multi turn data
-- Diverse document sources

+exllamav2-quantized version of Llama-3-8B-RAG-v1 from glaiveai: https://huggingface.co/glaiveai/Llama-3-8B-RAG-v1
+bpw: 6.0
+head-bpw: 8.0
+example usage with exllamav2:
 ```python
+from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
+from exllamav2.generator import ExLlamaV2Sampler, ExLlamaV2DynamicGenerator
+model_path = /path/to/model_folder
+config = ExLlamaV2Config(model_path)
+model = ExLlamaV2(config)
+cache = ExLlamaV2Cache(model, max_seq_len = 4096, lazy = True)
+model.load_autosplit(cache, progress = True)
+tokenizer = ExLlamaV2Tokenizer(config)
+generator = ExLlamaV2DynamicGenerator(
+    model = model,
+    cache = cache,
+    tokenizer = tokenizer,
+)
+gen_settings = ExLlamaV2Sampler.Settings(
+    temperature = 1.0,
+    top_p = 0.1,
+    token_repetition_penalty = 1.0
+)
+outputs = generator.generate(
+    prompt = ["first input", "second input"], # string or list of strings
+    max_new_tokens = 1024,
+    stop_conditions = [tokenizer.eos_token_id],
+    gen_settings = gen_settings,
+    add_bos = True,
 )
+print(outputs)
 ```

output.safetensors → model.safetensors RENAMED Viewed

File without changes