QnA

AquilaX QnA (Securitron)

Overview

AquilaX QnA, aka Securitron, is a compact, instruction‑tuned transformer optimized for low‑latency CPU inference. It blends general-domain Q&A with precise AquilaX expertise, supporting real‑time streaming and minimal resource usage.

Model Specs

Name: AquilaX QnA (Securitron)
Architecture: Instruction‑tuned transformer
Fine‑Tuning: General conversational + AquilaX domain data
Context Window: 8192 tokens
Memory: ≥ 4 GB RAM (CPU)
Platforms: CPU (quantized) & CUDA GPU
API Access: Integrate via the.
Interactive Demo: Try the chatbot at Home.

Key Features

CPU-Optimized Performance:

Quantized for minimal memory usage and fast inference on CPUs.
No GPU required for efficient operation.

Dual Knowledge Base:

Handles general queries with clarity.
Delivers precise answers for AquilaX-specific topics.

Real-Time Streaming:

Supports token-by-token response generation for interactive experiences.

Context-Aware Responses:

Maintains a limited conversation history for coherent follow-ups.
Automatically manages history to optimize memory.

Custom System Prompt:

Configured as Securitron, ensuring professional and consistent responses.

Installation

Prerequisites

Python 3.8+
PyTorch (CPU or GPU version)
Transformers library
Optional: CUDA for GPU acceleration

Install Dependencies

pip install torch transformers

Download the Model

Load the model and tokenizer directly from Hugging Face:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("AquilaX-AI/QnA")
model = AutoModelForCausalLM.from_pretrained("AquilaX-AI/QnA")

Inference Example

The following code demonstrates how to perform inference with the AquilaX QnA model:

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("AquilaX-AI/QnA")
model = AutoModelForCausalLM.from_pretrained("AquilaX-AI/QnA")

# System prompt
prompt = "<|im_start|>system\nYou are Securitron, a helpful AI assistant.<|im_end|>"

# Initialize history
history = []

# Set device
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.to(device)

while True:
    user_input = input("\nUser Question: ")
    if user_input.lower() == 'break':
        break

    # Format user input
    user = f"<|im_start|>user\n{user_input}<|im_end|>\n<|im_start|>assistant"
    history.append(user)
    history = history[-5:]  # Keep last 5 exchanges

    # Build prompt
    full_prompt = prompt + "\n".join(history)
    inputs = tokenizer(full_prompt, return_tensors="pt", truncation=True).input_ids.to(device)

    # Stream response
    streamer = TextStreamer(tokenizer, skip_prompt=True)
    response = model.generate(
        input_ids=inputs,
        streamer=streamer,
        max_new_tokens=512,
        use_cache=True,
        pad_token_id=151645,
        eos_token_id=151645,
        num_return_sequences=1
    )

    # Update history
    decoded = tokenizer.decode(response[0]).split('<|im_start|>assistant')[-1].split('<|im_end|>')[0].strip()
    history.append(decoded + "<|im_end|>")

Key Notes

Device: Automatically uses GPU if available; defaults to CPU.
Streaming: TextStreamer enables real-time response display.
History: Limits to 5 exchanges to optimize memory.

Input and Output Format

Input Format

<|im_start|>system
You are Securitron, a helpful AI assistant.
<|im_end|>
<|im_start|>user
{user_question}
<|im_end|>
<|im_start|>assistant

Output Format

<|im_start|>assistant
{generated_response}
<|im_end|>

Responses are streamed in real-time with TextStreamer.
Cleaned output is plain text for user display.

Performance Optimization

CPU Efficiency: Quantized model ensures low memory usage.
History Management: Limit to 5 exchanges to reduce memory overhead.
GPU Support: Enable CUDA for faster inference if available.
Response Length: Adjust max_new_tokens for shorter or longer outputs.

Deployment Considerations

Environment: Use a virtual environment to manage dependencies.
Error Handling: Add try-catch for robust error management.

Support and Contributions

PreviousQuery NextSecurity Assistant

Last updated 22 days ago

Was this helpful?

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer import torch # Load model and tokenizer tokenizer = AutoTokenizer.from_pretrained("AquilaX-AI/QnA") model = AutoModelForCausalLM.from_pretrained("AquilaX-AI/QnA") # System prompt prompt = "<|im_start|>system\nYou are Securitron, a helpful AI assistant.<|im_end|>" # Initialize history history = [] # Set device device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") model.to(device) while True: user_input = input("\nUser Question: ") if user_input.lower() == 'break': break # Format user input user = f"<|im_start|>user\n{user_input}<|im_end|>\n<|im_start|>assistant" history.append(user) history = history[-5:] # Keep last 5 exchanges # Build prompt full_prompt = prompt + "\n".join(history) inputs = tokenizer(full_prompt, return_tensors="pt", truncation=True).input_ids.to(device) # Stream response streamer = TextStreamer(tokenizer, skip_prompt=True) response = model.generate( input_ids=inputs, streamer=streamer, max_new_tokens=512, use_cache=True, pad_token_id=151645, eos_token_id=151645, num_return_sequences=1 ) # Update history decoded = tokenizer.decode(response[0]).split('<|im_start|>assistant')[-1].split('<|im_end|>')[0].strip() history.append(decoded + "<|im_end|>")