Bubba is a fine-tuned LLM based on OpenAI’s Chat GPT-5. This release packages the fine-tuned weights (or adapters) for practical, low-latency instruction following, summarization, reasoning, and light code generation. It is intended for local or self-hosted environments and RAG (Retrieval-Augmented Generation) stacks that require predictable, fast outputs.

Quantized, and fine-tuned GGUF based on OpenAI’s gpt-oss-20b
Format: GGUF (for llama.cpp and compatible runtimes) • Quantization: Q4_K_XL (4-bit, K-grouped, extra-low loss)
File: bubba-20b-Q4_K_XL.gguf

⚙️ Streamlit UI

🐍 Code Repository

🧠 Overview

• This repo provides a 4-bit K-quantized .gguf for fast local inference of a 20B-parameter model derived from OpenAI’s gpt-oss-20b (as reported by the uploader).
• Use cases: general chat/instruction following, coding help, knowledge Q&A (see Intended Use & Limitations).
• Works with: llama.cpp, llama-cpp-python, KoboldCPP, Text Generation WebUI, LM Studio, and other GGUF-compatible backends.
• Hardware guidance (rule of thumb): ~12–16 GB VRAM/RAM for comfortable batch-1 inference with Q4_K_XL; CPU-only works too (expect lower tokens/s).

---

Key Features

• Instruction-tuned derivative of gpt-oss-20b for concise, helpful responses.
• Optimized defaults for short to medium prompts; strong compatibility with RAG pipelines.
• Flexible distribution: full finetuned weights or lightweight LoRA/QLoRA adapters.
• Compatible with popular runtimes and libraries (Transformers, PEFT, vLLM, Text Generation Inference).

⚠️ Provenance & license: This quant is produced from a base model claimed to be OpenAI’s gpt-oss-20b. Please review and comply with the original model’s license/terms. The GGUF quantization inherits those terms. See the License section.

⚙️ Fine-Tuning Datasets

File Name	Description
Balanced Budget and Emergency Deficit Control Act of 1985	Establishes statutory limits on federal spending and deficit control mechanisms, including sequestration procedures.
Budget Control Act of 2011	Sets discretionary spending caps and establishes enforcement mechanisms to control federal deficits.
Digital Accountability And Transparency Act of 2014	Requires standardized federal spending data and improved transparency through government-wide financial reporting.
Federal Account Symbols And Titles Book	Defines Treasury account symbols and official titles used for federal budgetary and accounting purposes.
Federal Acquisition Regulation	Establishes uniform policies and procedures governing the acquisition of goods and services by federal agencies.
Federal Government Standards For Internal Controls	Defines the internal control framework for federal agencies to ensure accountability, integrity, and compliance.
Federal Managers Financial Integrity Act of 1982	Requires agencies to establish internal controls and report annually on their effectiveness.
Federal Trust Fund Accounting Guide	Provides accounting guidance for the management and reporting of federal trust funds.
Financial Management Regulations DOD 7000-14-R	Establishes DoD-specific financial management policies, procedures, and accounting requirements.
Fiscal Responsibility Act	Establishes statutory measures intended to improve fiscal discipline and control federal spending.
Government Auditing Standards	Sets professional standards for audits of government organizations, programs, activities, and functions.
Government Invoicing User Guide	Provides guidance on federal invoicing standards and processes for government transactions.
Government Performance and Results Act of 1993	Requires agencies to engage in strategic planning and performance measurement to improve program effectiveness.
GPRA Modernization Act of 2010	Updates GPRA by strengthening performance management, cross-agency goals, and accountability.
OMB Circular A-11 Preparation Submission And Execution Of The Budget	Provides comprehensive guidance for preparing, submitting, and executing the President’s Budget.
OMB Circular A-11 Section 120 Apportionment Process	Defines the apportionment process used to control the rate of obligation of budgetary resources.
OMB Circular A-123 Managements Responsibility for Enterprise Risk Management and Internal Control	Defines management responsibilities for internal control and enterprise risk management across federal agencies.
Federal Trust Fund Accounting Guide	Establishes requirements for federal agency financial statements and reporting.
Principles Of Federal Appropriations Law Volume One	Authoritative GAO guidance on foundational principles governing the use of federal appropriations.
Statements of Federal Federal Financial Accounting Concepts and Standards	Establishes accounting concepts and standards for federal financial reporting.
The Anti-Deficiency Act PL 97-258	Prohibits federal agencies from obligating or expending funds in excess of appropriations or before enactment.
The Anti-Deficiency Reform and Enforcement Act of 2018	Strengthens Anti-Deficiency Act enforcement and reporting requirements to improve fiscal accountability.
The Chief Financial Officers Act of 1990	Establishes agency Chief Financial Officers and modernizes federal financial management practices.
The Congressional Budget and Impoundment Control Act of 1974	Establishes the congressional budget process and restricts executive impoundment of appropriated funds.
Statutory Pay As You Go Act of 2010	Authorizes interagency agreements for the provision of goods and services on a reimbursable basis.
The Stafford Act	Provides the statutory framework for federal disaster response and emergency assistance.
Federal Trust Fund Accounting Guide	Provides additional appropriations authority beyond regular annual funding acts.
Title 2 Code of Federal Regulations – Uniform Administrative Requirements, Cost Principles, and Audit	Establishes uniform administrative, cost, and audit requirements for federal financial assistance.
Title 31 Code of Federal Regulations – Money and Finance	Codifies Treasury and federal financial management regulations governing money and finance.
US Standard General Ledger Account Definitions	Defines standardized account structures used for federal accounting and financial reporting.

Technical Specifications

Property	Value / Guidance
Base model	gpt-oss-20b (decoder-only Transformer)
Parameters	~20B (as per upstream)
Tokenizer	Use the upstream tokenizer associated with gpt-oss-20b
Context window	Determined by the upstream base; set accordingly in your runtime
Fine-tuning	Supervised Fine-Tuning (SFT); optional preference optimization (DPO/ORPO)
Precision	FP16/BF16 recommended; 4-bit (bnb) for single-GPU experimentation
Intended runtimes	Hugging Face Transformers, PEFT, vLLM, TGI (Text Generation Inference)

Note: Please adjust any specifics (context length, tokenizer name) to match the exact upstream build you use for gpt-oss-20b.

---

Files

File / Folder	Description
README.md	This model card
config.json / tokenizer files	Configuration and tokenizer artifacts (from upstream)
pytorch_model.safetensors	Full fine-tuned weights (if released as full model)
adapter_model.safetensors	LoRA/QLoRA adapters only (if released as adapters)
training_args.json (optional)	Minimal training configuration for reproducibility

Only one of “full weights” or “adapters” may be included depending on how you distribute Bubba.

---

📝 Intended Use & Limitations

Intended Use

• Instruction following, general dialogue
• Code assistance (reasoning, boilerplate, refactoring)
• Knowledge/Q&A within the model’s training cutoff

Out-of-Scope / Known Limitations

• Factuality: may produce inaccurate or outdated info
• Safety: can emit biased or unsafe text; apply your own filters/guardrails
• High-stakes decisions: not for medical, legal, financial, or safety-critical use

🎯 Quick Start

Examples: Using the Bubba LLM (Fine-tuned from gpt-oss-20b)

This guide shows several ways to run Bubba locally or on a server. Examples cover full weights, LoRA/QLoRA adapters, vLLM, and Text Generation Inference (TGI), plus prompt patterns and RAG.

---

🐍 Python (Transformers) — Full Weights

Install

pip install "transformers>=4.44.0" accelerate torch --upgrade

Load and generate

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "your-namespace/gpt-oss-20b-GGUF-finetuned"
tok = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "In 5 bullet points, explain retrieval-augmented generation and when to use it."
inputs = tok(prompt, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9
)
print(tok.decode(out[0], skip_special_tokens=True))

Notes

• device_map="auto" will place weights across available GPUs/CPU.
• Prefer BF16 if supported; otherwise FP16. For VRAM-constrained experiments, see 4-bit below.

---

🧩 Python (PEFT) — Adapters on Top of the Base

Install

pip install "transformers>=4.44.0" peft accelerate torch --upgrade

Load base + LoRA/QLoRA adapters

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_name = "gpt-oss-20b-GGUF"   # replace with the exact upstream base you use
lora_name = "your-namespace/gpt-oss-20b-GGUF-finetuned"

tok = AutoTokenizer.from_pretrained(base_name, use_fast=True)
base = AutoModelForCausalLM.from_pretrained(
    base_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base, lora_name)

prompt = "Draft a JSON spec with keys: goal, steps[], risks[], success_metric."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, temperature=0.6, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

---

💾 4-bit (bitsandbytes) — Memory-Efficient Loading

Install

pip install "transformers>=4.44.0" accelerate bitsandbytes --upgrade

Load with 4-bit quantization

    from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
    import torch

    bnb = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
    )

    model_name = "your-namespace/gpt-oss-20b-GGUF-finetuned"
    tok = AutoTokenizer.from_pretrained(model_name, use_fast=True)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        quantization_config=bnb,
        device_map="auto"
    )

    prompt = "Explain beam search vs. nucleus sampling in three short bullets."
    inputs = tok(prompt, return_tensors="pt").to("cuda" if torch.cuda.is_available() else "cpu")
    out = model.generate(**inputs, max_new_tokens=160, temperature=0.7, top_p=0.9)
    print(tok.decode(out[0], skip_special_tokens=True))

---

🚀 Serve with vLLM (OpenAI-compatible API)

Install and launch (example)

    pip install vllm
    python -m vllm.entrypoints.openai.api_server \
      --model your-namespace/Bubba-gpt-oss-20b-finetuned \
      --dtype bfloat16 --max-model-len 8192 \
      --port 8000

Call the endpoint (Python)

    import requests, json
    url = "http://localhost:8000/v1/chat/completions"
    headers = {"Content-Type": "application/json"}

    data = {
      "model": "your-namespace/gpt-oss-20b-GGUF-finetuned",
      "messages": [
        {"role": "system", "content": "You are concise and factual."},
        {"role": "user", "content": "Give a 4-step checklist for evaluating a RAG pipeline."}
      ],
      "temperature": 0.7,
      "max_tokens": 256,
      "stream": True
    }

    with requests.post(url, headers=headers, data=json.dumps(data), stream=True) as r:
        for line in r.iter_lines():
            if line and line.startswith(b"data: "):
                chunk = line[len(b"data: "):].decode("utf-8")
                if chunk == "[DONE]":
                    break
                print(chunk, flush=True)

---

📦 Serve with Text Generation Inference (TGI)

Run the server (Docker)

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e MODEL_ID=your-namespace/Bubba-gpt-oss-20b-finetuned \
  ghcr.io/huggingface/text-generation-inference:latest

Call the server (HTTP)

curl http://localhost:8080/generate \
  -X POST -d '{
    "inputs": "Summarize pros/cons of hybrid search (BM25 + embeddings).",
    "parameters": {"max_new_tokens": 200, "temperature": 0.7, "top_p": 0.9}
  }' \
  -H "Content-Type: application/json"

---

🧠 Prompt Patterns

Direct instruction (concise)

You are a precise assistant. In 6 bullets, explain evaluation metrics for retrieval (Recall@k,
MRR, nDCG). Keep each bullet under 20 words.

Constrained JSON output

System: Output only valid JSON. No prose.
User: Produce {"goal":"", "steps":[""], "risks":[""], "metrics":[""]} for testing a QA bot.

Guarded answer

If the answer isn’t derivable from the context, say “I don’t know” and ask for the missing info.

Few-shot structure

Example:
Q: Map 3 tasks to suitable embedding dimensions.
A: 256: short titles; 768: support FAQs; 1024: multi-paragraph knowledge base.

---

📚 Basic RAG

# 1) Retrieve
chunks = retriever.search("compare vector DBs for legal discovery", k=5)

# 2) Build prompt
context = "\n".join([f"• {c.text} [{c.source}]" for c in chunks])
prompt = f"""
You are a helpful assistant. Use only the context to answer.
Context:
{context}

Question:
What selection criteria should teams use when picking a vector DB for scale and cost?
"""

# 3) Generate (Transformers / vLLM / TGI)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, temperature=0.6, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

📁 1. Document Ingestion

from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = TextLoader("docs/corpus.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=700, chunk_overlap=150)
docs = splitter.split_documents(documents)

---

🔍 2. Embedding & Vector Indexing

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS

embedding = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(docs, embedding)

---

🔄 3. Retrieval + Prompt Formatting

retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
retrieved_docs = retriever.get_relevant_documents("What role does Bubba play in improving document QA?")

context = "\n\n".join([doc.page_content for doc in retrieved_docs])

prompt = f"""
You are Bubba, a reasoning-heavy assistant. Use only the context below to answer:

<context>
{context}
</context>

<question>
What role does Bubba play in improving document QA?
</question>
"""

---

🧠 4. LLM Inference with Bubba

./main -m bubba/gpt-oss-20b-GGUF -p "$prompt" -n 768 -t 16 -c 4096 --color

Bubba’s output will include a context-aware, citation-grounded response backed by the retrieved input.

---

📝 Notes

• Bubba (20B parameter model) may require more memory than smaller models like Bro or Leeroy.
• Use a higher -c value (context size) to accommodate longer prompts with more chunks.
• GPU acceleration is recommended for smooth generation if your hardware supports it.

---

⚙️ Parameter Tips

• Temperature: 0.6–0.9 (lower = more deterministic)
• Top-p: 0.8–0.95 (tune one knob at a time)
• Max new tokens: 128–384 for chat; longer for drafting
• Repetition penalty: 1.05–1.2 if loops appear
• Batch size: use padding_side="left" and dynamic padding for throughput
• Context length: set to your runtime’s max; compress context via selective retrieval

---

🛟 Troubleshooting

• CUDA OOM:
  Lower max_new_tokens; enable 4-bit; shard across GPUs; reduce context length.
• Slow throughput:
  Use vLLM/TGI with tensor/PP sharding; enable paged attention; pin to BF16.
• Messy JSON:
  Use a JSON-only system prompt; set temperature ≤0.6; add a JSON schema in the prompt.
• Domain shift:
  Consider small adapter tuning on your domain data; add retrieval grounding.

---

🔍 Minimal Batch Inference Example

prompts = [
  "List 5 key features of FAISS.",
  "Why would I choose pgvector over Milvus?"
]
inputs = tok(prompts, return_tensors="pt", padding=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=160, temperature=0.7, top_p=0.9)
for i, seq in enumerate(out):
    print(f"--- Prompt {i+1} ---")
    print(tok.decode(seq, skip_special_tokens=True))

---

Inference Tips

• Prefer BF16 if available; otherwise FP16. For limited VRAM, try 4-bit (bitsandbytes) to explore.
• Start with max_new_tokens between 128–384 and temperature 0.6–0.9; tune top_p for stability.
• For RAG, constrain prompt length and adopt strict chunking/citation formatting for better grounding.

---

📘 WebUI

• Place the GGUF in text-generation-webui/models/bubba/gpt-20b-Q4_K_XL/
• Launch with the llama.cpp loader (or llama-cpp-python backend)
• Select the model in the UI, adjust context length, GPU layers, and sampling

🧩 KoboldCPP

./koboldcpp \
  -m bubba\gpt-oss-20b-Q4_K_XL.gguf \
  --contextsize 4096 \
  --gpulayers 35 \
  --usecublas

⚡ LM Studio

1. Open LM Studio → Models → Local models → Add local model and select the .gguf.
2. In Chat, pick the model, set Context length (≤ base model max), and adjust GPU Layers.
3. For API use, enable Local Server and target the exposed endpoint with OpenAI-compatible clients.

❓ Prompting

This build is instruction-tuned (downstream behavior depends on your base). Common prompt patterns work:

Simple instruction

Write a concise summary of the benefits of grouped 4-bit quantization.

ChatML-like

<|system|>
You are a helpful, concise assistant.
<|user|>
Compare Q4_K_XL vs Q5_K_M in terms of quality and RAM.
<|assistant|>

Code task

Task: Write a Python function that computes perplexity given log-likelihoods.
Constraints: Include docstrings and type hints.

Tip: Keep prompts explicit and structured (roles, constraints, examples).
Suggested starting points: temperature 0.2–0.8, top_p 0.8–0.95, repeat_penalty 1.05–1.15.

• No special chat template is strictly required. Use clear instructions and keep prompts concise. For multi-turn workflows, persist conversation state externally or via your app’s memory/RAG layer.

Example system style

You are a concise, accurate assistant. Prefer step-by-step reasoning only when needed.
Cite assumptions and ask for missing constraints.

• Guro is a prompt library designed to supercharge AI agents and assistants with task-specific personas -ie, total randos.
• From academic writing to financial analysis, technical support, SEO, and beyond
• Guro provides precision-crafted prompt templates ready to drop into your LLM workflows.

⚙️ Performance & Memory Guidance (Rules of Thumb)

• RAM/VRAM for Q4_K_XL (20B): ~12–16 GB for batch-1 inference (varies by backend and offloading).
• Throughput: Highly dependent on CPU/GPU, backend, context length, and GPU offload.
  Start with -ngl as high as your VRAM allows, then tune threads/batch sizes.
• Context window: Do not exceed the base model’s maximum (quantization does not increase it).

💻 Files

• gpt-oss-20b-Q4_K_M.gguf — 4-bit K-quantized weights (XL variant)
• tokenizer.* — packed inside GGUF (no separate files needed)

Integrity: Verify your download (e.g., SHA256) if provided by the host/mirror.

⚙️ GGUF Format

1. Start from the base gpt-oss-20b weights (FP16/BF16).
2. Convert to GGUF with llama.cpp’s convert tooling (or equivalent for the base arch).
3. Quantize with llama.cpp quantize to Q4_K_XL.
4. Sanity-check perplexity/behavior, package with metadata.

Exact scripts/commits may vary by environment; please share your pipeline for full reproducibility if you fork this card.

🏁 Safety, Bias & Responsible Use

Large language models can generate plausible but incorrect or harmful content and may reflect societal biases. If you deploy this model:

• Add moderation/guardrails and domain-specific filters.
• Provide user disclaimers and feedback channels.
• Keep human-in-the-loop for consequential outputs.

🕒 License and Usage

This model package derives from Chat GPT-5 so you're responsible for ensuring your use complies with the upstream model license and any dataset terms. For commercial deployment, review OpenAI’s license and your organization’s compliance requirements.

• Bubba is published under the MIT General Public License v3

🧩 Attribution

If this quant helped you, consider citing like:

bubba/gpt-oss-20b-Q4_K_M.gguf (2025).
Quantized GGUF build derived from OpenAI’s gpt-oss-20b.
Retrieved from the Hugging Face Hub.

❓ FAQ

Does quantization change the context window or tokenizer?
No. Those are inherited from the base model; quantization only changes weight representation.

Why am I hitting out-of-memory?
Lower -ngl (fewer GPU layers), reduce context (-c), or switch to a smaller quant (e.g., Q3_K). Ensure no other large models occupy VRAM.

Best sampler settings?
Start with temp 0.7, top_p 0.9, repeat_penalty 1.1.
Lower temperature for coding/planning; raise for creative writing.

📝 Changelog

• v1.0 — Initial release of bubba/gpt-20b-Q4_K_XL.gguf.