Joshfcooper/jamba-chat-lora

Jamba Chat LoRA

This is a LoRA fine-tuned version of the Jamba model trained on chat conversations.

Model Description

• Base Model: LaferriereJC/jamba_550M_trained
• Training Data: UltraChat dataset
• Task: Conversational AI
• Fine-tuning Method: LoRA (Low-Rank Adaptation)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

# Load the model
model = AutoModelForCausalLM.from_pretrained(
    "LaferriereJC/jamba_550M_trained",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(model, "your-username/jamba-chat-lora")

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("LaferriereJC/jamba_550M_trained")

# Example usage
text = "User: How are you today?\nAssistant:"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Training Details

• Training Data: UltraChat dataset (subset)
• LoRA Config:
  • Rank: 16
  • Alpha: 32
  • Target Modules: Last layer feed forward experts
  • Dropout: 0.1
• Training Parameters:
  • Learning Rate: 5e-4
  • Optimizer: AdamW (32-bit)
  • LR Scheduler: Cosine
  • Warmup Ratio: 0.03