MedQA-Llama3.1-8B-HELIX-v2

Trilingual (French / English / Moroccan Darija) medical reasoning LoRA adapter on top of meta-llama/Llama-3.1-8B-Instruct, trained via the HELIX-FT v2 curriculum on a single RunPod L40S GPU.

Research artefact only. Not a medical device, not clinically validated. The scores below are not compared with a matched Llama-3.1-8B-Instruct baseline, so they do not show that this adapter improves on its base.

Training pipeline

This adapter sits on top of TWO stacked components, both available on HF:

1. Base: meta-llama/Llama-3.1-8B-Instruct (gated)
2. Intermediate adapter: BrainHealthAI/MedQA-Llama3.1-8B-SFT-Big — merged into the base to produce a dense intermediate BrainMed-Base-v1
3. HELIX-v2 (this adapter) trained on the dense intermediate in 3 stages:
   • Stage A — Continued SFT on BrainHealthAI/MedUnified (22,505 trilingual training rows). A MinHash LSH overlap filter was applied against MedQA-USMLE / MedMCQA / PubMedQA / MMLU-medical; the filter has not been independently audited and cross-stage leakage remains to be checked.
   • Stage B — SASR (step-wise adaptive SFT + GRPO, 3K samples). Reward = `0.2·format
     • 0.8·SB_CS(answer, gold) − drug_penalty`. The penalty (up to 0.30) targets drugs cited in the answer that appear neither in the gold answer nor in the Dorosz vocabulary; it is a narrow lexical proxy, not a general hallucination detector. SB_CS is both optimized here and reported in the evaluation below, so it is not independent evidence of a SASR benefit.
   • Stage C — CoT polish on BrainHealthAI/BrainMedCoT (2,768 training traces with structured <think>...</think>{answer} format; all reasoning traces are LLM-generated)

QLoRA (r=64, α=128, dropout=0.1) on 7 attention/MLP target modules. paged_adamw_8bit, bf16, sequence packing, max_seq_len=2048. No controlled ablation of stages A/B/C exists.

Inference

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

BASE     = "meta-llama/Llama-3.1-8B-Instruct"
SFT_BIG  = "BrainHealthAI/MedQA-Llama3.1-8B-SFT-Big"
HELIX_V2 = "BrainHealthAI/MedQA-Llama3.1-8B-HELIX-v2"

# Step 1: load the base + merge the SFT-Big intermediate adapter
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
m = PeftModel.from_pretrained(base, SFT_BIG)
m = m.merge_and_unload()

# Step 2: attach the HELIX-v2 adapter
model = PeftModel.from_pretrained(m, HELIX_V2)
tok = AutoTokenizer.from_pretrained(BASE)

# Generate (with the <think> structure trained in Stage C)
prompt = tok.apply_chat_template(
    [{"role": "system", "content": "You are a medical reasoning assistant."},
     {"role": "user",   "content": "Patient presents with..."}],
    tokenize=False, add_generation_prompt=True,
)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Evaluation (historical, descriptive)

From eval_v2.json (2026-05-31). MCQ method: log-likelihood argmax over option letters, 4-bit model, first n items of each set (no random sampling, no confidence intervals, no per-item outputs published). PubMedQA uses pqa_labeled without the abstract context, so it is not comparable with standard PubMedQA results.

Benchmark	Accuracy	n
MedQA-USMLE	52.50%	200
MedMCQA	41.50%	200
PubMedQA (question only)	53.50%	200
MMLU-clinical_knowledge	61.00%	200
MMLU-medical_genetics	67.00%	100
MMLU-professional_medicine	54.00%	200
MMLU-college_medicine	57.23%	173
Darija-MCQ-500	55.00%	200

Language (MedUnified held-out)	SB_CS	ROUGE-L	n
EN	0.695	0.163	100
FR	0.630	0.123	100
DARIJA	0.581	0.031	100

Generation cases differ across languages (not paired translations), so these differences are not a causal language effect. SB_CS and ROUGE-L measure textual similarity, not medical correctness, completeness or safety.

Limitations

• No matched baseline. Llama-3.1-8B-Instruct and SFT-Big were not evaluated with the same items, prompts and scoring. Do not infer a gain from published scores obtained elsewhere.
• Darija-MCQ-500 is a project-built set with known weaknesses: source_pair_id is empty on all 500 items, and answer-style cues are strong — choosing the shortest option alone is correct on 250/500 items, and a shortest-option-plus-keyword rule scores 66% on the 200 evaluated items versus 55% for this adapter. The Darija score is therefore not evidence of Darija medical competence. A rebuilt, paired and physician-reviewed set is required.
• Reward and metric overlap: SB_CS is part of the SASR reward and of the evaluation.
• Not for clinical use. Research artefact only; any clinical application would require independent clinical evaluation and is outside the scope of CC BY-NC 4.0.

Details and audit counts: BrainHealthAI/BRAIN-train (Training_v2).

License

CC BY-NC 4.0 — research / non-commercial. The base model meta-llama/Llama-3.1-8B-Instruct has its own Meta Llama 3 Community License; users must comply with both.

Citation

@misc{medqa_llama3_helix_v2_2026,
  author       = {BrainHealthAI},
  title        = {MedQA-Llama3.1-8B-HELIX-v2: Trilingual Medical Reasoning Adapter},
  year         = 2026,
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/BrainHealthAI/MedQA-Llama3.1-8B-HELIX-v2}},
}

Related artefacts: BrainHealthAI/MedUnified (training data), BrainHealthAI/BrainMedCoT (CoT corpus), BrainHealthAI/MedQA-Darija-MCQ-500 (Darija eval).