This is a llava-phi-3-mini-hf fine-tune, produced at the request of redaihf through P-E-W's Heretic (v1.3.0) abliteration engine with Self-Organizing Maps & Magnitude-Preserving Orthogonal Ablation enabled.

Note: You may want to configure RoPE with YaRN or LongRoPE with llama.cpp to extend this models minuscule context size. I decided to just release it and deal with the RoPE later when I have more time.

Heretication Results

Score Metric	Value	Parameter	Value
Refusals	4/416	direction_index	17.80
KL Divergence	0.0105	attn.o_proj.max_weights.0	0: 1.48
Initial Refusals	403/416	attn.o_proj.max_weights.1	1: 1.02
		attn.o_proj.max_weights.2	2: 1.35
		attn.o_proj.max_weights.3	3: 0.44
		attn.o_proj.max_weights.4	4: 0.19
		attn.o_proj.max_weight_position	17.99
		attn.o_proj.min_weights.0	0: 0.94
		attn.o_proj.min_weights.1	1: 0.68
		attn.o_proj.min_weights.2	2: 1.04
		attn.o_proj.min_weights.3	3: 0.12
		attn.o_proj.min_weights.4	4: 0.14
		attn.o_proj.min_weight_distance	4.49
		mlp.down_proj.max_weights.0	0: 0.27
		mlp.down_proj.max_weights.1	1: 0.29
		mlp.down_proj.max_weights.2	2: 0.13
		mlp.down_proj.max_weights.3	3: 0.30
		mlp.down_proj.max_weights.4	4: 0.50
		mlp.down_proj.max_weight_position	25.66
		mlp.down_proj.min_weights.0	0: 0.13
		mlp.down_proj.min_weights.1	1: 0.22
		mlp.down_proj.min_weights.2	2: 0.00
		mlp.down_proj.min_weights.3	3: 0.05
		mlp.down_proj.min_weights.4	4: 0.33
		mlp.down_proj.min_weight_distance	4.25

Appendix

Empty system prompt.

Heretication Rituals

   [Trial 298] Refusals:  2/416, KL divergence: 0.0179
   [Trial 290] Refusals:  3/416, KL divergence: 0.0157
   [Trial  30] Refusals:  4/416, KL divergence: 0.0105
 » [Trial 190] Refusals:  5/416, KL divergence: 0.0097
   [Trial 180] Refusals:  8/416, KL divergence: 0.0086
   [Trial 124] Refusals: 14/416, KL divergence: 0.0067
   [Trial  34] Refusals: 140/416, KL divergence: 0.0037
   [Trial 149] Refusals: 308/416, KL divergence: 0.0036
   [Trial  46] Refusals: 330/416, KL divergence: 0.0028
   [Trial  12] Refusals: 390/416, KL divergence: 0.0027
   [Trial 184] Refusals: 396/416, KL divergence: 0.0016
   [Trial  81] Refusals: 399/416, KL divergence: 0.0010
   [Trial 139] Refusals: 401/416, KL divergence: 0.0007
   [Trial  52] Refusals: 403/416, KL divergence: 0.0003

PIQA Benchmarks

┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T190    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7998 │         0.8003 │
│           │ acc_stderr,none      │     0.0093 │         0.0093 │
│           │ acc_norm,none        │     0.8030 │         0.8052 │
│           │ acc_norm_stderr,none │     0.0093 │         0.0092 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T180    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7998 │         0.8003 │
│           │ acc_stderr,none      │     0.0093 │         0.0093 │
│           │ acc_norm,none        │     0.8020 │         0.8052 │
│           │ acc_norm_stderr,none │     0.0093 │         0.0092 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T298    ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.7971 │         0.8003 │
│           │ acc_stderr,none      │     0.0094 │         0.0093 │
│           │ acc_norm,none        │     0.8020 │         0.8052 │
│           │ acc_norm_stderr,none │     0.0093 │         0.0092 │
└───────────┴──────────────────────┴────────────┴────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Metric               ┃    T30     ┃      Base      ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ PIQA      │ acc,none             │     0.8014 │         0.8003 │
│           │ acc_stderr,none      │     0.0093 │         0.0093 │
│           │ acc_norm,none        │     0.8025 │         0.8052 │
│           │ acc_norm_stderr,none │     0.0093 │         0.0092 │
└───────────┴──────────────────────┴────────────┴────────────────┘

Residual Geometry

┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Layer ┃ S(g,b) ┃ S(g*,b*) ┃  S(g,r) ┃ S(g*,r*) ┃ S(b,r) ┃ S(b*,r*) ┃    |g| ┃   |g*| ┃    |b| ┃   |b*| ┃    |r| ┃   |r*| ┃   Silh ┃
┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│     1 │ 0.9936 │   0.9929 │  0.0216 │   0.0191 │ 0.1346 │   0.1376 │   1.45 │   1.45 │   1.46 │   1.46 │   0.17 │   0.17 │ 0.1511 │
│     2 │ 0.9642 │   0.9589 │  0.0506 │   0.0224 │ 0.3136 │   0.3050 │   2.29 │   2.29 │   2.41 │   2.41 │   0.64 │   0.68 │ 0.2595 │
│     3 │ 0.9719 │   0.9696 │  0.1552 │   0.1313 │ 0.3832 │   0.3700 │   3.66 │   3.67 │   3.91 │   3.92 │   0.93 │   0.97 │ 0.2001 │
│     4 │ 0.9703 │   0.9695 │  0.2505 │   0.2359 │ 0.4772 │   0.4670 │   4.66 │   4.67 │   5.14 │   5.13 │   1.28 │   1.29 │ 0.1396 │
│     5 │ 0.9745 │   0.9743 │  0.0893 │   0.0620 │ 0.3106 │   0.2854 │   6.26 │   6.29 │   6.56 │   6.55 │   1.48 │   1.48 │ 0.1083 │
│     6 │ 0.9562 │   0.9559 │ -0.0331 │  -0.0519 │ 0.2609 │   0.2439 │   6.98 │   7.01 │   7.22 │   7.22 │   2.12 │   2.12 │ 0.1218 │
│     7 │ 0.9438 │   0.9445 │  0.0288 │   0.0214 │ 0.3575 │   0.3486 │   9.05 │   9.08 │   9.68 │   9.69 │   3.20 │   3.18 │ 0.1341 │
│     8 │ 0.9377 │   0.9391 │  0.0656 │   0.0620 │ 0.4082 │   0.4012 │  10.24 │  10.29 │  11.19 │  11.21 │   3.90 │   3.86 │ 0.1337 │
│     9 │ 0.9279 │   0.9303 │  0.1726 │   0.1718 │ 0.5275 │   0.5213 │  12.10 │  12.19 │  14.03 │  14.07 │   5.31 │   5.24 │ 0.1428 │
│    10 │ 0.9071 │   0.9099 │  0.1434 │   0.1429 │ 0.5467 │   0.5406 │  13.91 │  14.01 │  16.44 │  16.48 │   6.99 │   6.91 │ 0.1417 │
│    11 │ 0.9030 │   0.9060 │  0.1366 │   0.1332 │ 0.5491 │   0.5403 │  14.91 │  15.05 │  17.67 │  17.73 │   7.66 │   7.57 │ 0.1437 │
│    12 │ 0.8872 │   0.8897 │  0.1475 │   0.1452 │ 0.5872 │   0.5808 │  16.55 │  16.70 │  20.23 │  20.30 │   9.44 │   9.37 │ 0.1585 │
│    13 │ 0.8445 │   0.8473 │  0.1424 │   0.1423 │ 0.6503 │   0.6463 │  19.49 │  19.66 │  25.39 │  25.50 │  13.74 │  13.69 │ 0.1866 │
│    14 │ 0.8443 │   0.8466 │  0.0671 │   0.0622 │ 0.5914 │   0.5839 │  22.94 │  23.15 │  28.38 │  28.46 │  15.24 │  15.18 │ 0.1825 │
│    15 │ 0.8100 │   0.8101 │  0.0340 │   0.0268 │ 0.6137 │   0.6078 │  26.52 │  26.83 │  33.57 │  33.78 │  19.70 │  19.81 │ 0.1925 │
│    16 │ 0.7032 │   0.6975 │ -0.0128 │  -0.0199 │ 0.7019 │   0.7025 │  31.05 │  31.46 │  43.60 │  44.20 │  31.00 │  31.68 │ 0.2510 │
│    17 │ 0.6838 │   0.6742 │ -0.0500 │  -0.0614 │ 0.6946 │   0.6958 │  36.62 │  37.15 │  50.84 │  51.63 │  37.15 │  38.20 │ 0.2725 │
│    18 │ 0.6740 │   0.6622 │ -0.0578 │  -0.0717 │ 0.6986 │   0.6999 │  42.03 │  42.72 │  58.64 │  59.66 │  43.39 │  44.82 │ 0.2551 │
│    19 │ 0.6560 │   0.6407 │ -0.0358 │  -0.0531 │ 0.7308 │   0.7327 │  50.76 │  51.63 │  74.31 │  75.75 │  56.13 │  58.25 │ 0.2824 │
│    20 │ 0.6628 │   0.6464 │ -0.0053 │  -0.0250 │ 0.7453 │   0.7466 │  59.57 │  60.68 │  89.35 │  91.17 │  66.91 │  69.58 │ 0.2887 │
│    21 │ 0.6313 │   0.6141 │ -0.0113 │  -0.0328 │ 0.7683 │   0.7687 │  67.24 │  68.49 │ 105.04 │ 107.02 │  81.47 │  84.51 │ 0.3088 │
│    22 │ 0.6080 │   0.5919 │ -0.0108 │  -0.0301 │ 0.7873 │   0.7878 │  75.56 │  77.00 │ 122.54 │ 124.96 │  97.30 │ 100.77 │ 0.3172 │
│    23 │ 0.6014 │   0.5844 │  0.0091 │  -0.0104 │ 0.8044 │   0.8054 │  84.09 │  85.62 │ 141.55 │ 144.43 │ 113.09 │ 117.22 │ 0.3150 │
│    24 │ 0.5822 │   0.5654 │ -0.0201 │  -0.0383 │ 0.8012 │   0.8025 │  91.28 │  92.98 │ 152.50 │ 155.73 │ 124.01 │ 128.54 │ 0.3078 │
│    25 │ 0.5650 │   0.5479 │ -0.0230 │  -0.0407 │ 0.8119 │   0.8136 │  96.87 │  98.72 │ 165.89 │ 169.62 │ 136.91 │ 142.01 │ 0.3008 │
│    26 │ 0.5666 │   0.5498 │ -0.0331 │  -0.0506 │ 0.8048 │   0.8064 │ 106.75 │ 108.78 │ 179.75 │ 183.72 │ 148.20 │ 153.66 │ 0.2900 │
│    27 │ 0.5527 │   0.5360 │ -0.0511 │  -0.0681 │ 0.8040 │   0.8058 │ 114.26 │ 116.50 │ 191.92 │ 196.25 │ 160.15 │ 166.07 │ 0.2826 │
│    28 │ 0.5507 │   0.5337 │ -0.0456 │  -0.0619 │ 0.8087 │   0.8110 │ 119.50 │ 121.69 │ 202.94 │ 207.59 │ 169.57 │ 175.89 │ 0.2734 │
│    29 │ 0.5539 │   0.5373 │ -0.0334 │  -0.0492 │ 0.8136 │   0.8160 │ 136.13 │ 138.53 │ 234.04 │ 239.33 │ 194.97 │ 202.10 │ 0.2750 │
│    30 │ 0.5702 │   0.5539 │ -0.0210 │  -0.0358 │ 0.8093 │   0.8122 │ 149.81 │ 152.10 │ 255.01 │ 260.58 │ 209.53 │ 217.10 │ 0.2623 │
│    31 │ 0.5806 │   0.5651 │ -0.0236 │  -0.0362 │ 0.8003 │   0.8040 │ 161.47 │ 163.61 │ 269.21 │ 274.98 │ 219.25 │ 227.01 │ 0.2551 │
│    32 │ 0.6540 │   0.6412 │ -0.0135 │  -0.0211 │ 0.7476 │   0.7537 │  29.84 │  30.05 │  44.92 │  45.72 │  33.99 │  35.09 │ 0.2460 │
└───────┴────────┴──────────┴─────────┴──────────┴────────┴──────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
g = mean of residual vectors for good prompts
g* = geometric median of residual vectors for good prompts
b = mean of residual vectors for bad prompts
b* = geometric median of residual vectors for bad prompts
r = refusal direction for means (i.e., b - g)
r* = refusal direction for geometric medians (i.e., b* - g*)
S(x,y) = cosine similarity of x and y
|x| = L2 norm of x
Silh = Mean silhouette coefficient of residuals for good/bad clusters

Model

llava-phi-3-mini is a LLaVA model fine-tuned from microsoft/Phi-3-mini-4k-instruct and CLIP-ViT-Large-patch14-336 with ShareGPT4V-PT and InternVL-SFT by XTuner.

Note: This model is in HuggingFace LLaVA format.

Resources:

GitHub: xtuner
Official LLaVA format model: xtuner/llava-phi-3-mini
GGUF LLaVA model: xtuner/llava-phi-3-mini-gguf
XTuner LLaVA format model: xtuner/llava-phi-3-mini-xtuner

Details

Model	Visual Encoder	Projector	Resolution	Pretraining Strategy	Fine-tuning Strategy	Pretrain Dataset	Fine-tune Dataset	Pretrain Epoch	Fine-tune Epoch
LLaVA-v1.5-7B	CLIP-L	MLP	336	Frozen LLM, Frozen ViT	Full LLM, Frozen ViT	LLaVA-PT (558K)	LLaVA-Mix (665K)	1	1
LLaVA-Llama-3-8B	CLIP-L	MLP	336	Frozen LLM, Frozen ViT	Full LLM, LoRA ViT	LLaVA-PT (558K)	LLaVA-Mix (665K)	1	1
LLaVA-Llama-3-8B-v1.1	CLIP-L	MLP	336	Frozen LLM, Frozen ViT	Full LLM, LoRA ViT	ShareGPT4V-PT (1246K)	InternVL-SFT (1268K)	1	1
LLaVA-Phi-3-mini	CLIP-L	MLP	336	Frozen LLM, Frozen ViT	Full LLM, Full ViT	ShareGPT4V-PT (1246K)	InternVL-SFT (1268K)	1	2

Results

Model	MMBench Test (EN)	MMMU Val	SEED-IMG	AI2D Test	ScienceQA Test	HallusionBench aAcc	POPE	GQA	TextVQA	MME	MMStar
LLaVA-v1.5-7B	66.5	35.3	60.5	54.8	70.4	44.9	85.9	62.0	58.2	1511/348	30.3
LLaVA-Llama-3-8B	68.9	36.8	69.8	60.9	73.3	47.3	87.2	63.5	58.0	1506/295	38.2
LLaVA-Llama-3-8B-v1.1	72.3	37.1	70.1	70.0	72.9	47.7	86.4	62.6	59.0	1469/349	45.1
LLaVA-Phi-3-mini	69.2	41.4	70.0	69.3	73.7	49.8	87.3	61.5	57.8	1477/313	43.7

Quickstart

Chat by `pipeline`

from transformers import pipeline
from PIL import Image    
import requests

model_id = "xtuner/llava-phi-3-mini-hf"
pipe = pipeline("image-to-text", model=model_id, device=0)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"

image = Image.open(requests.get(url, stream=True).raw)
prompt = "<|user|>\n<image>\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud<|end|>\n<|assistant|>\n"

outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 200})
print(outputs)
>>> [{'generated_text': '\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud (1) lava'}]

Chat by pure `transformers`

import requests
from PIL import Image

import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration

model_id = "xtuner/llava-phi-3-mini-hf"

prompt = "<|user|>\n<image>\nWhat are these?<|end|>\n<|assistant|>\n"
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"

model = LlavaForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True, 
).to(0)

processor = AutoProcessor.from_pretrained(model_id)


raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)

output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))
>>> What are these? These are two cats sleeping on a pink couch.

Reproduce

Please refer to docs.

Citation

@misc{2023xtuner,
    title={XTuner: A Toolkit for Efficiently Fine-tuning LLM},
    author={XTuner Contributors},
    howpublished = {\url{https://github.com/InternLM/xtuner}},
    year={2023}
}

Downloads last month: 21

Safetensors

Model size

4B params

Tensor type

F16

Model tree for MuXodious/llava-phi-3-mini-hf-SOMPOA-heresy

Base model

xtuner/llava-phi-3-mini-hf

Finetuned

(1)

this model

Dataset used to train MuXodious/llava-phi-3-mini-hf-SOMPOA-heresy

Collection including MuXodious/llava-phi-3-mini-hf-SOMPOA-heresy

experimenta-haeretica

Collection

Heretical experimentation • 38 items • Updated Aug 22 • 4