Instructions to use laskar-ks/qwen-1.5b-product-category-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use laskar-ks/qwen-1.5b-product-category-classifier with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct") model = PeftModel.from_pretrained(base_model, "laskar-ks/qwen-1.5b-product-category-classifier") - Notebooks
- Google Colab
- Kaggle
Product Category Classifier (Qwen2.5-1.5B + LoRA)
Selects the correct product category from a shortlist of 8β9 candidates produced by a retrieval step. This is the rerank stage β it does not replace retrieval.
Trained on Shopify's official product-catalogue dataset (38,511 rows after cleaning), evaluated on the official test split (9,652 rows).
Results
| Method | Accuracy |
|---|---|
| Random (1 of ~8) | 12.0% |
Embedding cosine (multilingual-e5-small) |
35.8% |
| Qwen2.5-1.5B zero-shot | 34.3% |
| Qwen2.5-1.5B + LoRA | 66.0% Β± 0.9% |
95% CI on n=9,652. Differences below ~1.9 points are not distinguishable from noise.
The zero-shot baseline had a 0.0% parse failure rate, so the entire 31.7-point gain is accuracy β none of it comes from the model learning to follow the output format.
Both the fine-tuned model and the zero-shot baseline are evaluated with constrained
scoring: the logits for digit tokens 1βn are compared directly, so an out-of-range
answer is structurally impossible for either.
Accuracy by difficulty
difficulty = how many of the candidates share the same top-level category as the
correct answer. 1 = the answer stands alone; 9 = all candidates are siblings.
| difficulty | n | zero-shot | LoRA | delta |
|---|---|---|---|---|
| 1 | 998 | 29.6% | 83.1% | +53.5 |
| 2 | 651 | 30.4% | 70.0% | +39.6 |
| 3 | 655 | 30.7% | 68.5% | +37.9 |
| 4 | 710 | 32.5% | 61.3% | +28.7 |
| 5 | 831 | 32.9% | 61.9% | +29.0 |
| 6 | 1,010 | 35.0% | 66.6% | +31.6 |
| 7 | 1,332 | 38.1% | 64.9% | +26.7 |
| 8 | 2,864 | 39.4% | 63.9% | +24.4 |
| 9 | 601 | 20.0% | 53.6% | +33.6 |
The largest gain is at difficulty 1, which is counterintuitive β those should be the
easiest cases. The likely explanation is that these rows encode Shopify's internal
taxonomy conventions rather than semantic similarity. For example, a Dutch scrub brush
(Alpina schrobborstel) is labelled Shaving Brushes even though Scrub Brushes is
among the candidates. That is not noise: it is a consistent, non-obvious convention, and
the model learns it. Embedding similarity cannot.
Effect of product descriptions
| n | zero-shot | LoRA | |
|---|---|---|---|
| Description present (β₯20 chars) | 8,861 | 34.7% | 66.4% |
| Description absent | 791 | 30.1% | 61.1% |
Titles carry most of the signal. Catalogues with missing or poor descriptions lose about 5 points, not the task.
Generalization to unseen categories
Three ablation models, all with hyperparameters identical to the main run. Only the training data differs. Runs B2 and C are evaluated on the same 1,494-row slice.
| Run | Training data | Held-out categories seen? | Accuracy |
|---|---|---|---|
| A | 38,511 (100%) | yes | 66.0% |
| C | 10,537 (27%) | yes | 60.0% Β± 2.5% |
| B2 | 10,537 (27%) | never | 59.1% Β± 2.5% |
| B | ~33,000 (86%) | only as wrong answers | 14.9% Β± 2.2% |
| zero-shot | β | β | 31.8% |
C vs B2 β the main finding. These two differ by 0.9 points, well inside the margin of error. Having seen a category during training gives essentially no advantage. The entire drop from 66.0% to 59.1% is explained by training on 27% of the data, not by facing categories the model had never encountered.
The model is not memorising a category list. It learns how to read a product and match it against taxonomy structure, and that ability transfers intact to leaves absent from training. Practically: Shopify's taxonomy grows, and this model handles new leaves without retraining.
B vs B2 β the interesting failure. Removing a category from the answers while leaving it among the candidates is not an ablation. In Run B, 15% of training rows still contained the held-out categories as distractors, so the model saw them thousands of times and always as the wrong choice. It learned an active negative bias: those categories scored 17 points below the untuned base model.
Takeaway for anyone training a reranker: candidate composition in the training data matters as much as label distribution. Ignorance is far better than a learned negative bias.
Caveat: Run B used a different held-out set (701 leaves, n=1,008) than B2 and C (1,566 leaves, n=1,494), so its magnitude is not directly comparable. The 44-point gap far exceeds any plausible set-to-set variation, but the sets are not identical.
Inference cost
Benchmarked on A100 40GB, bfloat16, median prompt 368 tokens, constrained scoring (one forward pass per product, no generation loop).
| Metric | Value |
|---|---|
| Best throughput | 47.5 products/sec (batch 8) |
| Time per 1,000 products | 21.1 sec |
| Cost per 1,000 products | $0.0084 |
| 100,000 products | $0.84, ~35 minutes |
At $1.43/hr for an A100 40GB (Spheron, 2026-08-23).
Comparison to a hosted API, with an important limitation. At $0.15 per 1M input tokens, the same 1,000 products would cost about $0.0736 in input tokens alone β roughly 8.8Γ more.
This compares cost, not accuracy. No API model was evaluated on this test set. A larger hosted model would very likely score above 66.0%. The case for a local 1.5B model rests on cost, data residency, and throughput predictability β not on matching frontier models on accuracy. Anyone choosing between the two should measure both.
Two things that do not appear in the cost table but often decide the question:
- Catalogue data never leaves your infrastructure. Product naming and category structure are competitive assets.
- No rate limits. Batching 100,000 products through an API means retries and queueing; on your own GPU, throughput is bounded only by hardware.
Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from huggingface_hub import snapshot_download
BASE = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
tok.padding_side = "left"
if tok.pad_token is None:
tok.pad_token = tok.eos_token
base = AutoModelForCausalLM.from_pretrained(
BASE, torch_dtype=torch.bfloat16, device_map="cuda"
)
path = snapshot_download("laskar-ks/product-category-classifier-qwen-1.5b")
model = PeftModel.from_pretrained(base, path).eval()
SYSTEM = (
"You are a product taxonomy classifier. "
"Given a product and a numbered list of candidate categories, "
"reply with ONLY the number of the correct category. No explanation."
)
def classify(title, description, candidates):
listing = "\n".join(f"{i+1}. {c}" for i, c in enumerate(candidates))
user = (
f"Product title: {title}\n"
f"Description: {description or '(no description)'}\n\n"
f"Candidate categories:\n{listing}\n\n"
f"Answer with the number only (1-{len(candidates)}):"
)
prompt = tok.apply_chat_template(
[{"role": "system", "content": SYSTEM},
{"role": "user", "content": user}],
tokenize=False, add_generation_prompt=True,
)
enc = tok(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
logits = model(**enc).logits[0, -1, :]
# Constrained scoring: only digits 1..n are considered
digit_ids = [tok(str(i + 1), add_special_tokens=False)["input_ids"][0]
for i in range(len(candidates))]
return candidates[int(torch.argmax(logits[digit_ids]).item())]
print(classify(
"Alpina schrobborstel | 2 stuks",
"Ruige schrobborstel voor hardnekkige schoonmaak. Materiaal: kunststof.",
["Home & Garden > Household Supplies > Household Cleaning Supplies > Scrub Brushes",
"Health & Beauty > Personal Care > Shaving & Grooming > Shaving Brushes"],
))
The model expects candidates to come from a retrieval step. Feeding it arbitrary or unrelated candidate lists is outside what it was trained for.
Training
| Base model | Qwen/Qwen2.5-1.5B-Instruct |
| Method | LoRA, r=16, alpha=32, dropout=0.05 |
| Target modules | q/k/v/o_proj, gate/up/down_proj |
| Trainable params | 18,464,768 (1.18%) |
| Epochs | 1 |
| Effective batch | 16 (4 Γ grad accum 4) |
| Learning rate | 2e-4, cosine, 75 warmup steps |
| Precision | bfloat16, gradient checkpointing on |
| Hardware | A100 40GB (Colab Pro), ~40 min |
| Seed | 42 |
Candidate order is randomly permuted during both training and evaluation, so the model cannot learn a positional shortcut.
Loss is computed only on the answer token (prompt tokens masked with -100). Without this, LoRA capacity is spent memorising prompt text that is always supplied anyway.
Data preparation
Derived from Shopify/product-catalogue (Apache 2.0). Official train/test splits preserved.
- Dropped
product_image(text-only task) - Removed rows with an empty title, or where
ground_truth_categorywas absent frompotential_product_categories - Deduplicated on
product_titlewithin each split (81 removed from train, 6 from test) - 39 titles appeared in both the official train and test splits. Removed from train only; the test split was left untouched so results stay comparable to other work on this dataset
- Truncated titles to 200 chars, descriptions to 1,000 chars (one description ran to 46,773 chars)
- Added
difficultyandhas_desccolumns for the analyses above
Final: 38,511 train / 9,652 test.
Reproducibility
Hardware: A100 40GB (Colab Pro). Seed 42 throughout.
Package versions matter here β the HF ecosystem moved quickly around this release and several combinations fail:
torch 2.11.0+cu128
transformers 5.15.0
peft 0.13.2
Known issue. peft==0.13.2 calls hf_hub_download(use_auth_token=...), which newer
huggingface_hub releases removed. Loading the adapter directly by repo ID raises a
misleading Can't find 'adapter_config.json' error. Download first, then load from the
local path (as in the usage example above), or pin huggingface_hub<0.26.
transformers 5.x also removed warmup_ratio from TrainingArguments; warmup_steps
works on both 4.x and 5.x.
Limitations
- Rerank only. Requires a retrieval step to produce 8β9 candidates. It does not classify into the full Shopify taxonomy from scratch.
- Cost comparison is not an accuracy comparison. No API model was benchmarked on this test set.
- Ceiling unknown. The dataset's own labels were not audited. Some appear to encode non-obvious conventions rather than errors, but no manual audit was run, so the maximum achievable accuracy on this test set is not established.
- English-dominant. The training data includes Spanish, Dutch, Russian, and German products, but the distribution was not measured and per-language accuracy was not broken out.
- Single epoch, no validation set. Training loss was still declining at the end of epoch 1. Whether a second epoch helps was not tested, because no validation split was held out to detect overfitting.
License
Apache 2.0, matching the source dataset.
- Downloads last month
- 27