hassan7272/urdu-finance-embeddings

🏦 FinGuard Urdu Finance Embeddings

The first domain-specific embedding model for Pakistani financial queries across Urdu, Roman Urdu, and English

---

The Problem This Solves

Pakistan has 220 million people, millions of active fintech users (Easypaisa, JazzCash, Raast), and a growing Islamic finance sector — yet almost no NLP infrastructure exists for Pakistani financial queries.

The challenge is threefold:

• Pakistanis write finance questions in three mixed forms: Urdu script (کیا زکوٰۃ واجب ہے), Roman Urdu (zakat ka hisab kaise karein), and English (how to calculate zakat)
• Islamic finance has specialized vocabulary (Murabaha, Riba, Musharakah, Nisab) that generic multilingual models misunderstand
• Existing models trained on Wikipedia/news fail completely on Pakistani financial context

This model was built from the ground up to fix that.

---

What Makes This Different

Feature	Generic Multilingual Models	This Model
Roman Urdu queries	❌ Poor	✅ Trained on it
Islamic finance terms	❌ No context	✅ Domain fine-tuned
Pakistani banking context	❌ None	✅ HBL, Meezan, NBP, UBL
Mixed-language queries	❌ Struggles	✅ Handles naturally
RAG retrieval accuracy	~60–70%	93.9% @ Top-1

---

Model Performance

Evaluated using InformationRetrievalEvaluator on a held-out validation set of 147 financial queries across all 8 categories.

Metric	Score	What It Means
Accuracy@1	93.9%	Correct answer ranked #1 out of corpus
Accuracy@3	99.3%	Correct answer in top 3
Accuracy@10	100.0%	Never misses — perfect recall at 10
MRR@10	0.966	Near-perfect mean reciprocal rank
NDCG@10	0.975	Excellent ranking quality
MAP@100	0.966	Consistent precision across the board

Baseline (zero fine-tuning): Accuracy@1 ≈ 72.8% → +21 points gained from domain training

---

Quick Start

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("hassan7272/urdu-finance-embeddings")

# Works in all three forms — no preprocessing needed
queries = [
    "zakat ka hisab kaise karein",                    # Roman Urdu
    "زکوٰۃ کا حساب کیسے کریں",                        # Urdu script
    "how to calculate zakat in Pakistan",             # English
]

# Your answer corpus
answers = [
    "Zakat 2.5% hoti hai jo nisab se zyada savings par lagti hai...",
    "Meezan Bank Islamic saving account mein profit milta hai...",
]

query_embeddings  = model.encode(queries,  convert_to_tensor=True)
answer_embeddings = model.encode(answers,  convert_to_tensor=True)

scores = util.cos_sim(query_embeddings, answer_embeddings)
print(scores)

RAG Retrieval (FAISS)

from sentence_transformers import SentenceTransformer
import faiss, numpy as np

model = SentenceTransformer("hassan7272/urdu-finance-embeddings")

# Build index over your answer corpus
answers = ["answer 1 ...", "answer 2 ...", ...]
embeddings = model.encode(answers, normalize_embeddings=True)

index = faiss.IndexFlatIP(embeddings.shape[1])   # inner product = cosine on normalized
index.add(embeddings.astype(np.float32))

# Retrieve
query = "Easypaisa se ghar ka kiraya kaise bharein"
q_emb = model.encode([query], normalize_embeddings=True).astype(np.float32)
scores, indices = index.search(q_emb, k=10)      # top-10 retrieval

---

Training Pipeline

This model was produced through a full ML engineering pipeline built from scratch — not just a fine-tuning script.

Stage 1 — Dataset Engineering

The training data comes from hassan7272/urdu-finance-qa, a custom 1,510-record Q&A dataset across 8 Pakistani financial categories.

Text normalization handled:

• Urdu Unicode character variants (multiple encodings of the same letter)
• Roman Urdu repeated character collapse (kyaaaa → kyaa)
• Spacing/punctuation normalization across all three languages
• Language detection per record (ur / roman_ur / en)

Stage 2 — Hard Negative Mining

Instead of random negatives, a category-aware hard negative miner was built:

Query: "zakat ka hisab kaise karein"  (Islamic Finance)
Random negative: "online bill payment kaise karein"   ← too easy, model ignores
Hard negative:   "loan ka interest kaise calculate hota hai"  ← looks relevant, is WRONG

The confusable category map forces the model to discriminate on intent, not just topic. For example, islamic_finance negatives are drawn from loans_credit and personal_finance — the most semantically similar but semantically incorrect categories. Keyword overlap filtering ensures negatives are hard but not trivially different.

Stage 3 — Training Strategy

2,752 training examples were built from three sources:

• Primary: (question_ur, answer_ur) pairs — native Urdu / Roman Urdu
• Hard negatives: (question_ur, hard_negative, answer_ur) triplets — explicit confusion signal
• Cross-lingual (30%): (question_en, answer_en) pairs — free cross-lingual alignment

Loss: MultipleNegativesRankingLoss (scale=20, cosine similarity)

• Every other pair in the batch becomes an automatic in-batch negative
• With 16 batch size → each example sees 15 automatic negatives + 1 explicit hard negative
• Model learns to push correct pairs together and wrong pairs apart in embedding space

Training config:

Base model    : paraphrase-multilingual-mpnet-base-v2
Epochs        : 4
Batch size    : 16
Learning rate : 5e-5
FP16          : True (mixed precision)
Platform      : Kaggle GPU (CUDA)
Train time    : 7 minutes

Stage 4 — Evaluation

InformationRetrievalEvaluator was used at every 86 steps during training, treating the entire validation answer set as a retrieval corpus. This is the same evaluation setup used in real RAG systems — not a toy cosine similarity test.

Training progression:

Epoch	Step	NDCG@10
1.0	86	0.9547
1.16	100	0.9546
2.0	172	0.9746 ← best

Model converged at epoch 2 with train loss of 0.063 — a very low value indicating clean convergence without overfitting on the 1.5k dataset.

---

Dataset Coverage

Category	Records	Description
Personal Finance	251	Budgeting, savings, emergency funds
Islamic Finance	240	Zakat, Riba, Murabaha, Sukuk
Financial Education	237	Concepts, terminology, literacy
Banking	222	HBL, MCB, NBP, Meezan, UBL
Investment	183	Mutual funds, stocks, real estate
Loans & Credit	159	Home loans, car financing, credit cards
Digital Finance	141	Easypaisa, JazzCash, Raast, SadaPay
Bills & Payments	77	Utility bills, tax payments, DISCO

---

Architecture Context

This model is Phase 1 of the FinGuard RAG system — a full retrieval-augmented generation pipeline for Pakistani financial advisory:

User Query (Urdu / Roman Urdu / English)
        ↓
  Query Normalization
        ↓
  Hybrid Search (this model + BM25)
        ↓
  Top-10 Retrieval
        ↓
  MMR (diversity filtering)
        ↓
  Cross-Encoder Reranker → Top-3
        ↓
  LLM Answer Generation

The embedding model (this model) handles the vector search component with 93.9% accuracy at Top-1.

---

Limitations

• Trained on synthetic Q&A data — real-world distribution may differ slightly
• Coverage is Pakistan-specific; Indian Urdu financial context may vary
• Answers are from 2024 — regulatory/rate information may be outdated
• Roman Urdu spelling variation is partially handled but highly informal text may still vary

---

Citation

If you use this model in your research or application, please cite:

@misc{hassan2025finguard,
  title        = {FinGuard Urdu Finance Embeddings: Domain-Specific Multilingual Embeddings for Pakistani Financial RAG},
  author       = {Hassan},
  year         = {2025},
  publisher    = {HuggingFace},
  url          = {https://huggingface.co/hassan7272/urdu-finance-embeddings},
  note         = {Fine-tuned on urdu-finance-qa dataset with hard negative mining and MultipleNegativesRankingLoss}
}

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title     = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author    = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    year      = "2019",
    url       = "https://arxiv.org/abs/1908.10084",
}

---

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.3
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.8.3

---

Built as part of the FinGuard RAG project — Islamic Finance Advisory for Pakistan 🇵🇰