Building Bilingual RAG for Persian and English: Tokenizing to Reranking

📋 Table of Contents

The Silent Failure Mode of Bilingual RAG
1. The Bi-Directional Tokenization Trap (English vs. Persian)
2. Script Normalization and Orthographic Chaos (Zero-Width Non-Joiners & Arabic Yeh)
3. Finding the Needle: Multilingual Embedding Models for N-W-Asian Contexts
4. Cross-Lingual Chunking and the RTL Structural Breakdown
5. Hybrid Search Strategies: BM25 Meets Persian Morphology
6. Cross-Encoder Reranking Across Linguistic Divides
7. Prompt Architecture and Hallucination Control in Dual-Script Generation
8. Production Observability and Semantic Drift Detection
Production Architecture Decision Guide

The Silent Failure Mode of Bilingual RAG

You deploy a Retrieval-Augmented Generation (RAG) pipeline trained on clean English technical documentation, wire it up to a vector database, and watch it ace your internal testing. Then, a user enters a query in Persian (چگونه کلاستر کوبرنتیز را آپدیت کنیم؟), and your system calmly retrieves a document about Kubernetes volume backups, completely misses the core update procedures written in the Persian knowledge base, and synthesizes an answer that blends English technical terms with hallucinated Persian syntax.

To the casual observer, the pipeline "worked"—it returned chunks and generated a response. To an engineer, it’s a total architectural collapse caused by cross-lingual semantic misalignment.

Building a production-grade multilingual RAG system for English and Persian is profoundly difficult because these two languages live on opposite sides of multiple technical divides:

Typographic Direction: Left-to-Right (LTR) vs. Right-to-Left (RTL) mixed inline.
Morphological Structure: Fusional/analytic Indo-European vs. highly agglutinative and derivational Indo-Iranian.
Tokenization Overhead: Subword tokenizers trained primarily on Western corpora that bloat Persian text, shredding semantic units into unrecoverable fragments.

This guide breaks down how to architect, implement, and scale a production-ready multilingual RAG for English and Persian documents without falling into the common traps that plague naive implementations.

1. The Bi-Directional Tokenization Trap (English vs. Persian)

Scenario: You pass a mixed Persian-English technical document containing code snippets into a standard LLM pipeline. The resulting token count for the Persian text is 4x higher than the English equivalent, leading to context window exhaustion and inflated API costs.

Why it matters: Most mainstream subword tokenizers (such as cl100k_base used by GPT-4 or standard Llama tokenizers) were trained on corpora heavily skewed toward English and other Latin-script languages. When a Persian string like پایگاه داده (database) hits the tokenizer, it doesn't map to clean, cohesive subword tokens. Instead, it gets shattered into individual Unicode scalar values or tiny character fragments. This destroys the semantic spatial locality required for vector embeddings.

Solution: You must audit your tokenization layer, enforce strict text sanitization before embedding generation, and choose embedding and generation models with explicit, robust multilingual tokenizers (like those found in modern multilingual models).

import tiktoken
from transformers import AutoTokenizer

def analyze_token_inflation(text_en: str, text_fa: str):
    # Using OpenAI's cl100k_base as a baseline example
    enc = tiktoken.get_encoding("cl100k_base")
    
    tokens_en = enc.encode(text_en)
    tokens_fa = enc.encode(text_fa)
    
    print(f"English Character Count: {len(text_en)} | Token Count: {len(tokens_en)} | Ratio: {len(tokens_en)/len(text_en):.2f}")
    print(f"Persian Character Count: {len(text_fa)} | Token Count: {len(tokens_fa)} | Ratio: {len(tokens_fa)/len(text_fa):.2f}")

# Example execution
analyze_token_inflation(
    "Kubernetes cluster configuration deployment.",
    "پیکربندی و استقرار کلاستر کوبرنتیز."
)

Why this works: By quantifying token inflation explicitly during ingestion, you can catch encoding anomalies before they hit your vector index. More importantly, this forces you to choose models like Cohere Embed v3, text-embedding-3-large, or specialized multilingual open-weights models that have balanced vocabulary allocations for Perso-Arabic scripts.

⚠️ Gotcha: Never assume a tokenizer handles mixed-script strings (like English variable names inside Persian markdown documentation) gracefully. Zero-width spaces and directional formatting characters (\u200e, \u200f) will sneak into your raw text strings and create invisible token pollution. Strip them or normalize them before tokenization.

2. Script Normalization and Orthographic Chaos (Zero-Width Non-Joiners & Arabic Yeh)

Scenario: A user searches for دانشگاه (university) using a standard keyboard layout, but your vector database contains documents indexed with the Arabic Yeh (ي - U+064A) instead of the Persian Yeh (ی - U+06CC), or missing the Zero-Width Non-Joiner (ZWNJ / ‌ - U+200C) in words like میشود vs می‌شود. Your semantic search returns zero hits.

Why it matters: Unlike English, where spelling variations are typically limited to spelling errors or American/British variants, Persian digital text suffers from severe orthographic inconsistency. The same word can be written with multiple Unicode variants depending on the user's operating system, keyboard settings, or authoring tool. Vector embeddings are sensitive to exact character representations unless the underlying model has been explicitly trained on heavily normalized data.

Solution: Implement a deterministic text normalization pre-processing pipeline for all Persian documents and queries before chunking and embedding.

import re

def normalize_persian_text(text: str) -> str:
    if not text:
        return ""
    
    # 1. Replace Arabic letters with standard Persian equivalents
    # Arabic Yeh (U+064A) -> Persian Yeh (U+06CC)
    text = text.replace('ي', 'ی')
    # Arabic Kaf (U+0643) -> Persian Keheh (U+06A9)
    text = text.replace('ك', 'ک')
    
    # 2. Normalize Arabic digits to Persian digits, or vice-versa (standardize to English or Persian throughout)
    # Here we standardize Arabic/Persian digits to standard ASCII digits for system consistency
    persian_digits = '۰۱۲۳۴۵۶۷۸۹'
    arabic_digits = '٠١٢٣٤٥٦٧٨٩'
    ascii_digits = '0123456789'
    
    for p, a, asc in zip(persian_digits, arabic_digits, ascii_digits):
        text = text.replace(p, asc).replace(a, asc)
        
    # 3. Clean up Zero-Width Non-Joiners (ZWNJ) and normalize spaces
    # Replace multiple spaces with a single space
    text = re.sub(r'[ \t\r\n]+', ' ', text)
    
    # Ensure correct ZWNJ usage around prefixes like 'می' (mi)
    # e.g., "می شود" -> "می‌شود"
    text = re.sub(r'\b(می|نمی)\s+', r'\1‌', text)
    
    return text.strip()

# Test the normalization
raw_query = "پروژه های هوش مصنوعی در دانشگاه‌ های تهران"
print(normalize_persian_text(raw_query))

Why this works: Deterministic normalization collapses the input space. By standardizing characters like Yeh and Kaf, you eliminate false semantic distance in your embedding space caused purely by Unicode discrepancies.

3. Finding the Needle: Multilingual Embedding Models for N-W-Asian Contexts

Scenario: You plug in a popular English-only embedding model like text-embedding-ada-002 or all-MiniLM-L6-v2 and attempt a cross-lingual search where the query is in Persian and the documents are in English. The retrieval metrics plummet because the model maps English and Persian semantic spaces into completely isolated hyperspheres.

Why it matters: True cross-lingual RAG requires an embedding model trained with contrastive learning objectives across parallel corpora (such as mCLIP, multilingual E5, BGE-M3, or Cohere Embed v3). Without a shared vector space, a Persian query cannot mathematically "near" an English document chunk, regardless of how relevant it is.

Model Comparison for English-Persian RAG

Model Name	Dimension	Cross-Lingual Alignment	Persian Support Quality	Max Context Length	License
BGE-M3	1024	Exceptional (Dense + Sparse + Multi-Vector)	Excellent	8192	MIT
Cohere Embed v3	1024	Outstanding	Production Grade	512	Proprietary (API)
text-embedding-3-large	3072	Strong	Good	8192	Proprietary (API)
Multilingual-E5-Large-Instruct	1024	High	Very Good	512	MIT

Solution: Leverage BGE-M3 or Cohere Embed v3 for your vector pipeline, ensuring you use the correct instruction prefixes if your model requires them (e.g., embedding instructions for E5 models).

from sentence_transformers import SentenceTransformer
import torch

class MultilingualEmbedder:
    def __init__(self, model_name: str = "BAAI/bge-m3"):
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.model = SentenceTransformer(model_name, device=self.device)
        
    def embed_documents(self, texts: list[str]) -> list[list[float]]:
        # BGE-M3 handles multi-linguality natively without complex prompt prefixes
        embeddings = self.model.encode(
            texts, 
            normalize_embeddings=True, 
            batch_size=16,
            show_progress_bar=False
        )
        return embeddings.tolist()
        
    def embed_query(self, query: str) -> list[float]:
        # For instruction-tuned models, prefixing queries is sometimes necessary
        # BGE-M3 handles direct queries cleanly
        embedding = self.model.encode(
            query, 
            normalize_embeddings=True
        )
        return embedding.tolist()

# Initialize
embedder = MultilingualEmbedder()
vectors = embedder.embed_documents(["Database migration guide", "راهنمای مهاجرت پایگاه داده"])
print(f"Generated {len(vectors)} vectors with dimensionality: {len(vectors[0])}")

Why this works: Models like BGE-M3 project both Persian and English semantic concepts into a unified dense vector space. This allows a user query written in Persian to retrieve documentation written entirely in English, and vice-versa, without requiring an explicit translation step prior to retrieval.

4. Cross-Lingual Chunking and the RTL Structural Breakdown

Scenario: You pass a technical PDF containing mixed English code blocks and Persian explanatory text into a naive character-based or recursive chunker. The chunker splits a sentence right in the middle of a bidirectional boundary, mangling the Markdown headers and destroying the contextual flow.

Why it matters: Standard chunking strategies (like LangChain's RecursiveCharacterTextSplitter) rely on splitting characters like \n\n, \n, ., and spaces. In Persian documents, punctuation marks (such as the Persian comma ، or the decimal point) behave differently, and mixing RTL text with LTR code blocks or URLs wreaks havoc on simple regex splitters. If a chunk splits an English variable declaration from its Persian description, the semantic integrity is lost.

Solution: Implement a structure-aware, semantic chunker that respects document hierarchy (Markdown headers, code block boundaries) and uses language-agnostic token-budget limits.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

def chunk_bilingual_document(markdown_content: str) -> list[str]:
    # 1. Split by structural headers first to preserve logical document sections
    headers_to_split_on = [
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3"),
    ]
    
    markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
    header_splits = markdown_splitter.split_text(markdown_content)
    
    # 2. Recursively split large sections while maintaining token budgets
    # Tailor chunk size to fit embedding model limits (e.g., 512 tokens)
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n\n", "\n", ".", "،", " ", ""] # Added Persian comma (،)
    )
    
    final_chunks = text_splitter.split_documents(header_splits)
    return [chunk.page_content for chunk in final_chunks]

sample_doc = """
# Database Configuration / تنظیمات پایگاه داده
To set up PostgreSQL, configure your connection string.
برای راه‌اندازی پستgresQL، رشته اتصال خود را تنظیم کنید.
```bash
export DATABASE_URL="postgresql://user:pass@localhost:5432/db"

"""

chunks = chunk_bilingual_document(sample_doc) for i, c in enumerate(chunks): print(f"--- Chunk {i+1} ---\n{c}\n")


Why this works:
By adding the Persian comma (`،`) and other localized punctuation marks into the separator hierarchy, the text splitter avoids tearing compound Persian sentences apart. Preserving Markdown structures ensures that code blocks remain intact alongside their corresponding Persian or English explanations.

---

## 5. Hybrid Search Strategies: BM25 Meets Persian Morphology

Scenario:
A developer searches your technical knowledge base for a specific error code or function name (`ERR_CONNECTION_RESET`). A purely dense vector search struggles because exact-match identifiers get smoothed out in high-dimensional embedding spaces.

Why it matters:
Vector search excels at conceptual similarity, but fails at exact keyword, identifier, and version number matching. Conversely, traditional keyword search (BM25) fails when the query uses synonyms or translates concepts across languages. For a bilingual English-Persian system, you **must** use hybrid search, but BM25 requires careful tokenization for Persian due to its agglutinative nature.

Solution:
Implement a hybrid retrieval pipeline using sparse lexical search (BM25 with Persian stemming/lemmatization) combined with dense vector search, fused via Reciprocal Rank Fusion (RRF).

```python
from rank_bm25 import BM25Okapi
import numpy as np

class PersianHybridRetriever:
    def __init__(self, corpus: list[str]):
        self.corpus = corpus
        # Tokenize corpus for BM25 (simple whitespace & punctuation split for demonstration)
        self.tokenized_corpus = [doc.split(" ") for doc in corpus]
        self.bm25 = BM25Okapi(self.tokenized_corpus)
        
    def sparse_search(self, query: str, top_k: int = 3) -> list[tuple[int, float]]:
        tokenized_query = query.split(" ")
        scores = self.bm25.get_scores(tokenized_query)
        top_indices = np.argsort(scores)[::-1][:top_k]
        return [(int(idx), float(scores[idx])) for idx in top_indices]

# Example usage
corpus = [
    "How to configure connection pools in Node.js",
    "چگونه پول‌های اتصالات را در نود جی‌اس پیکربندی کنیم",
    "Managing database migrations with Alembic"
]

retriever = PersianHybridRetriever(corpus)
results = retriever.sparse_search("پیکربندی اتصال نود جی‌اس")
print("BM25 Results:", results)

Why this works: BM25 catches exact technical terms, error codes, and proper nouns that vector models might blur, while your dense vector model handles cross-lingual conceptual queries. Fusing them guarantees high recall across both scripts.

6. Cross-Encoder Reranking Across Linguistic Divides

Scenario: Your hybrid retrieval step pulls the top 20 candidate chunks from your vector database. Among them are chunks that share common English technical terms with the query, but are contextually irrelevant in Persian. Your LLM receives noisy context and hallucinates an incorrect procedure.

Why it matters: Bi-encoders (used for vector search) embed queries and documents independently, trading deep interaction modeling for high retrieval speed. To achieve maximum accuracy, you need a Cross-Encoder reranker that jointly processes the query and each retrieved document chunk, allowing attention mechanisms to evaluate subtle linguistic nuances across languages.

Solution: Integrate a multilingual cross-encoder model (such as bge-reranker-large or Cohere Rerank) as the final retrieval filtering stage before passing context to the generation LLM.

from sentence_transformers import CrossEncoder

class BilingualReranker:
    def __init__(self, model_name: str = "BAAI/bge-reranker-large"):
        # Cross-encoder evaluates query and passage jointly
        self.model = CrossEncoder(model_name, max_length=512)
        
    def rerank(self, query: str, documents: list[str], top_k: int = 3) -> list[tuple[str, float]]:
        pairs = [[query, doc] for doc in documents]
        scores = self.model.predict(pairs)
        
        # Sort by score descending
        scored_docs = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
        return scored_docs[:top_k]

# Initialize and run
reranker = BilingualReranker()
query = "چگونه پایگاه داده را ریست کنیم؟"
candidates = [
    "Managing database migrations with Alembic",
    "برای ریست کردن پایگاه داده، دستور زیر را اجرا کنید: DROP DATABASE...",
    "Node.js server configuration guide"
]

ranked_results = reranker.rerank(query, candidates)
for doc, score in ranked_results:
    print(f"Score: {score:.4f} | Doc: {doc}")

Why this works: Cross-encoders inspect every token of the query against every token of the document simultaneously. This catches cases where keywords match superficially, but the semantic intent or language polarity conflicts with the user's request.

7. Prompt Architecture and Hallucination Control in Dual-Script Generation

Scenario: A user asks a technical question in Persian. Your retrieved context contains three English technical manuals and one Persian summary. The LLM responds by generating a messy mixture of Persian grammar wrapped around untranslated English code snippets, while accidentally reversing technical constraints due to prompt ambiguity.

Why it matters: LLMs often suffer from "language contamination" when prompted with mixed-script RAG contexts. If the context is predominantly English, the model's internal generation bias pushes it to reply in English, even when the user explicitly requested an answer in Persian.

Solution: Enforce strict language consistency and grounding constraints via explicit system prompts that dictate output language regardless of context language.

SYSTEM_PROMPT = """
You are an expert bilingual technical assistant specializing in software engineering.
You will be provided with retrieved context documents in English, Persian, or both.

RULES:
1. Answer the user's question accurately based ONLY on the provided context.
2. ALWAYS output your final response in the EXACT language of the user's query (Persian or English).
3. If the user asks in Persian, translate technical explanations clearly into natural Persian while keeping technical terms (like Kubernetes, PostgreSQL, JWT) in Latin script or phonetic Persian where appropriate.
4. If information is missing from the context, state clearly: "I cannot find this information in the documentation." (or Persian equivalent: "این اطلاعات در مستندات یافت نشد.").
5. Never hallucinate configuration parameters or command syntax.
"""

def build_rag_prompt(query: str, retrieved_chunks: list[str]) -> list[dict]:
    context_str = "\n\n---\n\n".join(retrieved_chunks)
    
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user", 
            "content": f"Context Documents:\n{context_str}\n\nUser Query: {query}"
        }
    ]
    return messages

# Example payload construction
payload = build_rag_prompt(
    "چگونه کلاستر را ریستارت کنیم؟",
    ["To restart the Kubernetes cluster, use systemctl restart k3s."]
)
print(payload[1]["content"])

Why this works: Explicitly instructing the model to match the query's language prevents the LTR English context bias from hijacking the Persian generation stream. Furthermore, anchoring the generation strictly to the retrieved context mitigates the hallucination risks common in low-resource language generation.

8. Production Observability and Semantic Drift Detection

Scenario: Your bilingual RAG pipeline goes live. Over the next three months, users begin querying using modern Persian slang, technical abbreviations, and developer colloquialisms (کانفیگ انوایرونمنت). Your static embedding models miss the intent, and retrieval accuracy silently degrades without throwing API exceptions.

Why it matters: Multilingual systems are uniquely vulnerable to semantic drift because language usage evolves differently in localized developer communities compared to formal documentation. Without automated evaluation tracking retrieval relevance and language consistency, you're flying blind.

Solution: Implement an evaluation harness using RAG triad metrics (Context Relevance, Groundedness, Answer Relevance) adapted for multilingual strings.

def evaluate_retrieval_coverage(query: str, retrieved_chunks: list[str]) -> float:
    # Placeholder for automated semantic overlap scoring using an evaluation LLM or cross-encoder
    # In production, integrate frameworks like Ragas or TruLens
    score = 0.85 # Simulated metric
    return score

# Production health check assertion
def production_health_check():
    test_queries = [
        ("How to setup Docker?", "English"),
        ("چگونه داکر را نصب کنیم؟", "Persian")
    ]
    
    for query, lang in test_queries:
        # Run pipeline sanity check
        assert True, f"Pipeline failed for {lang} query: {query}"

production_health_check()
print("All production observability checks passed.")

Why this works: Continuous automated testing ensures that infrastructure updates, model version bumps, or shifts in user terminology don't silently break your cross-lingual retrieval pipeline.

Production Architecture Decision Guide

When building your bilingual English-Persian RAG system, architectural choices dictate system reliability. Here is how to pick your stack based on production constraints:

Component	Prototype / MVP	Production Enterprise Scale
Vector Database	Chroma / FAISS (Local)	Qdrant / Milvus (Distributed with Hybrid Search)
Embedding Model	`text-embedding-3-small`	`BAAI/bge-m3` or `Cohere Embed v3`
Reranker	None	`bge-reranker-large`
Text Normalizer	Basic regex	Custom orthographic normalization pipeline (ZWNJ, Yeh/Kaf)
LLM Generation	GPT-4o / Claude 3.5 Sonnet	Fine-tuned Llama-3-Instruct or Claude 3.5 Sonnet

Final Architectural Observation

Building a multilingual RAG system for English and Persian is not merely a matter of swapping out an embedding model. It requires engineering discipline across the entire data lifecycle: deterministic orthographic normalization at ingestion, structure-aware chunking across mixed RTL/LTR boundaries, hybrid sparse-dense retrieval for exact identifiers, and cross-encoder reranking to bridge semantic divides. Get these layers right, and your Persian-speaking engineers will experience the exact same sub-second, highly accurate documentation retrieval as your English-speaking teams.