How to Test Multilingual LLMs Across Markets, Dialects, and Tasks

Start with markets, not a language list

A multilingual benchmark should begin with deployment decisions, not a spreadsheet of ISO language codes. Spanish in Mexico, Spain and Argentina represents three markets with different vocabulary, politeness norms, regulations and customer expectations. Arabic requires choices across Modern Standard Arabic and regional varieties such as Egyptian, Gulf and Moroccan Arabic. Even English changes materially between Britain, India, Nigeria and the United States. Define each test unit as a combination of market, language variety, user group and product task. That framing reveals where failure creates commercial, legal or safety risk.

Build a coverage matrix using actual traffic forecasts, revenue exposure and task criticality. A practical first release might cover eight markets, three user segments and ten high-volume tasks, rather than claiming support for 25 languages with shallow testing. Allocate more examples to consequential workflows: a medical symptom assistant may need 500 reviewed cases per priority market, while a low-risk slogan generator can be assessed with 100. Include expected input languages, acceptable output varieties, scripts, code-switching patterns and literacy levels. The benchmark should mirror how people communicate, including Romanised Hindi, mixed French-Arabic messages in North Africa and informal abbreviations in Brazilian Portuguese.

Set explicit release gates before collecting data. For example, require at least 90 per cent task completion, fewer than 2 per cent severe safety failures and no statistically significant performance gap greater than five percentage points between the primary and lowest-performing priority market. Thresholds will vary, but they force teams to distinguish essential reliability from cosmetic polish. A model that sounds elegant in Parisian French but repeatedly misunderstands Canadian tax terminology is not globally capable; it is locally fluent in the wrong place.

Design tasks around real user journeys

Generic question-answering tests rarely predict product performance. Build scenarios from support logs, search queries, failed conversations and interviews with regional users. A banking assistant should be tested on card disputes, transfer limits, identity checks and fraud warnings, not merely on translated trivia. An ecommerce model needs to understand local sizes, currencies, delivery conventions and return policies. Ask whether the model can complete the journey, request missing information and avoid inventing policies. Fluency is useful only when it advances the user’s goal.

Use a balanced task portfolio. Roughly 40 per cent of cases can represent frequent routine requests, 30 per cent difficult but plausible requests, 20 per cent adversarial or safety-sensitive cases and 10 per cent emerging needs. Include single-turn prompts, multi-turn conversations, long documents, retrieval-assisted answers and tool calls. A Japanese travel assistant might need to interpret an indirect complaint, compare rail passes and produce a booking-ready itinerary. A German support bot may need to extract a serial number from noisy text before selecting the correct warranty process.

Preserve comparability without forcing every market into identical prompts. Maintain a shared core of conceptually equivalent cases, then add market-specific modules. The core allows cross-market analysis; local modules measure fitness for use. A United Kingdom payroll question about PAYE should not be mechanically translated for France, where the relevant institutions and terminology differ. Test the same underlying capability—explaining deductions—through locally valid scenarios. Record whether each item is translated, transcreated or independently authored so that analysts do not mistake content differences for model differences.

Separate translation quality from native-language performance

Translation is one component of multilingual quality, not a proxy for it. Evaluate it in both directions and across content types: conversational text, legal notices, product instructions, marketing copy and structured fields. Score adequacy, meaning preservation, terminology, register, grammar and formatting separately. A translation can be grammatically polished yet dangerous if it drops a negation from dosage instructions. Conversely, a slightly awkward support response may still be operationally correct. Weight dimensions according to risk rather than averaging them into a flattering headline score.

Use professionally translated references where a stable answer exists, but avoid treating one reference as the only valid output. Two expert translators can produce equally accurate sentences with different syntax. Combine reference-based metrics such as chrF or COMET with blinded human review and targeted error annotation. Automatic scores are efficient for regression testing across thousands of items, while reviewers detect mistranslated names, inappropriate honorifics and context errors. For high-risk domains, require adjudication by a subject specialist and a native-language reviewer rather than relying on bilingual generalists alone.

Test native-language reasoning independently by authoring prompts directly in the target variety. Translation-derived datasets often retain English assumptions, sentence structures and cultural references. Compare performance on parallel prompts, independently authored prompts and translated prompts; a large gap signals dataset bias or weak local competence. Include cross-lingual operations as well: summarising a Polish document in English, answering a Portuguese question from a Spanish source, or preserving Arabic names when generating French text. These tasks expose failures hidden by monolingual evaluation.

Measure cultural nuance, terminology and dialects

Cultural evaluation should focus on observable behaviour rather than stereotypes. Test forms of address, levels of directness, humour, taboo subjects, date formats, units and assumptions about households or institutions. A model should know when German ‘Sie’ is appropriate, distinguish Japanese business politeness from casual speech and avoid treating every family as a nuclear household. Reviewers should flag responses that are technically correct but socially implausible, such as an aggressively direct debt reminder in a market where face-saving language is expected.

Create controlled terminology sets for each domain and region. In healthcare, legal and financial services, one incorrect term can change the meaning of an answer. Maintain glossaries with preferred, prohibited and context-dependent forms, then test both generation and comprehension. Canadian French insurance language should not default to terminology used in France. Spanish ‘computadora’, ‘ordenador’ and ‘computador’ carry regional signals even when each is understood. Measure term accuracy, consistency across turns and the model’s ability to explain specialist concepts in plain language.

Dialect testing needs natural data and careful labelling. Recruit speakers from multiple age groups, cities and socioeconomic backgrounds; do not ask one reviewer to represent an entire region. Include spelling variation, colloquialisms, phonetic transcription, code-switching and speech-recognition noise where relevant. Report results by dialect cluster and task rather than collapsing them into a single Arabic or English score. A 92 per cent average can conceal 97 per cent performance on formal language and 72 per cent on a dialect used by millions of customers.

Stress-test safety in local context

Safety policies must travel across languages without becoming culturally blind. Translate the underlying risk taxonomy, but localise the scenarios, entities and euphemisms people actually use. Test self-harm, harassment, fraud, weapons, sexual content, medical advice and political manipulation in formal language, slang, misspellings and code-switched prompts. Attackers often exploit lower-resource languages because safeguards are less robust there. Evaluate both over-refusal and under-refusal: a model that blocks legitimate reproductive-health information can harm users as surely as one that provides unsafe instructions.

Severity-weighted scoring is more informative than raw refusal rates. Assign greater penalties to actionable harmful guidance, disclosure of personal data or confident medical misinformation than to mild tone errors. Run direct prompts, role-play attacks, encoded requests and multi-turn escalation. A user may begin with an innocent chemistry question before seeking dangerous synthesis instructions. Measure whether the system recognises the trajectory, offers safe alternatives and preserves the user’s language rather than abruptly switching to English at the moment support matters most.

Local experts should review policy edge cases. Laws and social risks differ across markets, especially around elections, protected characteristics, medicines and financial promotions. The objective is not to make the model enforce every local norm uncritically, but to identify where product policy, legal obligations and user welfare intersect. Keep a red-team suite that changes quarterly, and reserve at least 20 per cent of it as a hidden set. If development teams repeatedly tune against a public list, the benchmark will measure memorisation rather than resilience.

Build a defensible human-evaluation operation

Human review is the costliest part of multilingual testing and the easiest to weaken. Recruit native or near-native evaluators who live in, or have recent experience of, the target market. Screen them with task-specific calibration rather than credentials alone. Give reviewers a rubric with anchored examples for scores from one to five, definitions of critical errors and guidance on acceptable variation. Separate linguistic reviewers from domain experts when necessary, then use adjudication for disagreements involving both language and factual accuracy.

Measure reviewer reliability. Double-score at least 20 per cent of items during early rounds and 10 per cent once the programme stabilises. Track Krippendorff’s alpha or another suitable agreement statistic by market and dimension; values below about 0.67 warrant investigation for consequential judgements. Low agreement may indicate ambiguous prompts, poor rubric design or genuine cultural plurality. Do not simply discard dissent. Reviewer comments often reveal that a supposedly universal ‘correct’ answer is valid only for one region or social group.

Protect evaluators and data subjects. Remove unnecessary personal information from logs, obtain consent for sensitive research data and provide wellbeing support for reviewers exposed to harmful content. Pay rates should reflect local expertise and task complexity, not merely local wage arbitrage. Record provenance, licence, demographic coverage and annotation history for every dataset. A benchmark that cannot explain where its examples came from, who labelled them and under what conditions is difficult to audit and risky to reuse.

Score gaps, uncertainty and production drift

Publish a scorecard that keeps dimensions visible: task success, factuality, language quality, terminology, cultural fit, safety and latency. Avoid a single composite score unless stakeholders can inspect its weights. Report sample sizes and confidence intervals. If one model scores 86 per cent and another 84 per cent on 120 examples, the apparent lead may be noise. Use paired tests where models answer the same items, and calculate market-level gaps so improvements in high-volume languages cannot mask regressions elsewhere.

Slice results by dialect, task, risk level, input length and interaction type. Then inspect error clusters, not just averages. Ten failures caused by incorrect currency conversion require a different fix from ten caused by misunderstood slang. Maintain a severity ledger linking each failure to probable causes such as retrieval, prompting, model capability, tool integration or policy. This turns evaluation into an engineering system. Every major defect should produce a regression test, while redundant or low-value items should be retired to prevent the suite from becoming static and predictable.

Benchmarking must continue after launch. Sample production conversations by market, subject to privacy controls, and compare them with the test distribution. Monitor refusal rates, escalation rates, user corrections, language switching and unresolved sessions. Trigger review when a metric moves beyond a predefined tolerance, such as a three-percentage-point drop in task success or a doubling of severe incidents. Re-run priority suites after model, prompt, retrieval or policy changes. Regional events, new slang and regulatory updates create drift even when the underlying model remains unchanged. A credible multilingual programme is therefore a maintained measurement infrastructure, not a one-off certification exercise.