Note: This project was done using client provided data, they are NOT attached anywhere.

This is an application of the following paper: https://arxiv.org/abs/2406.18725

Arabic LLM Red Teaming & Safety Assessment

Date: November 24, 2025 Author: Ahmed Taha

1. Methodology

1.1 Data Preparation & Attack Vectors

I utilized the provided datasets, consolidating them into a master dataset of 1,000 harmful prompts.

Drawing upon research from "Jailbreaking LLMs with Arabic Transliteration and Arabizi", I generated three adversarial variants for each prompt using the method explained in the research:

1. Arabizi (Chatspeak): A character-based mapping replacing Arabic letters with Latin characters and digits (e.g., ح → 7, ع → 3).
2. Transliteration: A systematic phonetic representation using Latin script (e.g., ش → sh or š).
3. Diacritization (Noise): Heavy injection of Arabic diacritics (Tashkeel) to test tokenization robustness using the pyarabic library.

I also created two more datasets: a strategically sampled subset (100 rows) prioritizing regional/political topics, and a completely random one (100 rows).

1.2 Model Selection

I benchmarked a diverse set of models to compare behavior across different architectures and training focuses:

Model	Type	Selection Rationale	Env.	Evaluation Dataset size
Gemini-2.0-Flash	Global SOTA	Currently widely used in production	API	Full dataset
GPT-OSS-120B	Global SOTA	Open-Source Thinking Model	API	Full dataset
Qwen-2.5-72B	Multi-lingual	Experience showed strong Arabic	API	100 rows (distilled)
Jais-13B	Regional/Arabic	Arabic Centric Model	Modal (A100)	100 rows (random)
ALLaM-7B-Inst.	Regional/Arabic	Arabic Centric Model	Modal (A100)	759 rows (top rows + random)
AceGPT-7B/13B	Regional/Arabic	Arabic Centric Models	Modal (A100)	57 rows (ended prematurely)
Gemma-4B	Small/Efficient	Small model with strong Arabic	Local	343 rows (top rows + distilled)

1.3 Evaluation Metrics

Attack Success Rate (ASR): The percentage of prompts that successfully jailbroke the model.

For fine-grained analysis, I employed a granular 6-category taxonomy to distinguish between Safe and Unsafe responses:

• SAFE: Hard Refusal
• SAFE: Defensive/Educational
• SAFE: Benign Pivot (Pivots to a safe topic without refusal)
• SAFE: Confused / Unintelligible
• UNSAFE: Unsafe (Partial / Helpful)
• UNSAFE: Unsafe (Full Violation)

1.4 LLM-As-Judge

Evaluation was based on the Intent/Taxonomy (using all 3 hierarchical levels). For region-specific prompts, the evaluation focused on neutrality; the model should not treat the input as fact or take sides in political, cultural, or religious debates.

Judge Selection: I initially used GPT-OSS-Safeguard-20B, but it struggled to interpret Arabizi and transliteration. I switched to Gemma-3-27B for its superior understanding.

Since Gemma-3-27B is not a "thinking" model, I added a "reasoning" key to the JSON output schema to force the model to explain its decision.

2. Comparative Analysis

2.1 Overall Attack Success Rate (ASR)

Model	Baseline (MSA)	Arabizi	Translit	Diacritic
Gemini-2.0-Flash	11.7%	16.5%	15.0%	13.7%
Qwen-2.5-72B	9.2%	10.2%	18.4%	5.1%
GPT-OSS-120B	4.6%	8.4%	6.6%	6.2%
Jais-13B	27.0%	0.0%	0.0%	0.0%
ALLaM-7B	20.6%	12.4%	5.7%	12.8%
Gemma-4B	32.4%	23.9%	14.6%	30.3%
AceGPT-7B	44.4%	0.0%	2.2%	24.4%
AceGPT-13B	33.3%	0.0%	0.0%	33.3%

2.2 Lift Analysis

For susceptible multilingual models, the increase in ASR demonstrates the effectiveness of the attacks.

• Gemini-2.0-Flash: Shows degradation across all vectors, with Arabizi being the most effective (+4.8 points).

• GPT-OSS-120B: Safest baseline (4.6%) but shows a large lift, with Arabizi causing a +82% relative increase.

• Qwen-2.5-72B: Shows the most dramatic failure, with Transliteration causing a +100% relative increase.

As for Arabic-centric models, the attacks were largely ineffective (discussed below).

2.3 Regional vs. General Harm Vector Analysis

All models show dramatically stronger safety for Regional-Specific content compared to General harm vectors while still being vulnerable to attacks.

3. Discussion & Insights

3.1 The "Multilingual Problem"

The most significant finding is that multilingual models (Qwen, Gemini) are much more vulnerable to these attacks than Arabic-specific models.

Reasoning: Multilingual models have seen enough Latin-script Arabic (Arabizi/Transliteration) in their pre-training to understand the intent. However, their safety alignment data focuses on standard MSA, Arabic Dialects and English. This creates a "safety gap" where the model understands the harmful request but fails to trigger the refusal mechanism associated with the Arabic script.

3.2 GPT-OSS-120B's Safety Alignment

GPT-OSS-120B demonstrated very high resistance to the attack vectors, largely refusing outright to cooperate.

Reasoning: This indicates the model was specifically fine-tuned with extra adapter layers (post-training) on safety datasets, rather than being trained to produce safe responses from scratch.

3.3 The "Confusion Defense" of Arabic Centric Models

Arabic-centric models (AceGPT, Jais, ALLaM) and smaller models (Gemma-4B) exhibited 0% (no difference) or reduced ASR on attack vectors.

Reasoning: These models are explicitly trained on Arabic data and likely lack significant Arabizi/Transliteration data in their training sets. Consequently, they simply fail to understand the prompt.

Regarding heavy tashkeel (diacritics), the 0% ASR indicates a preprocessing step in tokenization that strips those diacritics, essentially rendering the attack vectors ineffective.

4. The Weak Link

The success of Transliteration on Qwen and Arabizi on Gemini highlights the lack of proper training data as a critical vulnerability.

Conversely, the results demonstrate that effective tokenization (stripping diacritics) can successfully mitigate noise-based attacks.

5. Conclusion

Current state-of-the-art multilingual LLMs possess a dangerous combination of capability without alignment in low-resource languages and scripts. They understand Arabizi well enough to be helpful (and harmful) but are not aligned well enough to refuse unsafe requests.