touati-kamel/Algerian-Youtube-Comments

Algerian YouTube Comments Dataset & Extraction Pipeline

A large-scale, curated, and high-entropy NLP corpus of 55,365 authentic Algerian dialect comments collected from popular Algerian YouTube channels, cleaned through an 18-step NLP preprocessing engine, and classified using Google Gemini AI models to isolate genuine Algerian Darija (الدارجة الجزائرية) and Algerian Arabizi (العرابيزي).

• Hugging Face Hub: touati-kamel/Algerian-Youtube-Comments
• Primary Extraction Script: comments_pipeline.py
• Colab Notebook: comments_colab.ipynb

---

---

Dataset Overview

Attribute	Specification
Total Rows	55,365 verified comments
Total File Size	~15 MB (Compressed Snappy Parquet)
Primary Split	train
Languages / Dialects	Algerian Arabic (arq), Algerian Arabizi (Latin + numbers 3, 7, 9, 5), Algerian Franco-Arabe code-switching
Excluded Content	Modern Standard Arabic (MSA / الفصحى), Metropolitan French, English, copypasta, emojis-only, promotional spam
Minimum Text Length	Strictly > 100 clean characters ($\ge 101$), $\ge 60$ linguistic letters, $\ge 8$ words, $\ge 6$ unique words
Modality	Text (Monolingual / Code-Switched Dialectal)

---

Quickstart (Load with Python)

1. Using Hugging Face datasets

from datasets import load_dataset

# Load the Algerian YouTube Comments dataset
dataset = load_dataset("touati-kamel/Algerian-Youtube-Comments")

# Inspect sample row
print(dataset["train"][0])
# {
#   'comment_id': 'UgyHn5TNV3ECXwBVf1Z4AaABAg',
#   'comment_text': 'Hada youcef avec sa chaine est parmi les premiers youtubeurs algériens...'
# }

print(f"Total verified comments: {len(dataset['train']):,}")

2. Using pandas directly from Parquet

import pandas as pd

df = pd.read_parquet("hf://datasets/touati-kamel/Algerian-Youtube-Comments/data/*.parquet")
print(df.head())

3. Using polars (Blazing Fast)

import polars as pl

df = pl.read_parquet("hf://datasets/touati-kamel/Algerian-Youtube-Comments/data/*.parquet")
print(df.schema)
print(df.select(["comment_id", "comment_text"]).head(5))

---

Linguistic Scope & Target Varieties

Colloquial Algerian digital communication features heavy code-switching, specialized orthography, and two major writing scripts. The pipeline isolates and preserves two authentic dialect categories:

1. Algerian Darija (الدارجة الجزائرية - Arabic Script)

• Native vocabulary, regional idioms, and grammatical markers:
  • Example: "يعطيك الصحة خويا لعزيز محتوى فور بزاف والله غير راك داير حالة صحيت"
  • Example: "راه عندنا 3 سنين وحنا نستناو في الجزء الجديد واسبع"

2. Algerian Arabizi (العرابيزي / Franco-Arabe - Latin Script)

• Phonetic Algerian Arabic written with Latin script and numerical substitutions for Arabic sounds not found in French/English:
  • 3 = ع (Ayn), 7 = ح (Haa), 9 = ق (Qaf), 5 = خ (Khaa)
  • Example: "ya3tik saha khoya wlh ghir rak dayer hala had lvideo chbab bzaf"
  • Example: "c'est top khouya rabi yahfdhek ya3tik essaha sahbi rak m3ellem"

Strict Discard Criteria

To maximize entropy and maintain high dialectal purity for Large Language Model (LLM) pre-training and fine-tuning, the following are discarded:

• Modern Standard Arabic (الفصحى): Generic news comments, formal religious invocations without colloquial phrasing.
• Metropolitan French & English: Pure French or English comments lacking Algerian dialect terms.
• Low-Entropy Spam: Emojis, repeated words (روعة روعة روعة), chain prayers, promotional links.

---

Pipeline Architecture

comments_pipeline.py operates as a high-speed, zero-video-download stream pipeline:

flowchart TD
    A["Target Algerian YouTube Channels"] --> B["Video Enumerator (Metadata Only)"]
    B --> C["Direct InnerTube Comment Extractor\n(Zero Audio/Video Download)"]
    C --> D["18-Step NLP Preprocessing & Sanitizer\n(Stripping Emojis, Tatweel, Links, Typo Normalization)"]
    D --> E{"Length & Entropy Gate\nLength > 100 Chars?\nUnique Words >= 6?"}
    E -- No --> X["Discard (Low Information Density)"]
    E -- Yes --> F["Batch Slicer (25 Comments / Call)"]
    F --> G["Google Gemini Multi-Model AI Classifier\n(gemini-flash / gemini-pro hierarchy)"]
    G --> H{"Is Algerian Darija\nor Arabizi?"}
    H -- No (MSA / French / English / Spam) --> X
    H -- Yes --> I["In-Memory Accumulator"]
    I --> J{"Accumulated >= 50 Verified Comments?"}
    J -- Yes --> K["Epoch Finalizer\n(train_epoch_XXXXXX.jsonl & .parquet)"]
    K --> L["Direct Real-Time Push to Hugging Face Hub\n(touati-kamel/Algerian-Youtube-Comments)"]
    K --> M["Sync Checkpoint Tracker\n(processed_tracker.json)"]

---

18-Step Preprocessing & Quality Filtering

Every comment passes through CommentPreprocessor in comments_pipeline.py:

1. Unicode NFKC Normalization: Resolves composite characters, stylized fonts, mathematical alphanumeric glyphs, and presentation forms.
2. HTML Entity & Tag Decoding: Strips tags (<br>, <a>) and unescapes entities (", &).
3. URL & Domain Removal: Removes HTTP/HTTPS links, bare URLs, and domain patterns (.dz, .com, .fr).
4. Timestamp Scrubbing: Strips video timestamps (02:45, 1:15:30).
5. User Mentions & Hashtags: Strips @channel tags while preserving #tag text as clean words.
6. Phone & WhatsApp Contact Stripping: Removes Algerian domestic numbers (05/06/07...), international prefixes (+213), and raw digit sequences.
7. Unicode Emoji & Pictograph Stripping: Matches and strips all emoji planes (SMP, SIP, TIP, SSP, Dingbats, and variation selectors).
8. ASCII & Text Emoticon Removal: Cleans XD, :), :-D, <3, -_-, ^^, etc.
9. Decorative Glyphs & ASCII Art: Removes decorative symbols, borders, and ASCII banners.
10. Invisible & Control Characters: Eliminates zero-width spaces (\u200b), bidirectional markers (\u202a–\u202e), and control bytes.
11. Keyboard Typo Normalization: Normalizes Persian/Urdu keyboard typos to Arabic (ک -> ك, ی -> ي, ہ -> ه, and Eastern numerals -> standard digits).
12. Tatweel / Kashida Stripping: Removes horizontal Arabic elongation (ـ).
13. Tashkeel / Harakat Removal: Removes diacritics (Fathah, Dammah, Kasrah, Sukun, Tanween).
14. Elongation Spam Compression: Collapses excessive character repetitions to a maximum of 2 (ههههههههه -> هه, وااااااو -> وااو, looooool -> lool).
15. Word Copypasta Collapse: Collapses consecutive identical words (صحيت صحيت صحيت صحيت -> صحيت صحيت).
16. Punctuation Normalization: Reduces punctuation floods (!!!! -> !, ؟؟؟؟ -> ؟).
17. Whitespace Compacting: Collapses non-breaking spaces and linebreaks.
18. Strict Entropy & Length Gate:
    • Length: Clean text must be strictly > 100 characters ($\ge 101$).
    • Letter Density: Must contain at least 60 authentic Arabic or Latin letters.
    • Word Count: At least 8 words.
    • Lexical Diversity: At least 6 distinct unique words (rejects spam repetitions like "روعة روعة روعة...").

---

Source Channels & Topical Diversity

Comments were harvested across major Algerian YouTube channels spanning entertainment, technology, culinary arts, satire, horror stories, and societal discussions:

Channel	Category / Genre
@zarouta_youcef	Satirical comedy, social sketches, street culture
@mimhodz	Social commentary, comedic podcasts, youth culture
@ViperBeyaz	Entertainment, satire, vlogging
@RaoufBelkacemi	Storytelling, gaming, youth discussions
@SIFOUREVIEWS	Technology, smartphones, hardware reviews in Darija
@la3zizdz	Entertainment, reactions, social podcasts
@ilyesderradji	Vlogs, travel, lifestyle
@iAbdouboutaleb	Gaming, community challenges, entertainment
@AymanCha7el / @Aymancha7el2	Daily vlogs, comedy, viral entertainment
@Lola.DzHorror	Horror stories, folklore, community narratives
@Raklita / @RaklitaExtra	Youth culture, comedic reactions, societal trends
@OumWalidcuisine	Traditional & modern Algerian cuisine, household recipes
@cuisine_om_jawad	Algerian pastry, traditional baking, home cooking
@rifkabjmofficiel	Lifestyle, humor, travel vlogs
@Sultan_Achour_10	Algerian historical comedy drama, television series
@TH8Professional	Tech tutorials, software, digital finance in Darija

---

Dataset Schema

Each entry in the dataset conforms to the following schema:

{
  "comment_id": "UgyHn5TNV3ECXwBVf1Z4AaABAg",
  "comment_text": "Hada youcef avec sa chaine est parmi les premiers youtubeurs algériens MAIS AUSSI parmi les plus banales et plus vides de sens.."
}

Field Descriptions

Feature	Type	Description
comment_id	string	Unique YouTube comment identifier (used for deduplication and provenance tracking).
comment_text	string	High-quality, normalized text in authentic Algerian Darija (Arabic script) or Algerian Arabizi (Latin script).

---

Intended Use & Ethics

• Target Applications:
  • Pre-training and fine-tuning open-source LLMs on low-resource North African dialects.
  • Dialect identification (DID) and language identification (LID).
  • Sentiment analysis and opinion mining in Algerian social media contexts.
  • Automatic Speech Recognition (ASR) language model rescoring.
• Privacy & Anonymization:
  • All usernames, author avatars, channel IDs, user tags (@user), phone numbers, and URLs have been removed.
  • The dataset consists exclusively of public comments published on publicly accessible YouTube videos.

---

License & Citation

This dataset and its extraction pipeline are licensed under the MIT License.

@dataset{touati_algerian_youtube_comments_2026,
  author       = {Kamel Touati},
  title        = {Algerian YouTube Comments: A Curated Corpus of Algerian Darija and Arabizi},
  year         = {2026},
  publisher    = {Hugging Face},
  version      = {1.0.0},
  url          = {https://huggingface.co/datasets/touati-kamel/Algerian-Youtube-Comments}
}