dsfsi/setswana-sentiment

DSFSI Setswana Sentiment

A Setswana (ISO 639-3: tsn) Twitter sentiment dataset of 3,555 tweets annotated by three native-speaker annotators. The dataset is released alongside full per-annotation timestamps, language-identification metadata, and per-annotator labels to enable both downstream modelling and annotation-quality research.

Dataset Structure

Three classification splits plus a full config are provided.

Split	Examples	Use
train	2,762	Fine-tuning / training
validation	346	Model selection and hyperparameter tuning
test	346	Final evaluation
full	3,555	All tweets incl. per-annotator labels, metadata, and timestamps

The train/validation/test splits contain only the three core sentiment classes (Positive, Negative, Neutral), stratified 80/10/10 by consensus label with seed 42. Examples whose consensus label is Mixed, Indeterminate, or Disagreement are excluded from these splits but are retained in full.

Which columns to use

• For training / evaluating a sentiment classifier, use:

  • Input: text (cleaned tweet text — see "Text preprocessing" below)
  • Target: consensus_label (values: Positive, Negative, Neutral)

• For annotation-quality / disagreement research, additionally use:

  • ann1_label, ann2_label, ann3_label — individual annotator labels
  • ann1_timestamp, ann2_timestamp, ann3_timestamp — UTC submission times
  • consensus_type — whether the consensus is unanimous, majority, or disagreement

• For language-related filtering, use lid_top1_lang / lid_top1_score and the top2 / top3 fallbacks.

Columns

Column	Type	Description
example_id	string	Unique identifier for the annotation example
batch	int	0 = training/calibration batch; 1–7 = production batches
text	string	Preprocessed tweet text — the field intended as model input
token_count	int	Number of tokens in the tweet
lid_top1_lang	string	Top AfroLID language prediction
lid_top1_score	float	Confidence score for lid_top1_lang
lid_top2_lang	string	Second-best language prediction
lid_top2_score	float	Confidence score for lid_top2_lang
lid_top3_lang	string	Third-best language prediction
lid_top3_score	float	Confidence score for lid_top3_lang
ann1_label	string	Label from annotator 1 (one of: Positive, Negative, Neutral, Mixed, Indeterminate)
ann1_timestamp	string	UTC ISO-8601 timestamp of annotator 1's label submission
ann2_label	string	Label from annotator 2
ann2_timestamp	string	UTC timestamp of annotator 2's submission
ann3_label	string	Label from annotator 3
ann3_timestamp	string	UTC timestamp of annotator 3's submission
consensus_label	string	Majority-vote label. Disagreement if all three annotators disagree
consensus_type	string	unanimous (3/3), majority (2/3), or disagreement (0 majority)
label_id	int	(splits only) Integer id for consensus_label — Positive: 0, Negative: 1, Neutral: 2

Text preprocessing

The text field is already preprocessed: usernames, mentions, URLs, and other potentially identifying information have been replaced with placeholder tokens, hashtags normalised, and casing lowered. The original raw tweet text is not included in this release for privacy reasons.

Label Schema

Label	Description
Positive	The tweet expresses positive sentiment
Negative	The tweet expresses negative sentiment
Neutral	The tweet expresses neutral sentiment
Mixed	The tweet expresses both positive and negative sentiment
Indeterminate	The tweet is unintelligible, in another language, or cannot be confidently categorised

Annotators

Three undergraduate native speakers of Setswana annotated the corpus. Identities are anonymised as Ann.1, Ann.2, and Ann.3 (stable across all records).

Label distribution (full)

Label	Count	% of corpus
Neutral	1,489	41.9%
Negative	1,445	40.6%
Positive	520	14.6%
Disagreement	47	1.3%
Indeterminate	33	0.9%
Mixed	21	0.6%

Inter-annotator agreement (aggregate, Randolph's free-marginal κ): 0.76 ("excellent" by convention).

Quick start

from datasets import load_dataset

# Classification splits (Positive / Negative / Neutral only)
ds = load_dataset("dsfsi/setswana-sentiment")
print(ds["train"][0]["text"], "→", ds["train"][0]["consensus_label"])

# Full dataset with all metadata and per-annotator labels
full = load_dataset("dsfsi/setswana-sentiment", "full")

Intended Use

• Training and evaluating sentiment classifiers for Setswana and related Bantu languages.
• Research on annotator disagreement, annotation-quality monitoring, and temporal effects in multi-annotator campaigns.
• Benchmarking multilingual and African-language pre-trained language models.

Limitations

• The corpus is drawn from Twitter (2021–2022) and reflects the register, topics, and demographics of that platform at that time.
• Negative and Neutral labels dominate the label space (~82% combined); Positive is under-represented.
• The dominant pairwise confusion is Negative vs. Neutral, reflecting the indirect and ironic phrasing common in Setswana political and social commentary — this is a property of the data, not annotator quality.
• Annotations were collected asynchronously; per-tweet inter-annotator time span is a strong predictor of agreement and is reported alongside every example.

License

Released under the NOODL license.

Citation

If you use this dataset, please cite:

@misc{abdulmumin2026temporalsimultaneitypredictsannotation,
      title={Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora}, 
      author={Idris Abdulmumin and Mokgadi Penelope Matloga and Tadesse Destaw Belay and Botshelo Kondowe and Letlhogonolo Mohleleng and Hareaipha Nkopo Letsoalo and Shamsuddeen Hassan Muhammad and Vukosi Marivate},
      year={2026},
      eprint={2605.27239},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.27239}, 
}