Datasets:

SlayerLab
/

polish-dynaword

Tasks:

Text Generation

Languages:

Size:

ArXiv:

Tags:

License:

Dataset card Files Files and versions

polish-dynaword / src

339 kB

8 contributors

History: 24 commits

PiotrSty's picture

Add source: sejm_api — contemporary parliamentary speeches (2023 onward)

503c9db verified 26 days ago

__pycache__
Add source: sejm_api — contemporary parliamentary speeches (2023 onward) 26 days ago
build_dynaword.py

7.91 kB
Release v0.2.1 metadata schema update 3 months ago
clean_hplt_v3.py

9.28 kB
Add source: european_hplt_v3_pl (HPLT v3.0 Polish, top WDS bins, CC0) (#5) about 2 months ago
clean_samorzad_gov_pl.py

19.2 kB
feat(data): add samorzad.gov.pl institutional corpus (#13) 27 days ago
compare_tokenizers.py

1.78 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
count_vocab.py

1.19 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
discover_govpl.py

8.86 kB
Add source: govpl (gov.pl government press releases) (#9) about 2 months ago
eval_bpb.py

2.19 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
fetch_eli.py

5.57 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
fetch_eltec.py

3.07 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
fetch_europeana.py

10.4 kB
Release v0.3.1 Europeana preview shard (#4) 2 months ago
fetch_govpl.py

7.98 kB
Add source: govpl (gov.pl government press releases) (#9) about 2 months ago
fetch_njkp.py

13.4 kB
Add source: nkjp1m 2 months ago
fetch_sejm_api.py

27.6 kB
Add source: sejm_api — contemporary parliamentary speeches (2023 onward) 26 days ago
fetch_wikimedia.py

4.04 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
fetch_wiktionary_examples.py

9.97 kB
feat(data): add wiktionary_examples (8.6k docs / 1.12M tok) (#14) 27 days ago
filter_common_corpus.py

11.2 kB
Add European HPLT Polish web filtering workflow 3 months ago
filter_european_hplt.py

15.7 kB
Shard and parallelize HPLT filtering 3 months ago
govpl_subsites.json

13 kB
Add source: govpl (gov.pl government press releases) (#9) about 2 months ago
launch_hplt_parallel.py

3.31 kB
Add launcher for parallel HPLT filtering 3 months ago
make_docs.py

21.1 kB
feat(data): add samorzad.gov.pl institutional corpus (#13) 27 days ago
make_linkedin_card.py

4.26 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
make_training_mix.py

9.35 kB
Add v0.3 source license review and quality mix workflow 3 months ago
normalize_schema.py

4.22 kB
Release v0.2.1 metadata schema update 3 months ago
pattern_frequency_report.py

8.2 kB
Release v0.2.3 with community HPLT contribution about 2 months ago
plot_version_growth.py

2.31 kB
feat(data): add samorzad.gov.pl institutional corpus (#13) 27 days ago
review_source_candidates.py

5.83 kB
Add v0.3 source license review and quality mix workflow 3 months ago
rights.py

2.15 kB
Release v0.3.1 Europeana preview shard (#4) 2 months ago
samorzad_gov_pl_publishers.json

14 kB
feat(data): add samorzad.gov.pl institutional corpus (#13) 27 days ago
sample_quality.py

2.86 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
sources.py

23 kB
Add source: sejm_api — contemporary parliamentary speeches (2023 onward) 26 days ago
test_europeana_contract.py

1.34 kB
Release v0.3.1 Europeana preview shard (#4) 2 months ago
test_govpl_contract.py

3.9 kB
Add source: govpl (gov.pl government press releases) (#9) about 2 months ago
test_hplt_v3_contract.py

2.97 kB
Release v0.2.3 with community HPLT contribution about 2 months ago
test_samorzad_gov_pl_contract.py

7.43 kB
feat(data): add samorzad.gov.pl institutional corpus (#13) 27 days ago
test_sejm_api_contract.py

8.22 kB
Add source: sejm_api — contemporary parliamentary speeches (2023 onward) 26 days ago
tokenize_shards.py

2.91 kB
Squash: collapse LFS history to reclaim private storage 3 months ago
train_bpe.py

2.63 kB
Squash: collapse LFS history to reclaim private storage 3 months ago