Language models trained only on Polish from 1800 to 1918

Wieszcz-XIX is a corpus of 3.1 billion words of Polish published between 1800 and 1918, and language models trained from scratch on it alone. The corpus was audited for text later than 1918, and the models write in the language and spelling of their period.

words in the corpus
3.1B
documents
294,369
years of publication
1800–1918
parameters in the largest model
349M

About

Large language models learn from text gathered across the internet, so they write about the nineteenth century in present-day language and with present-day knowledge. Wieszcz learned only from print published up to 1918. It knows the vocabulary, spelling and facts of those texts.

Models like these can serve research on the language and literature of the period. On the same corpus we also measure how model quality depends on size when no more data can be added.

The project is independent and is run by Szymon Kocur.

Goals

A corpus of period Polish
We collect Polish texts published up to 1918, clean them and release them with a ledger that records the source and rights basis of every document.
A model to talk to
We want to train a model that answers anyone’s question in the language of the period, with no modern vocabulary.
More knowledge from the same data
The amount of period text is finite, so we test how model design and training affect how much of it a model retains.
Open releases
We publish code, weights, data and measurements together with a description of known defects: OCR errors, a residue of later text and the prejudices of the period.

Corpus

The corpus has 3.1 billion words in 294,369 documents, most of them periodical issues. The texts come from library scans held by the Internet Archive, and 2,714 documents are transcriptions from Wolne Lektury.

We removed OCR noise and near-identical duplicates. Each source keeps its own spelling. The provenance ledger gives the source and rights basis of every document.

The bound of the corpus is the publication date in the library catalogue, then audited by content. The Wolne Lektury part still holds 90 documents that are certainly later text, 0.04% of the corpus.

Models

We trained three models of one design, with 47, 107 and 349 million parameters. All learned from the same corpus, for the same number of steps and from the same seed, so they differ only in size.

They are base models, which means they continue a given text. The largest is on Hugging Face, and the weights of all three are on Zenodo.

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "SKocur/wieszcz-xix-349m"
tok = AutoTokenizer.from_pretrained(name)
net = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True)

Paper

The corpus, the models, the audit of the time bound and the measurement of bias are described in a preprint:

Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch

Read on arXiv

More

  • Media

    Project description, figures, logo and how to cite.

  • Support

    What support pays for and how to help.

  • Contact

    Addresses for general matters, data and rights in texts.