Language models trained only on Polish from 1800 to 1918
Wieszcz-XIX is a corpus of 3.1 billion words of Polish published between 1800 and 1918, and language models trained from scratch on it alone. The corpus was audited for text later than 1918, and the models write in the language and spelling of their period.
- words in the corpus
- 3.1B
- documents
- 294,369
- years of publication
- 1800–1918
- parameters in the largest model
- 349M
About
Large language models learn from text gathered across the internet, so they write about the nineteenth century in present-day language and with present-day knowledge. Wieszcz learned only from print published up to 1918. It knows the vocabulary, spelling and facts of those texts.
Models like these can serve research on the language and literature of the period. On the same corpus we also measure how model quality depends on size when no more data can be added.
The project is independent and is run by Szymon Kocur.
Goals
- A corpus of period Polish
- We collect Polish texts published up to 1918, clean them and release them with a ledger that records the source and rights basis of every document.
- A model to talk to
- We want to train a model that answers anyone’s question in the language of the period, with no modern vocabulary.
- More knowledge from the same data
- The amount of period text is finite, so we test how model design and training affect how much of it a model retains.
- Open releases
- We publish code, weights, data and measurements together with a description of known defects: OCR errors, a residue of later text and the prejudices of the period.
Corpus
The corpus has 3.1 billion words in 294,369 documents, most of them periodical issues. The texts come from library scans held by the Internet Archive, and 2,714 documents are transcriptions from Wolne Lektury.
We removed OCR noise and near-identical duplicates. Each source keeps its own spelling. The provenance ledger gives the source and rights basis of every document.
The bound of the corpus is the publication date in the library catalogue, then audited by content. The Wolne Lektury part still holds 90 documents that are certainly later text, 0.04% of the corpus.
Models
We trained three models of one design, with 47, 107 and 349 million parameters. All learned from the same corpus, for the same number of steps and from the same seed, so they differ only in size.
They are base models, which means they continue a given text. The largest is on Hugging Face, and the weights of all three are on Zenodo.
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "SKocur/wieszcz-xix-349m"
tok = AutoTokenizer.from_pretrained(name)
net = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True)Paper
The corpus, the models, the audit of the time bound and the measurement of bias are described in a preprint:
Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch