Researched
Pretrained Language Models
GPT (June 2018) and BERT (October 2018) are first trained on huge unlabelled text and then adapted to tasks.
Open in the interactive tree →Pretraining on raw text gave models general language knowledge, and one model could be fine-tuned to many jobs. It is the recipe behind all later large language models.
Prerequisites
Unlocks
- AI for Low-Resource Languages2022-2026
- Large Language Models2022Pretraining on raw text is the recipe of every large language model