Language & LLMs

What Is ELECTRA?

ELECTRA is a language model pretraining approach that replaces some input tokens with plausible alternatives and trains a discriminator to detect which tokens were replaced. This replaced-token detection task is more sample-efficient than masked language modeling. It allows strong results with less compute than comparable BERT models.