Text Synth - Technical Notes
Text Synth is build using the GPT-2 language model released by OpenAI. It is a neural network of 1.5 billion parameters based on the Transformer architecture.
GPT-2 was trained to predict the next word on a large database of 40 GB of internet texts.
This implementation is original because instead of using a GPU, it runs using only 4 cores of a Xeon E5-2640 v3 CPU at 2.60GHz. With a single user, it generates 10 tokens per second. It is programmed in plain C using the LibNC library. A Linux executable can be downloaded here.
Thanks to OpenAI for providing their GPT-2 model.News:
- 2020-08-05: the model now uses the following sampling parameters: top-p=0.9, top-k=40 and temperature=1.0.