TextSynth Server

1 Introduction
2 Quick Start
- 2.1 Linux
  - 2.1.1 First steps
  - 2.1.2 GPU usage (commercial version only)
- 2.2 Windows
3 Utilities
4 License

1 Introduction

TextSynth Server is a web server proposing a REST API to large language models. They can be used for example for text completion, question answering, classification, chat, translation, image generation, ...

It has the following characteristics:

Supports many Transformer variants (GPT-J, GPT-NeoX, GPT-Neo, OPT, Fairseq GPT, M2M100, CodeGen, GPT2, T5, RWKV, LLAMA) and Stable Diffusion.
Integrated REST JSON API for text completion, translation and image generation.
Very high performance for small and large batches on CPU and GPU. Support of dynamic batching to handle a large number of simultaneous requests.
Efficient custom 8 bit and 4 bit quantization.
Larger models work optimally on lower cost GPUs (e.g. RTX 3090, RTX A6000) thanks to efficient quantization.
All is included in a single binary. Very few external dependencies (Python is not needed) so installation is easy.
Uses the LibNC library for simple tensor manipulation using the C language.
Simple command line tools (ts_test, ts_sd) are provided to test the various models.

The CPU version is released as binary code under the MIT license. The GPU version and the tools to convert and quantize the models are commercial software.

2 Quick Start

2.1 Linux

2.1.1 First steps

The TextSynth Server works only on x86 CPUs supporting AVX2 (all Intel CPUs since 2013 support it). The installation was tested on Fedora and CentOS/RockyLinux 8 distributions. Other distributions should work provided the libjpeg and libmicrohttpd libraries are installed.

Install the libjpeg and libmicrohttpd libraries. If you use Fedora, RHEL, CentOS or RockyLinux, you can type as root:
```
  dnf install libjpeg libmicrohttpd
```
ts_test can be used without these libraries. ts_sd needs libjpeg. ts_server needs libjpeg and libmicrohttpd.
Extract the archive and go into its directory:
```
  tar xtf ts_server-##version##.tar.gz

  cd ts_server-##version##
```
when ##version## is the version of the program.

Download one small example model:

  wget https://www2.bellard.org/models/gpt2_117M.bin

Use it to generate text with the "ts_test" utility:
```
  ./ts_test -m gpt2-117M.bin g "The Linux kernel is"
```
You can use more CPU cores with the -T option:
```
  ./ts_test -T 4 -m gpt2-117M.bin g "The Linux kernel is"
```
The optimal number of cores depends on the system configuration.
Start the server:
```
  ./ts_server ts_server.cfg
```
You can edit the ts_server.cfg JSON configuration file if you want to use another model.
Try one request:
```
  curl http://localhost:8080/v1/engines/gpt2_117M/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "The Linux kernel is", "max_tokens": 100}'
```
The full request syntax is documented at https://textsynth.com/documentation.html.

Now you are ready to load a larger model and to use it from your application.

2.1.2 GPU usage (commercial version only)

You need an Nvidia Ampere, ADA or Hopper GPU (e.g. RTX 3090, RTX A6000 or A100) in order to use the server with cuda 11.x or 12.x installed. Enough memory must be available to load the model.

First ensure that it is working on CPU (See First steps).

Then try to use the GPU with the ts_test utility:

  ./ts_test --cuda -m gpt2-117M.bin g  "The Linux kernel is"

Then edit the ts_server.cfg configuration to enable GPU support by uncommenting
```
  cuda: true
```
and run the server:
```
  ./ts_server ts_server.cfg
```

Assuming you have curl, Try one request:

  curl http://localhost:8080/v1/engines/gpt2_117M/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "The Linux kernel is", "max_tokens": 100}'

2.2 Windows

The TextSynth Server works only on x86 CPUs supporting AVX2 (all Intel CPUs since 2013 support it). The Windows support is experimental.

Extract the ZIP archive, launch the shell and go into its directory:
```
  cd ts_server-##version##
```
when ##version## is the version of the program.

Download one small example model:

  wget https://www2.bellard.org/models/gpt2_117M.bin

Use it to generate text with the "ts_test" utility:
```
  ts_test -m gpt2-117M.bin g "The Linux kernel is"
```
You can use more CPU cores with the -T option:
```
  ts_test -T 4 -m gpt2-117M.bin g "The Linux kernel is"
```
The optimal number of cores depends on the system configuration.
Start the server:
```
  ts_server ts_server.cfg
```
You can edit the ts_server.cfg JSON configuration file if you want to use another model.
Assuming you installed curl (you can download it from https://curl.se/windows/), try one request:
```
  curl http://localhost:8080/v1/engines/gpt2_117M/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "The Linux kernel is", "max_tokens": 100}'
```
The full request syntax is documented at https://textsynth.com/documentation.html.

Now you are ready to load a larger model and to use it from your application.

3 Utilities

Here are some examples with the utilities:

Text generation

./ts_test -m gpt2_117M.bin g "Hello, my name is"

Short text compression and decompression

./ts_test -m gpt2_117M.bin cs "Hello, how are you ?"

./ts_test ds "##msg##"

where ##msg## is the compressed message.

Translation

./ts_test -m m2m100_1_2B_q8.bin translate en fr "The dispute focuses \
on the width of seats provided on long-haul flights for economy \
passengers."

assuming you downloaded the m2m100_1_2B_q8.bin model.

Text to image

./ts_sd -o out.jpg "an astronaut riding a horse"

assuming you downloaded sd-v1-4.bin.

4 License

The CPU version of the TextSynth Server software is provided as binary code under the MIT license (see the LICENSE file). The GPU version is commercial software. Please contact us for the exact licensing terms.