Tokens and Stream

We ended the previous entry with a functional but basic chatbot. Now we will make two improvements: we will activate the stream option to be able to receive and view responses little by little, and we will control the size of the context window to avoid exceeding the model’s limit in long conversations.

Context window and tokens

As we converse with the chatbot, we accumulate questions and answers that we resend to the model in subsequent calls. The information we send thus grows as the conversation progresses. We can guess that the amount of information that the model can handle is not infinite, and that at some point a limit must be reached, but what is this limit, and what determines it?

The limit is determined by what is known as the context window, which represents the amount of information that the model can handle in a call, including both the input and the output. The size of the context window is measured in tokens, and depends on the model we are using; for example, GPT-3.5 Turbo has versions with context windows of 4,096 and 16,385 tokens, while the more recent GPT-4 Turbo reaches up to 128,000 tokens.

Tokens are the text segments that the model handles for its processing and generation. Their size is variable, and can range from one character to a complete word. In English, their average size is around 4 characters, but it also changes according to the language, and also the model (the tokenization of GPT-3 was different from that used by later models). OpenAI has a tokenizer tool that shows how this split of text into tokens is performed.

Another reason to pay attention to the tokens is that they determine the cost of API calls. We are charged by number of tokens, both input and output, although output tokens tend to be more expensive.

The default response of the chat completions service provides information about the use of tokens in the call that has just been made, which can be found within the usage field. For example:

"usage": {
    "prompt_tokens": 9,
    "completion_tokens": 31,
    "total_tokens": 40
  }

This information identifies separately the tokens consumed by our call (prompt_tokens), by the response (completion_tokens), as well as the sum of the two (total_tokens).

The information from usage, in principle, could serve us to maintain control of the conversation size, but unfortunately it is not always returned. Specifically, if the streaming option is used (which we will see in detail below) the response will not include the information of usage. Therefore, to allow us more granular control, it becomes necessary to have a programmatic way to obtain the number of tokens of a message.

How to count tokens

To understand how tokens are counted, let’s make a simple call to chat completions using curl and analyze the response:

curl https://api.openai.com/v1/chat/completions -H "Authorization: Bearer api-key" -H "Content-Type: application/json" -d "{\"model\":\"gpt-3.5-turbo\",\"messages\":[{\"role\":\"user\",\"content\":\"Identify\"}]}"

{
  "id": "chatcmpl-8oEuxCf3P47tm6a0o9xaVNd0pn8Ie",
  "object": "chat.completion",
  "created": 1706984435,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Identify refers to the act of recognizing or determining the nature or identity of someone or something. It involves identifying and distinguishing the characteristics, features, or attributes that make an individual or object unique or specific. Identification can be achieved through observation, investigation, or analysis of available information or evidence."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 58,
    "total_tokens": 67
  },
  "system_fingerprint": null
}

We can see in usage that our call has consumed 9 tokens, and the response, 58. If we take the content of the response (choices[0].message.content) and pass it through Tokenizer we will see that it calculates exactly the 58 tokens we expect.

Now we do the same with the call message. We enter into Tokenizer separately the role (user) and the content (Identify), and it calculates 1 + 2 = 3 tokens. However, prompt_tokens is 9. Where does this difference come from?

To calculate the tokens of the input message we must add the following:

1. Tokens of each message, including both the role and the content. In our example, the input has a single message of 3 tokens (role + content).
2. Additional fixed cost of each message, which depends on the model. In our case we used gpt-3.5-turbo-0613 (gpt-3.5-turbo, is just an alias that points to the most recent model of the family), for which this additional fixed cost is 3 tokens.
3. 3 additional tokens that are added to the call, regardless of the type of message and that (as I have seen in the documentation) are independent of the model.

Adding everything up, we reach the 9 tokens that usage tells us.

What we still need to know is how to calculate the tokens of each message (point 1 on the list) from our program. For that, OpenAI provides tiktoken, a library in Python, but which is also ported to other languages, including Node.js. We will start by installing it:

pip install tiktoken
or
npm install tiktoken

I will describe how to work with tiktoken using Python, but its use in Node.js is almost identical. To start, we will import the library:

import tiktoken

To convert messages to tokens we will need to generate an encoding object. There are several ways to obtain it; we can do it by specifying the encoding form corresponding to our model (cl100k_base for all the most modern models), but we can also let tiktoken choose it for us by specifying the model. We will do the latter in the following way:

encoding = tiktoken.encoding_for_model('gpt-3.5-turbo-0613')

The encoding object has a decode method that receives a string of text and returns a list of numeric identifiers, one for each token into which the text is split. The length of this list is the number we are looking for: how many tokens the text consumes:

num_tokens = len( encoding.encode('Counting tokens') )  # num_tokens == 3

A final note about calculating tokens: the fixed number of tokens to add per message (point 2 on the previous list) is different depending on the model, but I have not found it formally documented anywhere what value to use for each one. To identify it I have made the find_fixed_per_message script, which you can find on GitHub for Python or Node.js. It is based on using a list of messages to compare the number obtained from usage and the one calculated with tiktoken, for each text model, to determine the value that the model uses. The result obtained is that all models of the GPT-4 and GPT-3.5-Turbo family add 3 tokens per message, with the exception of gpt-3.5-turbo-0301, which adds 4.

Controlling the window size

Controlling the number of tokens will allow us to know when we are approaching the limit of the window size, but what can we do when this happens? There are multiple strategies that can be adopted, among them:

• Truncation: Shortening the conversation history by eliminating the oldest messages, assuming that as the conversation progresses they will be the least relevant.
• Summarization: Summarizing the conversation, and using it in subsequent exchanges, instead of the complete history.
• Selection by relevance: Analyzing the conversation and keeping a history only with the relevant messages, eliminating the others.

For the new evolution of our grumpy chatbot, we will implement truncation, as it is the simplest strategy, and because with the size of the new context windows, in strictly chatting use, it is more than likely that the oldest messages no longer have anything to do with the conversation. For other uses, for example, translation of extensive texts, this strategy would not be valid and it would be necessary to opt for others.

To start, in addition to the list of messages we already have, we will create a second list (tokens_mensaje) where we will keep the token size we calculate for each of them. Thus, for the message introduced by the user:

tokens_mensaje.append(len(enc.encode('user')) + len(enc.encode(user_text)) + fixed_per_message)

Where fixed_per_message represents the value of 3 tokens (for this model) that we must add to each message.

We will do the same with each of the messages returned by the model. So that in the list tokens_mensaje we will always have the updated amount of tokens that we will send in the next call. We will use this to make a calculation before each call to the API and determine if we still have enough space in the window, and otherwise, we will delete messages from the beginning of the list until we have enough space.

    total_tokens = sum(tokens_mensaje)
    if total_tokens + 3 + max_response > window:
        i = 2
        total = tokens_mensaje[1]
        while total_tokens + max_response + 3 - total > window:
            total += tokens_mensaje[i]
            i += 1
        del messages[1:i]
        del tokens_mensaje[1:i]

One thing to keep in mind is that the context window sets the maximum for the sum of the input and output of the call. That is, it is not enough to ensure that our message history does not exceed the window size, we must also ensure that there is enough space left for the response. And how big should this space be that we leave? It will depend on our use. In this case, I have decided to leave at least enough space to be able to receive the longest response that has been received in the conversation so far. That value of the longest response received is saved in the variable max_response, and we will take it into account in the calculation of messages to delete.

The complete code of the evolved version of the grumpy chatbot is on GitHub for Python and Node.js.

The stream option

If you have used the grumpy chatbot and compared it with ChatGPT you will undoubtedly have noticed that, in addition to having a bad temper… it is slower. In fact, it is not slower, because showing the complete message (of a comparable size) takes something similar, but ChatGPT is writing the message little by little, while our chatbot waits until receiving the complete message and shows it all at once.

Fortunately, the OpenAI API gives us the opportunity to easily incorporate this functionality, calling the chat completions service with the stream option, and the libraries do everything for us. In Python, to gradually display the message it would be enough to do the following:

response = client.chat.completions.create(
    model = model,
    messages = messages,
    stream = True
)

response_text = ''
for piece in response:
    piece_txt = piece.choices[0].delta.content or '\n'
    response_text += piece_txt 
    print(piece_txt, end='', flush=True)

The only change in the call to the service is to add the parameter stream = True. After that, we just need to iterate through the response, which we’ll display, and also store to be able to finally save it in the message history of the conversation. Be aware that the last piece of the message will have a content value of None, which needs to be controlled to avoid errors (in the example, it’s just changed to a newline).

In Node.js we’ll do something very similar:

const completion_stream = await client.chat.completions.create({
    model: model,
    messages: messages,
    stream: true
});

let piece_list = [];
for await (const piece of completion_stream) {
    const piece_txt = piece.choices[0]?.delta?.content || '\n';
    process.stdout.write(piece_txt);
    piece_list.push(piece_txt);
}
let text_response = piece_list.join('');

An addional thing to consider when using the stream option is that the response message will change, and instead to getting choices[0].message we’ll need to get choices[0].delta. Furthermore, when the stream option is used, usage information is not included in the response, so if we need to control the number of tokens we have to use tiktoken (or some other equivalent library).

The complete code of the grumpy chatbot with stream can be found in GitHub for Python y Node.js.