Kickstart text classification with GPT-3, part 2

Pubished: October 22, 2022

Where we stopped

In the previous part we finished with the desire to get predictions for multiple samples and multiple categories. We will do it here using a Markdown table format.

What is Markdown?

Markdown is a lightweight markup language that you can use to add formatting elements to plaintext text documents. Created by John Gruber in 2004, Markdown is now one of the world’s most popular markup languages.

Basically, Markdown is a way to specify formatting in a plain text document, using special symbols. E.g. to make bold text, you’d write two asterisks before and after, like so: **bold**. The platforms which support markdown will automatically render this as bold text.

How Markdown tables are written in plain text

There is a way to create tables in Markdown, which, in the most basic version, looks like this:

| column 1 | column 2 | column 3 |
| --- | --- | --- |
| 1 | 2 | 3 |
| 4 | 2 | 3 |

And it will be rendered like so:

Why is this useful for us? Because GPT-3 has seen a lot of Markdown. And once we hint at the fact that what it is completing is a Markdown document, it will stick to this format pretty well. This will give us a stable and structured output, which we can easily parse.

So the main idea of the approach is that we will prompt GPT-3 in such a way that it will create a table of samples and labels. How exactly?

Organizing a response as a Markdown table

Here is the basic breakdown:

1. Provide the text to be classified as a list of samples.
2. Each sample needs to have some kind of identifier, which will be included in the table (in our example of Steam games classification, it can be the name of the game).
3. Create the first two rows of Markdown table: column titles and the separator.
4. Let GPT-3 do the rest of the magic.

Here are a couple of examples of prompts with GPT-3 completion:

Limiting values in the table

As you can notice above, the tables in the two examples are not exactly in the same format. In my experiments, I have seen Yes / No, T / F, true / false, + / -, TRUE / FALSE, X / O, X / nothing. Why is this important for us?

It is time to think about what we are going to do with the responses. Let’s remind ourselves that our goal is to kickstart classification. That is, we want to label some new data so that later we can train our own supervised classified. Let’s start with the intention to label 200 samples.

But with a single request to GPT-3 we can only label 5-10 samples (depending on the size of the prompt, we will discuss this below in Number of samples section section). Hence we need to automate the process of making requests and parsing the responses. And the first step is to make sure that the responses are uniform: we want the same type of values in the table, instead of many different ones.

I’ve found that the simplest way to do that is to include the values in the prompt ( note the change ):

In my experiments, this all but ensures that the values in the table will be TRUE or FALSE¹. We will also add some checks in our automation code later to make sure that GPT-3 doesn’t generate other values².

I will hint that there is a better way to ensure the stability of responses, using logit_bias. I may address this at the end of the series, but it is not essential – it is usually enough to modify the prompt as shown above.

Experimenting with the prompt

At this point, we’re close to fixing our prompt. Of course, the samples will change on every request, but the surrounding will stay the same. This is a good spot to experiment with the prompt a little more.

In particular, I’ve found that thinking about the labels (names of the columns in the table) is very useful. Since we’ve not provided any description of the labels, GPT-3 can only go with the names of the labels to infer their meaning. For example, we may have the label “Action” – but the word itself has multiple meanings. GPT-3 will consider the context (our prompt mentions games and genres), but we may want to try making it easier for GPT-3 to understand what we want. For example, instead of “Action”, we can make the label “Genre Action”. Or even make it more technical, as if it is a dump of the database, e.g. “genre_action”.

So what I usually do is pick some samples which I think GPT-3 should be able to classify and some samples which seem difficult for me and try to change the prompt and look at the responses for correctness. In parallel, I am trying to gauge the stability of the responses: how much do they change when I make modifications to the prompt?

I usually do 3-7 experiments like these. There is space for automation here, but I find this part fun and creative, and eventually not taking too much time.

Let’s try this in practice. First, I change the games a little, picking them such that they span different genres, and try our initial prompt:

The result is the following (rendering is for ease of reading):

At a first glance, this is not bad. I’d like it to classify Portal as Action and Platformer, but well, Puzzle is already pretty good. Still, I try to change the prompt a little. Remember that there are two main goals here: try to improve the responses and gauge the stability. I change the prompt so that it seems more like a database dump (lowercase labels with snake_case):

Aaaaand nothing has changed. This is good news for stability, and bad for improving our labels. Here I get another idea: what if I add price to the input data? Since platformers are usually cheaper than AAA games, maybe this would help.

Still, there are no changes. At this point, I am inclined to stop. The responses are rather stable, and the labels, while not perfect, are pretty good.

Limitations

Let’s talk a little about the limitations of this approach before we go further.

Stability of responses

We have to come to terms with the fact that everything influences GPT-3 response to a certain extent. This is inherent to the nature of the approach. Anything can change the probabilities of the responses: the wording of the prompt, a change in capitalization, a change in the order of the labels and samples³. We’re hoping that the responses are stable enough that we don’t have to think too much about it.

That said, in manual labelling, there is the same issue. Different people will label samples differently, people are also influenced by instructions and wording, by cognitive biases, such as the anchoring effect. So in reality it is not so different.

Truthfulness and typos

How correct are the responses of the model? There is a whole area of research dedicated to Large Language Model truthfulness (see this for example). Since the training set contains a lot of information from the web, nothing guarantees that it will be correct. In some ways, we are relying on the fact that the training set contains much more correct information than false.

However, there is another fun point, relating to the stability of responses. Depending on the problem, while GPT-3 may be able to solve it in principle, it may “think” that our prompt implies some degree of incorrectness. How’s that?

Think about an example: we try to generate responses to a math test. If our prompt implies that this is the control version of the test, it will tend to be correct. But if the prompt implies that this is a response of a student to the test, it may be slightly incorrect.

How does it relate to our problem? Well, if you make a typo somewhere in your prompt, while GPT-3 will for sure understand what you meant, it might “think” that the text it is generating is not, let’s say, “written by a professional”. This could lead to the response also being somewhat mistaken. So check your prompt several times for typos and wording.

Number of samples

How many samples can we insert in one request? GPT-3 has a hard limit on the total number of tokens (prompt + generations) – 2048 tokens⁴. Each token is 4 characters on average, so we can say that in total we can use around 8400 characters.

Depending on the size of each sample, we can determine the maximum number of samples to be put in on request. However, since GPT-3 charges per token, we’re not saving a lot by cramming as many samples as possible in each request. I prefer to stay way below the limit, usually at 5-7 samples per request.

What’s next

In this post, we’ve formatted the responses as Markdown tables, limited responses to TRUE / FALSE to improve stability, experimented with prompts and discussed some limitations and considerations of this approach.

In the next post, we will write Python code to automate requests and parse the responses to get a set of labels for each sample.