#7: Become a ChatGPT Power User: Part 1
Crafting good instructions that communicate your intent to an AI model, aka prompt engineering, is the key to getting accurate, relevant & coherent responses, and help unleash your vision.
Most people have merely scratched the surface in exploring ChatGPT’s full potential. Let’s fix that by looking at how to get the most out of ChatGPT through good prompts.
Understand model parameters
It’s hard for model developers to build a one-size-fits-all language model. So, they allow users to change a few parameters to customize the output. I’ll focus here on 2 main parameters — temperature and top-p.
Temperature: Recall from post #4 that when faced with an incomplete sentence like"Elephants are the ___," an LLM like ChatGPT generates options for the next word/token like "largest" with an 11% probability, "smartest" with a 7% probability, “gentlest” at 2% probability, and so on. The highest-ranked word is not always selected because introducing some randomness helps generate more interesting text. The "temperature" parameter controls the diversity and randomness of the output. While setting a low temperature often improves the consistency and accuracy of the response, it also reduces the creativity and diversity of the output. If you are writing a nonfiction piece, a biography, something fact-based, or even software code, consider using a low temperature around 0.4 or lower. But if you are writing a creative essay or poem, try a higher temperature around 0.7 or 0.8. Here is a simple example.
Prompt: Based on the text "elephants are the largest," please continue the text generation. Use a temperature value of 0.
Elephants are the largest mammals on Earth. Their size and weight distinguish them as the unparalleled giants of the animal kingdom. They possess unique features such as their long trunks, large ears, and impressive tusks. These magnificent creatures are known for their intelligence, strong family bonds, and complex social structures. Elephants play a crucial role in maintaining the balance of ecosystems, shaping landscapes, and enriching biodiversity.
Prompt: now redo the task above but with a temperature value of 1.
Elephants are the largest contributors to our planet's rich biodiversity. From their massive presence in the wild to their graceful interactions with other creatures, these gentle giants leave an indelible mark on our ecosystem. Their imposing size and fascinating behavioral patterns make them a captivating sight to behold. Whether it's their remarkable memory, their playful nature, or their intricate social dynamics, elephants continue to amaze and inspire us. Despite their massive stature, they remind us of the delicate balance of life and the importance of conservation efforts.
Notice how the first output is more factual than the second output.
Increasing the temperature increases the creativity/interestingness of the text but can occasionally make the output nonsensical. One solution is to lower the temperature. Another option is to simply limit the model from choosing highly unlikely words. Enter the “top_p” parameter.
Top-p: Top-p determines how far down the list of candidate words/tokens a model will consider. A higher top_p value expands the pool of words under consideration, while lower values narrow down the candidate set to the most likely words, favoring consistent and expected responses. In the earlier example, if the top_p value is set to 0.2, ChatGPT will only consider the top three tokens from the table we discussed – "largest," "smartest," and "gentlest" – because their probabilities (11%, 7% and 2%) add up to 20%. By doing so, highly unusual or factually incorrect choices, like "elephants are the largest mammals," are less likely to be generated.
For code generation, I would use a low temperature like 0.2 and low value of top_p like 0.1. For creative writing, I’d consider a higher temperature like 0.8. But if you are seeing unexpected or undesirable output, you can lower the top-p as follows:
Prompt: Based on the text "elephants are the largest," please continue the text generation. Use a temperature value of 0.8 and a top_p value of 0.7
Elephants are the largest land mammals on Earth, towering above other creatures in size and strength. They are revered for their majestic presence and remarkable intelligence. These gentle giants are known for their close-knit social groups and complex communication systems. With their impressive tusks and distinctive trunks, elephants exhibit a captivating mix of power and grace. In their natural habitats, they play a vital role in shaping ecosystems and maintaining ecological harmony. As magnificent beings, elephants serve as a reminder of the marvels of the animal kingdom.
Image source: Stable Diffusion
Chain of Thought (CoT) prompting to help the model reason
Suppose you want an LLM to reason through a problem. One of the best ways to do this is by providing a few examples with the reasoning process (aka Chain of Thought (CoT) prompting). Consider an example from a research paper by Kojima et al (based on GPT3 but general idea applies to recent versions as well).
Prompt: Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
A: The answer (arabic numerals) is
ChatGPT Response: 8
The answer is wrong. Let’s share a few examples in our prompt – also known of few-shot prompting.
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: The answer is 11.
Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
A:
ChatGPT Response: The answer is 8.
Wrong again. Next, we will not just share an example but also the reasoning.
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.
Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
A:
ChatGPT Response: The juggler can juggle 16 balls. Half of the balls are golf balls. So there are 16 / 2 = 8 golf balls. Half of the golf balls are blue. So there are 8 / 2 = 4 blue golf balls. The answer is 4.
With enough experimentation, we have found a winning prompt. Next consider this prompt.
Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
A: Let’s think step by step.
ChatGPT Response: There are 16 balls in total. Half of the balls are golf balls. That means that there are 8 golf balls. Half of the golf balls are blue. That means that there are 4 blue golf balls.
Simple adding “Let’s think step by step” to the prompt without any examples may sometimes suffice to get an LLM to reason well.
In short, keep these ideas in mind: (i) Set model parameters thoughtfully (ii) Provide examples and reasoning (CoT) where relevant (iii) Experiment with the prompt. More ideas in the next post.
This might be a very obvious/dumb question - how do I set the temperature and top_p value in ChatGPT?