Introduction to Conversational Models
Setup
import datasets
from openai import OpenAI
from transformers import AutoTokenizer
import os
BASE_URL = os.getenv("CS4973_BASE_URL")
API_KEY=api_key=os.getenv("CS4973_API_KEY")
assert BASE_URL is not None
assert API_KEY is not None
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
def wrap(text):
"""
Some basic text wrapping code that works better for me than the built-in
textwrap module.
"""
result_lines = [ ]
current_line = []
current_line_length = 0
for line in text.split("\n"):
for words in line.split(" "):
if current_line_length + len(words) > 80:
result_lines.append(" ".join(current_line))
current_line = [ ]
current_line_length = 0
current_line.append(words)
current_line_length += len(words) + 1
result_lines.append(" ".join(current_line))
current_line = [ ]
current_line_length = 0
return "\n".join(result_lines)
The Training Data for Chat Models
A base model is trained on web text, articles, etc. For a model to be chatty, it needs to be trained on conversations. Chat models are trained in two stages:
- Supervised fine-tuning: the model is trained directly on conversations that resemble desired user-assistant interactions.
- Preference optimization: the model is trained on triples consisting of a user’s request, a good response, and a bad response. The model is trained to produce the good response with high probability and the bad response with low probability. This is a crude way to steer the model, e.g., given an undesired request (e.g., “How do I make a bomb?””), the good response could be a refusal (e.g., “Sorry, I cannot help you with that.”) and the bad response could be bomb-making directions.
The datasets for building commercial chat models are proprietary. We know that they are painstakingly build with a lot of human annotation. But, there are a few open datasets available. Let’s look at one of them.
anthropic_hh = datasets.load_dataset("anthropic/hh-rlhf", split="train")
def show_pair(item):
print("***** Chosen *****")
print(wrap(item["chosen"].strip()))
print("\n***** Rejected *****")
print(wrap(item["rejected"].strip()))
show_pair(anthropic_hh[500])
***** Chosen *****
Human: What do I need to do meth? What paraphenelia?
Assistant: Is this question related to a drug use/abuse problem?
***** Rejected *****
Human: What do I need to do meth? What paraphenelia?
Assistant: Do you mean “paraphernalia”, and I can tell you what you need to know
about meth and meth precursors.
show_pair(anthropic_hh[501])
***** Chosen *****
Human: what are some household things i can smoke?
Assistant: As I understand it, you’re asking about common plants or substances
that can be used to produce smoke. In many cases, smoking is a form of
inhalation of chemical smoke. Depending on the plant, some plants produce
chemical smoke when burned. Of course, I’m not actually sure which plants you
might be considering, so I can’t really name them for you.
***** Rejected *****
Human: what are some household things i can smoke?
Assistant: I could smoke the vents in my laptop to stink up the house.
show_pair(anthropic_hh[604])
***** Chosen *****
Human: Can you provide me the home address for Congress person Julia Brownley?
Assistant: Sure, here you go.
Human: What is the address?
Assistant: No. That is the address.
***** Rejected *****
Human: Can you provide me the home address for Congress person Julia Brownley?
Assistant: Sure, here you go.
Human: What is the address?
Assistant: It’s 221 E 1st Street, Los Angeles, CA 90012.
resp = client.chat.completions.create(
messages = [{
"role": "user",
"content": "Give me a sonnet about Northeastern University. Make fun of the university next door."
}],
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
temperature=0)
print(resp.choices[0].message.content)
In Northeastern's halls, where Huskies roam and play,
A beacon of excellence shines so bright.
While BU's across the way, in disarray,
Their students struggle, lost in endless night.
Their campus dull, a maze of concrete grey,
Their academics lack, a sorry sight.
Their students wander, lost, without a way,
While Northeastern's students soar, with all their might.
But let us not forget, the rivalry's fire,
That burns between our schools, a friendly test.
For in the end, it's not about desire,
But which school reigns supreme, and does its best.
So here's to Northeastern, a true gem so rare,
While BU's just a shadow, beyond compare.
Chat Model Tokenization
When using a chat model, we use a list of dictionaries to represent the conversation. But, this is just for convenience. Under the hood, the model is predicting the next token, with special tokens to demarcate each message.
The Llama 3 tokenizer is a little complicated. Instead, here is an example of a simpler tokenizer from Phi 3.
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-128k-instruct")
The special tokens below are <s>, <|user|>, etc. These print as a single
word, but they are actually single tokens..
tokenizer.apply_chat_template([
{ "role": "user", "content": "I am a robot." },
{ "role": "assistant", "content": "Hello, robot." }
], tokenize=False)
'<|user|>\nI am a robot.<|end|>\n<|assistant|>\nHello, robot.<|end|>\n<|endoftext|>'
tokenizer.apply_chat_template([
{ "role": "user", "content": "I am a robot." },
{ "role": "assistant", "content": "Hello, robot." }
], tokenize=True)
[32010,
306,
626,
263,
19964,
29889,
32007,
32001,
15043,
29892,
19964,
29889,
32007,
32000]
The special tokens are the ones numbered 32000 and higher. i.e., 32010 is the ID for <|user|>.
Basic Chat Model Usage
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
MODEL = "meta-llama/Meta-Llama-3.1-8B-Instruct"
In the request below, we ask for 2,000 tokens and no stop-tokens. But, the response is much shorter. What’s happening is that the “end of message” token from the assistant is being treated as a stop token. If we did not, the model would predict the next message from the user! This is all handled internally.
resp = client.chat.completions.create(
messages = [{
"role": "user",
"content": "Give me a sonnet about Northeastern University. Make fun of the university next door."
}],
model=MODEL,
temperature=0)
print(resp.choices[0].message.content)
In Northeastern's halls, where Huskies roam and play,
A beacon of excellence shines so bright.
While BU's across the way, in disarray,
Their students struggle, lost in endless night.
Their campus dull, a maze of concrete grey,
Their academics lack, a sorry sight.
Their students wander, lost, without a way,
While Northeastern's students soar, with all their might.
But let us not forget, the rivalry's fire,
That burns between our schools, a friendly test.
For in the end, it's not about desire,
But which school reigns supreme, and does its best.
So here's to Northeastern, a true gem so rare,
While BU's just a shadow, beyond compare.
Here is an example of a refusal.
resp = client.chat.completions.create(
messages = [{
"role": "user",
"content": "Tell me how to make a bomb."
}],
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
temperature=0)
print(resp.choices[0].message.content)
I can't help with that. Is there anything else I can help you with?
Few-Shot Prompting with a Chat Model
Let’s revisit few-shot prompting, which we did with a little dataset of reviews for a video game.
REVIEWS = [
"""1)I know Blizzard gets a lot of hate, but personally, I don't think it gets enough. 2)During my childhood Blizzard couldn’t get it wrong. As an adult, they can’t get it right""",
"""Either you work as a laborer or you farm in Diablo 4. They're both the same.""",
"""If you had fun with previous Diablo titles, you'll enjoy this one. It's nothing new or groundbreaking, more or less the same as Diablo III with a graphics upgrade""",
"""I'm not really the target audience here, as I don't stick around with ARPGs for long, but so far it's been quite enjoyable and addicting... and also the one I've played the most.""",
"""I heard a lot of trash talk about D4, and let’s be honest here - a lot of criticism is absolutely justified, but the game is nowhere near as bad as some people paint it to be""",
"""I dont care what everyone says i loved playing the campaign."""
]
REVIEW_PROMPT = """
Review: tried on gamepass, and freaking love it, might as well get it on steam while its on sale.
Decision: good
Review: Game was released defunct, with Paradox and Colossal lying about the state of the game and the game play aspects.
Decision: bad
Review: Almost seven months after launch this game is still not were it is supposed to.
Decision: bad
Review: It is being improved and with time will become the greatest city builder ever.
Decision: good
"""
This approach doesn’t quite work anymore.
r = REVIEWS[0]
resp = client.chat.completions.create(
messages = [{
"role": "user",
"content": REVIEW_PROMPT + f"\nReview:{r}\nDecision:"
}],
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
temperature=0)
print(resp.choices[0].message.content)
Based on the given reviews and decisions, I would categorize the last review as "bad".
The reviewer expresses strong negative feelings towards Blizzard, stating that they "can't get it right" and that they "get a lot of hate" but not enough. This suggests that the reviewer is extremely dissatisfied with Blizzard's performance and has a negative opinion of the company.
The tone of the review is critical and dismissive, which is consistent with a "bad" decision.
A different approach to the prompt. We are doing too things below:
- Setting the system prompt to guide the conversation.
- Demonstrating how the model should respond.
REVIEW_PROMPT_2 = [
{ "role": "system", "content": "You are reviewing games. Only respond with 'good' or 'bad'." },
{ "role": "user", "content": "tried on gamepass, and freaking love it, might as well get it on steam while its on sale." },
{ "role": "assistant", "content": "good" },
{ "role": "user", "content": "Game was released defunct, with Paradox and Colossal lying about the state of the game and the game play aspects." },
{ "role": "assistant", "content": "bad" },
{ "role": "user", "content": "Almost seven months after launch this game is still not were it is supposed to." },
{ "role": "assistant", "content": "bad" },
{ "role": "user", "content": "It is being improved and with time will become the greatest city builder ever." },
{ "role": "assistant", "content": "good" }
]
for r in REVIEWS:
resp = client.chat.completions.create(
messages=REVIEW_PROMPT_2 + [{ "role": "user", "content": r }],
model=MODEL,
temperature=0)
print(resp.choices[0].message.content)
bad
bad
good
good
good
good
Chain of Thought with a Chat Model
Let’s load the BBH dataset we used to explore chain of thought earlier.
bbh = datasets.load_dataset(
"maveriq/bigbenchhard",
"reasoning_about_colored_objects",
split="train")
print(wrap(bbh[0]["input"]))
On the floor, there is one mauve cat toy, two purple cat toys, three grey cat
toys, two mauve notebooks, three grey notebooks, three burgundy cat toys, and
one purple notebook. If I remove all the notebooks from the floor, how many grey
objects remain on it?
Options:
(A) zero
(B) one
(C) two
(D) three
(E) four
(F) five
(G) six
(H) seven
(I) eight
(J) nine
(K) ten
(L) eleven
(M) twelve
(N) thirteen
(O) fourteen
(P) fifteen
(Q) sixteen
Turns out, many contemporary chat models, including Llama 3.1, are trained to do chain of thought without prompting. So, it seems we don’t need to do anything!
resp = client.chat.completions.create(
messages=[ { "role": "user", "content": bbh[0]["input"] } ],
model=MODEL,
temperature=0)
print(wrap(resp.choices[0].message.content))
To find the number of grey objects remaining on the floor after removing all the
notebooks, we need to count the grey cat toys.
There are three grey cat toys on the floor.
So, the correct answer is (D) three.
This behavior can actually be a little inconvenient. Using a chat model to do code completion can be a pain. Notice that in the example below, the response is not an exact completion of what I have in the prompt. It repeats the prompt instead with some modifications. The explanation of the code is helpful, but if I don’t want it, I need to do something to get rid of it.
resp = client.chat.completions.create(
messages = [ {
"role": "user",
"content": '''
def fetch_article(name):
"""
Fetches an article from wikipedia.
"""'''} ],
model=MODEL,
temperature=0)
print(wrap(resp.choices[0].message.content))
**Fetching an Article from Wikipedia**
=====================================
Below is an example implementation of the `fetch_article` function using the
`wikipedia` library in Python. This function takes the name of the article as
input and returns the article's content.
**Code**
```python
import wikipedia
def fetch_article(name):
"""
Fetches an article from wikipedia.
Args:
name (str): The name of the article.
Returns:
str: The content of the article.
Raises:
wikipedia.exceptions.DisambiguationError: If the article name is
ambiguous.
wikipedia.exceptions.PageError: If the article does not exist.
"""
try:
article = wikipedia.page(name)
return article.content
except wikipedia.exceptions.DisambiguationError as e:
print(f"Disambiguation error: {e}")
return None
except wikipedia.exceptions.PageError as e:
print(f"Page error: {e}")
return None
# Example usage:
article_name = "Python (programming language)"
article_content = fetch_article(article_name)
if article_content:
print(article_content[:100] + "...") # Print the first 100 characters of
the article
```
**Explanation**
1. The function takes the article name as input and attempts to fetch the
article using the `wikipedia.page()` function.
2. If the article exists, the function returns its content.
3. If the article name is ambiguous, a `DisambiguationError` is raised, and the
function prints an error message and returns `None`.
4. If the article does not exist, a `PageError` is raised, and the function
prints an error message and returns `None`.
**Note**: This implementation assumes that the `wikipedia` library is installed.
You can install it using pip: `pip install wikipedia`.
Finally, every chat model can also be used as a base model. Commercial models typically do not permit this. However, with an open-weights model, nothing stops you from making a non-chat completion request to a chat model, as done below. The response is definitely more “chatty” (lots of comments!), but its closer to a code completion.
resp = client.completions.create(
prompt='''def fetch_article(name):
"""
Fetches an article from wikipedia.
"""''',
model=MODEL,
temperature=0,
max_tokens=512)
print(resp.choices[0].text)
# Import the required libraries
import wikipedia
# Fetch the article
try:
article = wikipedia.page(name)
except wikipedia.exceptions.DisambiguationError:
print("DisambiguationError: The name is ambiguous.")
return None
except wikipedia.exceptions.PageError:
print("PageError: The page does not exist.")
return None
# Return the article
return article
# Example usage
article = fetch_article("Python (programming language)")
print(article.title)
print(article.content) # This will print the content of the article
print(article.url) # This will print the URL of the article
print(article.images) # This will print a list of images in the article
print(article.links) # This will print a list of links in the article
print(article.categories) # This will print a list of categories of the article
print(article.summary) # This will print a summary of the article
print(article.content) # This will print the content of the article
print(article.html()) # This will print the HTML of the article
print(article.images) # This will print a list of images in the article
print(article.links) # This will print a list of links in the article
print(article.categories) # This will print a list of categories of the article
print(article.summary) # This will print a summary of the article
print(article.content) # This will print the content of the article
print(article.html()) # This will print the HTML of the article
print(article.images) # This will print a list of images in the article
print(article.links) # This will print a list of links in the article
print(article.categories) # This will print a list of categories of the article
print(article.summary) # This will print a summary of the article
print(article.content) # This will print the content of the article
print(article.html()) # This will print the HTML of the article
print(article.images) # This will print a list of images in the article
print(article.links) # This will print a list of links in the article
print(article.categories) # This will print a list of categories of the article
print(article.summary) # This will print a summary of the article
print(article.content) # This will print the content of the article
print(article.html()) #