Add o3-mini support

[Nicba1010]

o3-mini support is missing

[tarekgh]

Is it known which vocab file the tokenizer is going to use with this model?

[kashankrm]

Based on what i know it is the same tokenizer as gpt 4o. they just added some extra stuff to it.
I did a small test like this:

from openai import OpenAI

client = OpenAI() 

def generate_response(prompt: str, model: str = "o3-mini") -> str:
    print(f"Generating response for {model} model...")
    response = client.chat.completions.create(
        model=model,
        messages=[
        {"role": "user", "content": prompt}
    ],
    )
    # print(response.choices[0].message.content)
    print("input tokens:", response.usage.prompt_tokens)
    print("reasoning tokens:", response.usage.completion_tokens_details.reasoning_tokens)
    print("output tokens:", response.usage.completion_tokens - response.usage.completion_tokens_details.reasoning_tokens)
    print("\n ---------------- \n")
    return response.choices[0].message.content


messages = [
    "Complete this sentence: The quick brown fox jumps",
    "kajsh  ekr jn as kemnralsekjr la sekjrl skejlsmcelkamc .skenrs",
    "code, make sure you have installed the OpenAI Python"
]
for message in messages:
    print("testing for message:", message)
    generate_response(message)
    generate_response(message, model="o1")
    generate_response(message, model="o1-mini")
    generate_response(message, model="o1-preview")
    generate_response(message, model="gpt-4o")

here is the output:

testing for message: Complete this sentence: The quick brown fox jumps
Generating response for o3-mini model...
input tokens: 15
reasoning tokens: 192
output tokens: 20

 ---------------- 

Generating response for o1 model...
input tokens: 15
reasoning tokens: 256
output tokens: 21

 ---------------- 

Generating response for o1-mini model...
input tokens: 17
reasoning tokens: 128
output tokens: 22

 ---------------- 

Generating response for o1-preview model...
input tokens: 17
reasoning tokens: 128
output tokens: 29

 ---------------- 

Generating response for gpt-4o model...
input tokens: 16
reasoning tokens: 0
output tokens: 6

 ---------------- 

testing for message: kajsh  ekr jn as kemnralsekjr la sekjrl skejlsmcelkamc .skenrs
Generating response for o3-mini model...
input tokens: 34
reasoning tokens: 128
output tokens: 42

 ---------------- 

Generating response for o1 model...
input tokens: 34
reasoning tokens: 384
output tokens: 56

 ---------------- 

Generating response for o1-mini model...
input tokens: 36
reasoning tokens: 192
output tokens: 41

 ---------------- 

Generating response for o1-preview model...
input tokens: 36
reasoning tokens: 320
output tokens: 39

 ---------------- 

Generating response for gpt-4o model...
input tokens: 35
reasoning tokens: 0
output tokens: 29

 ---------------- 

testing for message: code, make sure you have installed the OpenAI Python
Generating response for o3-mini model...
input tokens: 17
reasoning tokens: 320
output tokens: 326

 ---------------- 

Generating response for o1 model...
input tokens: 17
reasoning tokens: 320
output tokens: 434

 ---------------- 

Generating response for o1-mini model...
input tokens: 19
reasoning tokens: 448
output tokens: 622

 ---------------- 

Generating response for o1-preview model...
input tokens: 19
reasoning tokens: 896
output tokens: 403

 ---------------- 

Generating response for gpt-4o model...
input tokens: 18
reasoning tokens: 0
output tokens: 428

 ---------------- 

notice that o3-mini and o1 have same input tokens, o1-preview and o1-mini have same and then we have gpt-4o. And all of them are offset by few tokens. I also did a similar test with a system prompt and there the difference was about 9 tokens, but always predictable.

[fast-blender-add-ons-dot-com]

This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page

[fast-blender-add-ons-dot-com]

But in response to the other guy 03 mini and o1-mini don't have the same tokens it seems weird To have to sit here and do math... when it's easy to implement into the library but 03 mini and o1-mini have the same tokens input and output.... I mean there's this complex library I don't know all the parameters of it but it seems like there should be somewhere where you could explicitly define that 03 mini has $1.10 for input tokens and $4.40 for output tokens and just make a dictionary based on that and then Design your code around that dictionary so whenever there's like a new model or they change the prices then you just gotta update the price in the dictionary and then the code would just use that...

[Nicba1010]

This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page

Is that supposed to be in response to me? If so, I have no idea what this word salad is supposed to convey.

[fast-blender-add-ons-dot-com]

OKI just took 5 minutes to do the math on the function and it appears when I give it the oh one mini model it outputs the same price as the 03 mini model so I'm just going to make all my code whenever he needs to calculate 03 many tokens use the OH-1 mini model parameter

[fast-blender-add-ons-dot-com]

I'm sure you could glean the context if you read my other posts on the issue that I posted on

…

On Thu, Feb 13, 2025 at 7:06 AM Roberto Anić Banić ***@***.***> wrote: This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page Is that supposed to be in response to me? If so, I have no idea what this word salad is supposed to convey. — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFQGIOA7JTRP62PIWOLJRT2PSYIFAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJWHA4TAMJZGM> . You are receiving this because you commented.Message ID: ***@***.***> [image: Nicba1010]*Nicba1010* left a comment (openai/tiktoken#374) <#374 (comment)> This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page Is that supposed to be in response to me? If so, I have no idea what this word salad is supposed to convey. — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFQGIOA7JTRP62PIWOLJRT2PSYIFAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJWHA4TAMJZGM> . You are receiving this because you commented.Message ID: ***@***.***>

[hauntsaninja]

This is resolved in tiktoken 0.9

[fast-blender-add-ons-dot-com]

Thank you for your help!

…

On Thu, Feb 13, 2025 at 10:04 PM Shantanu ***@***.***> wrote: This is resolved in tiktoken 0.9 — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFQGIMXX4MKRKCP4YLSBY32PWBPDAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJYGM2TSOJWGE> . You are receiving this because you commented.Message ID: ***@***.***> [image: hauntsaninja]*hauntsaninja* left a comment (openai/tiktoken#374) <#374 (comment)> This is resolved in tiktoken 0.9 — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFQGIMXX4MKRKCP4YLSBY32PWBPDAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJYGM2TSOJWGE> . You are receiving this because you commented.Message ID: ***@***.***>

[tarekgh]

adding the link of mapping the model's name to the encoding file just for clarity

tiktoken/tiktoken/model.py

Line 8 in e35ab09

"o1-": "o200k_base",

.

[fast-blender-add-ons-dot-com]

Thanks, I'll check it out!

…

On Fri, Feb 14, 2025 at 9:37 AM Tarek Mahmoud Sayed < ***@***.***> wrote: adding the link of mapping the model's name to the encoding file just for clarity https://github.com/openai/tiktoken/blob/e35ab0915e37b919946b70947f1d0854196cb72c/tiktoken/model.py#L8 . — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFQGINMVV3MBN3GUPPOQKT2PYSVFAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJZHEYDQNZXHA> . You are receiving this because you commented.Message ID: ***@***.***> [image: tarekgh]*tarekgh* left a comment (openai/tiktoken#374) <#374 (comment)> adding the link of mapping the model's name to the encoding file just for clarity https://github.com/openai/tiktoken/blob/e35ab0915e37b919946b70947f1d0854196cb72c/tiktoken/model.py#L8 . — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFQGINMVV3MBN3GUPPOQKT2PYSVFAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJZHEYDQNZXHA> . You are receiving this because you commented.Message ID: ***@***.***>