OpenAI

ChatGPT and other apps from the leading AI company

chatgpt
openai
competition

OpenAI History and Principles for summaries of the company and its models, including leaked details of GPT-4

Building GPT-4o mini

summary from The Aurorean Jul 2024

[!Aurorean summary] The team at OpenAI released their latest small model, GPT-4o mini. This model is reportedly around 8 billion parameters in size and appears to rival Google’s Gemini 1.5 Flash model and Meta’s newly released Llama 3.1 8B model in terms of cost and performance.

🔬How It Was Done: Unfortunately, OpenAI did not release a paper to explain how they developed their model in-depth. However, we can approximate their methodologies thanks to past research papers from OpenAI, Google and Meta’s newly minted Llama 3.1 paper.

  • Train the model on lots of high quality, self generated data. High quality training data has an outsized influence on model performance, so this first step is critical to success.

  • Incorporate a feedback mechanism and verifier models to help the smaller model identify and correct reasoning mistakes it may make.

  • Design the model to reason through problems in a step-by-step format, and use other models to check for good reasoning. In this case, the larger GPT-4o model can act as a teacher for the mini version to learn from and emulate.

  • Filter out low-quality data and poor reasoning skills whenever they are identified to reinforce better reasoning abilities. Have human reviewers provide feedback to help the smaller model produce responses people consider to be high quality.

  • When necessary, use Monte Carlo Tree Search to find more expansive ideas and potential solutions to reason through difficult problems.

🧮Key Results: The cost per token of GPT-4o mini is 20x cheaper compared to GPT-4o and an astonishing 99% cheaper compared to a 2022 model from OpenAI.

Evaluating GPT-4

Coagulopath started the discussion but no longer thinks it’s gotten worse. Using example queries about Italian history and rock music, he shows there’s much nuance in how to interpret the accuracy of an answer and it’s impossible to say whether the later models are better or worse.

in Is GPT-4 getting worse over time? AI Snake Oil argues that no, GPT-4 is not getting worse like some studies have claimed. Good discussion of how you would dissect an LLM.