LLM Leaderboard: Compare & Check Latest API Prices for LLMs
Compare and check the latest prices for LLM (Large Language Model) APIs from leading providers such as OpenAI, Mistral, Anthropic, Google, Meta, Perplexity, and more. Evaluate and rank the performance of over 50+ AI models (LLMs) across key metrics, including quality, context window, price, knowledge cutoff, and others. This in-depth comparison allows users to easily identify the best-suited LLM for their specific needs and budget.
LLM Leaderboard Highlights:
Quality: The highest quality models are GPT-4o and Llama 3.1 405B. These are followed by Claude 3.5 Sonnet and Llama 3.1 70B.
Context Window: The models with the largest context windows are Gemini 1.5 Pro (2 Million) and Gemini 1.5 Flash (1 Million). These are followed by Codestral-Mamba and Jamba Instruct.
Price ($ per M tokens): OpenChat 3.5 ($0.14) and Phi-3 Medium 14B ($0.14) are the cheapest models, followed by Gemma 7B and Llama 3.1 8B.
| MODEL | CREATOR | LICENSE | QUALITY | CONTEXT | INPUT $/1M | OUTPUT $/1M | KNOWLEDGE | FREE TRIAL |
|---|---|---|---|---|---|---|---|---|
| GPT-4o |  | Proprietary | 100 | 128k | $5.00 | $15.00 | Oct 2023 | https://openrouter.ai/chat?models=openai/gpt-4o |
| GPT-4 Turbo | | Proprietary | 94 | 128k | $10.00 | $30.00 | Dec 2023 | https://openrouter.ai/chat?models=openai%2Fgpt-4-turbo |
| GPT-4o Mini | | Proprietary | 88 | 128k | $0.15 | $0.60 | Oct 2023 | https://openrouter.ai/chat?models=openai/gpt-4o-mini |
| GPT-4 | | Proprietary | 84 | 8k | $30.00 | $60.00 | Sep 2021 | https://openrouter.ai/chat?models=openai/gpt-4 |
| GPT-3.5 Turbo Instruct | | Proprietary | 60 | 4k | $1.50 | $2.00 | Sep 2021 | https://openrouter.ai/chat?models=openai/gpt-3.5-turbo-instruct |
| GPT-3.5 Turbo | | Proprietary | 59 | 16k | $0.50 | $1.50 | Sep 2021 | https://openrouter.ai/chat?models=openai/gpt-3.5-turbo-0125 |
| Gemini 1.5 Pro |  | Proprietary | 95 | 2m | $3.50 | $10.50 | Nov 2023 | https://openrouter.ai/chat?models=google%2Fgemini-pro-1.5 |
| Gemini 1.5 Flash | | Proprietary | 84 | 1m | $0.35 | $1.05 | Nov 2023 | https://openrouter.ai/chat?models=google%2Fgemini-flash-1.5 |
| Gemma 2 (27B) | | Open | 78 | 8k | $0.80 | $0.80 | Jun 2024 | https://openrouter.ai/chat?models=google/gemma-2-27b-it |
| Gemma 2 (9B) | | Open | 71 | 8k | $0.20 | $0.20 | Jun 2024 | https://openrouter.ai/chat?models=google/gemma-2-9b-it |
| Gemma 7B | | Open | 45 | 8k | $0.15 | $0.15 | https://openrouter.ai/chat?models=google/gemma-7b-it | |
| Gemini 1.0 Pro | | Proprietary | 62 | 33k | $0.50 | $1.50 | Nov 2023 | https://openrouter.ai/chat?models=google%2Fgemini-pro |
| Llama 3.1 (405B) |  | Open | 100 | 128k | $5.33 | $9.50 | Dec 2023 | https://openrouter.ai/chat?models=meta-llama/llama-3.1-405b-instruct |
| Llama 3.1 (70B) | | Open | 95 | 128k | $0.89 | $0.89 | Dec 2023 | https://openrouter.ai/chat?models=meta-llama/llama-3.1-70b-instruct |
| Llama 3 (70B) | | Open | 83 | 8k | $0.90 | $0.90 | Dec 2023 | https://openrouter.ai/chat?models=meta-llama/llama-3-70b-instruct |
| Llama 3.1 (8B) | | Open | 66 | 128k | $0.16 | $0.16 | Dec 2023 | https://openrouter.ai/chat?models=meta-llama/llama-3.1-8b-instruct |
| Llama 3 (8B) | | Open | 64 | 8k | $0.20 | $0.20 | Mar 2023 | https://openrouter.ai/chat?models=meta-llama/llama-3-8b-instruct |
| Llama 2 Chat (70B) | | Open | 57 | 4k | $0.95 | $1.00 | Sep 2022 | https://openrouter.ai/chat?models=meta-llama/llama-2-70b-chat |
| Llama 2 Chat (13B) | | Open | 39 | 4k | $0.25 | $0.28 | Sep 2022 | https://openrouter.ai/chat?models=meta-llama/llama-2-13b-chat |
| Llama 2 Chat (7B) | | Open | 29 | 4k | $0.20 | $0.20 | Sep 2022 | https://huggingface.co/meta-llama/Llama-2-7b-chat-hf |
| Mistral Large 2 |  | Open | 91 | 128k | $3.00 | $9.00 | https://chat.mistral.ai/ | |
| Codestral | | Open | 33k | $1.00 | $3.00 | https://chat.mistral.ai/ | ||
| Codestral-Mamba | | Open | 256k | $0.25 | $0.25 | https://openrouter.ai/chat?models=mistralai/codestral-mamba | ||
| Mistral Large | | Proprietary | 76 | 33k | $4.00 | $12.00 | https://openrouter.ai/chat?models=mistralai/mistral-large | |
| Mixtral 8x22B | | Open | 71 | 65k | $1.20 | $1.20 | Sep 2021 | https://openrouter.ai/chat?models=mistralai/mixtral-8x22b-instruct |
| Mistral Small | | Proprietary | 71 | 33k | $1.00 | $3.00 | https://openrouter.ai/chat?models=mistralai/mistral-small | |
| Mistral Medium | | Proprietary | 70 | 33k | $2.70 | $8.10 | https://openrouter.ai/chat?models=mistralai/mistral-medium | |
| Mistral NeMo | | Open | 64 | 128k | $0.30 | $0.30 | Apr 2024 | https://openrouter.ai/chat?models=mistralai/mistral-nemo |
| Mixtral 8x7B | | Open | 61 | 33k | $0.50 | $0.55 | Dec 2023 | https://openrouter.ai/chat?models=mistralai/mixtral-8x7b-instruct |
| Mistral 7B | | Open | 40 | 33k | $0.20 | $0.20 | Dec 2023 | https://openrouter.ai/chat?models=mistralai/mistral-7b-instruct |
| Claude 3.5 Sonnet |  | Proprietary | 98 | 200k | $3.00 | $15.00 | Apr 2024 | https://openrouter.ai/chat?models=anthropic%2Fclaude-3.5-sonnet |
| Claude 3 Opus | | Proprietary | 93 | 200k | $15.00 | $75.00 | Aug 2023 | https://openrouter.ai/chat?models=anthropic%2Fclaude-3-opus |
| Claude 3 Sonnet | | Proprietary | 80 | 200k | $3.00 | $15.00 | Aug 2023 | https://openrouter.ai/chat?models=anthropic%2Fclaude-3-sonnet |
| Claude 3 Haiku | | Proprietary | 74 | 200k | $0.25 | $1.25 | Aug 2023 | https://openrouter.ai/chat?models=anthropic%2Fclaude-3-haiku |
| Claude 2.0 | | Proprietary | 70 | 100k | $8.00 | $24.00 | https://openrouter.ai/chat?models=anthropic%2Fclaude-2 | |
| Claude Instant | | Proprietary | 63 | 100k | $0.80 | $2.40 | https://openrouter.ai/chat?models=anthropic%2Fclaude-instant-1 | |
| Claude 2.1 | | Proprietary | 55 | 200k | $8.00 | $24.00 | https://openrouter.ai/chat?models=anthropic%2Fclaude-2.1 | |
| Sonar Large |  | Proprietary | 33k | $1.00 | $1.00 | https://openrouter.ai/chat?models=perplexity/llama-3-sonar-large-32k-chat | ||
| Sonar Small | | Proprietary | 33k | $0.20 | $0.20 | https://openrouter.ai/chat?models=perplexity/llama-3-sonar-small-32k-chat | ||
| OpenChat 3.5 |  | Open | 50 | 8k | $0.14 | $0.14 | https://openrouter.ai/chat?models=openchat/openchat-7b | |
| Command Light |  | Proprietary | 4k | $0.30 | $0.60 | https://coral.cohere.com/ | ||
| Command | | Proprietary | 4k | $1.25 | $2.00 | https://openrouter.ai/chat?models=cohere/command | ||
| Command-R+ | | Open | 75 | 128k | $3.00 | $15.00 | Mar 2024 | https://openrouter.ai/chat?models=cohere/command-r-plus |
| Command-R | | Open | 63 | 128k | $0.50 | $1.50 | Mar 2024 | https://openrouter.ai/chat?models=cohere/command-r |
| Phi-3 Medium (14B) |  | Open | 128k | $0.14 | $0.14 | Oct 2023 | https://openrouter.ai/chat?models=microsoft/phi-3-medium-128k-instruct | |
| Reka Core |  | Proprietary | 90 | 128k | $3.00 | $15.00 | Nov 2023 | https://chat.reka.ai/ |
| Reka Flash | | Proprietary | 78 | 128k | $0.80 | $2.00 | Nov 2023 | https://chat.reka.ai/ |
| Reka Edge | | Proprietary | 60 | 64k | $0.40 | $1.00 | Nov 2023 | https://chat.reka.ai/ |
| DBRX |  | Open | 62 | 33k | $1.20 | $1.20 | Dec 2023 | https://openrouter.ai/chat?models=databricks/dbrx-instruct |
| Jamba Instruct |  | Open | 63 | 256k | $0.50 | $0.70 | Mar 2024 | https://openrouter.ai/chat?models=ai21/jamba-instruct |
| Arctic |  | Open | 55 | 4k | $2.40 | $2.40 | https://openrouter.ai/chat?models=snowflake/snowflake-arctic-instruct | |
| Qwen2 (72B) |  | Open | 83 | 128k | $0.90 | $0.90 | https://openrouter.ai/chat?models=qwen/qwen-2-7b-instruct | |
| Yi-Large |  | Proprietary | 81 | 32k | $3.00 | $3.00 | https://openrouter.ai/chat?models=01-ai/yi-large | |
| DeepSeek-Coder-V2 |  | Open | 128k | $0.14 | $0.28 | https://openrouter.ai/chat?models=deepseek/deepseek-coder | ||
| DeepSeek-V2 | | Open | 82 | 128k | $0.14 | $0.28 | https://openrouter.ai/chat?models=deepseek/deepseek-chat |
Key Definitions
- Quality: The index shows the average performance across Chatbot Arena, MMLU, and MT-Bench, adjusted for comparison.
- Context Window: The maximum total number of tokens allowed for both input and output combined. The limit for output tokens is often much lower and varies depending on the model.
- Input Price: Cost per token sent to the API as part of the request or message, shown in US dollars per million tokens.
- Output Price: Cost per token produced by the model (output from the API), shown in US dollars per million tokens.
- Knowledge: The Knowledge cutoff for GPT models refers to the date after which the model no longer has access to new information.
Frequently Asked Questions
What is the LLM Leaderboard?
The LLM Leaderboard is a comprehensive tool designed to compare various Large Language Models (LLMs) based on multiple key metrics such as performance on benchmarks, specific capabilities, price, and other relevant factors.
Which LLM API is the best?
The “best” LLM API depends on your specific needs and budget. Our LLM price comparison tool helps you evaluate factors like quality, price, and features to find the ideal fit.
How do I compare LLM prices?
To compare LLM prices, use our LLM Leaderboard tool on the official ChatGBT website and check the cost per 1,000 tokens for each model. Compare these costs against your budget and consider the balance between price and the quality/performance you need.
