LLM SQL Generation Benchmark Results

We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts. Using a 200 million record dataset from the GH Archive uploaded to Tinybird, we asked the LLMs to generate SQL based on 50 prompts. The results are shown below and can be compared to a human baseline.

--
human
human
--
--
--
--
--
332.6 ms
31,006,852
759.83 MB
#1
anthropic
claude-opus-4.7-fast
82.29
99.26
65.32
1.638
1.02
883.98 ms
38,047,193
813.91 MB
#2
openai
gpt-chat-latest
82.10
99.16
65.05
1.793
1.00
818.72 ms
39,159,059
941.02 MB
#3
anthropic
claude-opus-4.7
81.17
96.86
65.48
4.738
1.00
2,800.88 ms
34,437,823
811.31 MB
#4
google
gemini-3-pro-preview
79.25
98.20
60.30
49.440
1.02
682.04 ms
46,150,625
1,055.25 MB
#5
google
gemini-3.5-flash
78.79
97.53
60.05
9.736
1.00
1,484.88 ms
50,048,646
1,060.75 MB
#6
google
gemini-2.5-pro
78.64
99.31
57.98
19.100
1.06
443.22 ms
42,878,115
826.88 MB
#7
moonshotai
kimi-k2-0905:exacto
78.50
97.95
59.06
4.547
1.02
1,397.327 ms
46,877,411
848.80 MB
#8
google
gemini-2.5-pro-preview-05-06
78.37
97.81
58.94
21.273
1.00
825.714 ms
41,502,252
947.30 MB
#9
google
gemini-2.5-pro-preview
78.26
98.09
58.43
21.296
1.00
1,031.96 ms
46,575,244
975.60 MB
#10
openrouter
horizon-beta
77.80
98.83
56.78
1.401
1.02
1,202.46 ms
72,041,510
1,141.62 MB
#11
anthropic
claude-3.7-sonnet
77.76
99.44
56.08
3.149
1.10
374.224 ms
40,099,998
824.57 MB
#12
moonshotai
kimi-k2
77.55
99.33
55.77
4.265
1.06
589.22 ms
49,539,148
903.11 MB
#13
openai
o3-mini
77.11
99.39
54.83
9.886
1.14
448.84 ms
49,432,133
844.29 MB
#14
x-ai
grok-4
76.89
97.98
55.81
61.602
1.00
677.06 ms
49,360,869
1,145.95 MB
#15
anthropic
claude-3.7-sonnet:thinking
76.86
96.79
56.93
40.858
1.00
1,425.82 ms
40,046,627
947.37 MB
#16
anthropic
claude-opus-4.5
76.66
99.06
54.26
2.907
1.02
4,677.22 ms
42,411,762
294.93 MB
#17
openai
gpt-5.1-codex-mini
76.32
98.98
53.65
9.049
1.02
639.54 ms
41,901,897
971.73 MB
#18
anthropic
claude-3.5-sonnet
76.06
99.70
52.41
3.234
1.02
388.96 ms
37,145,042
684.44 MB
#19
google
gemini-3-flash-preview
75.85
93.36
58.35
2.406
1.00
6,401.3 ms
42,874,619
927.15 MB
#20
anthropic
claude-opus-4.6
75.85
99.44
52.25
3.423
1.02
2,997 ms
37,271,990
261.77 MB
#21
google
gemini-3.1-pro-preview-customtools
75.82
92.15
59.48
20.360
1.02
3,833.3 ms
49,317,675
1,056.97 MB
#22
qwen
qwen3-235b-a22b-07-25
75.77
99.23
52.31
8.620
1.18
397.755 ms
38,751,330
781.80 MB
#23
qwen
qwen3-coder
75.66
99.83
51.48
4.571
1.04
457.224 ms
46,666,126
333.42 MB
#24
qwen
qwen3-coder-plus
75.58
99.76
51.39
2.303
1.02
679.163 ms
53,829,416
387.08 MB
#25
qwen
qwen3-32b
75.56
98.17
52.95
37.553
1.06
761.347 ms
44,676,197
795.72 MB
#26
openai
o3-pro
75.42
98.25
52.59
76.620
1.04
746.8 ms
52,804,037
936.55 MB
#27
openai
o4-mini
75.39
99.09
51.69
10.228
1.08
613.66 ms
52,581,751
940.75 MB
#28
anthropic
claude-opus-4.1
75.22
98.80
51.65
6.342
1.04
580.51 ms
39,294,543
936.76 MB
#29
qwen
qwen3-max
75.12
96.83
53.41
5.172
1.17
550.043 ms
40,419,858
895.38 MB
#30
openrouter
polaris-alpha
75.10
98.70
51.51
2.754
1.00
1,160.78 ms
71,383,218
1,133.85 MB
#31
anthropic
claude-opus-4.6-fast
75.03
99.24
50.82
2.813
1.02
2,053.426 ms
34,047,552
206.91 MB
#32
openrouter
horizon-alpha
75.03
98.65
51.40
1.362
1.02
1,358.24 ms
67,797,316
1,137.85 MB
#33
openai
gpt-4.1
75.01
99.95
50.08
2.074
1.00
421.6 ms
52,027,773
246.69 MB
#34
openai
gpt-4.5-preview
75.00
99.49
50.51
2.955
1.00
442.98 ms
41,636,677
756.28 MB
#35
openai
gpt-5-codex
74.96
98.90
51.01
11.124
1.00
596.52 ms
48,389,329
1,097.99 MB
#36
google
gemini-3.1-flash-lite
74.93
98.85
51.00
1.524
1.00
1,326.8 ms
50,146,961
936.15 MB
#37
openai
o3
74.84
99.84
49.84
16.292
1.04
549.54 ms
53,315,039
303.04 MB
#38
openai
gpt-oss-120b:exacto
74.82
98.05
51.60
5.261
1.08
1,351.14 ms
51,060,147
1,021.13 MB
#39
qwen
qwen3-next-80b-a3b-instruct
74.81
99.62
50.00
1.474
1.00
421.14 ms
38,561,447
879.98 MB
#40
openai
gpt-5.5
74.74
94.33
55.16
8.863
1.00
3,651.48 ms
39,451,456
950.34 MB
#41
qwen
qwen3-coder:exacto
74.73
98.20
51.26
1.292
1.02
1,383.265 ms
46,171,856
959.85 MB
#42
x-ai
grok-3-mini-beta
74.73
98.94
50.52
7.127
1.06
651.74 ms
55,296,404
869.75 MB
#43
anthropic
claude-4.5-sonnet
74.59
99.34
49.84
3.702
1.02
585.62 ms
43,365,288
907.62 MB
#44
openai
gpt-oss-20b
74.57
99.13
50.00
2.190
1.06
818.38 ms
54,736,481
995.56 MB
#45
openai
o4-mini-high
74.55
98.83
50.26
21.133
1.04
702.64 ms
68,364,075
1,005.01 MB
#46
anthropic
claude-sonnet-4
74.48
98.35
50.61
3.915
1.02
492.708 ms
41,642,822
913.54 MB
#47
deepseek
deepseek-chat-v3-0324
74.39
99.68
49.10
5.366
1.24
362.62 ms
39,914,537
612.03 MB
#48
nousresearch
hermes-3-llama-3.1-405b:free
74.38
99.32
49.45
7.260
1.12
552.64 ms
52,968,329
862.73 MB
#49
google
gemma-4-26b-a4b-it
74.35
98.07
50.63
5.060
1.06
6,086.8 ms
56,772,354
393.05 MB
#50
qwen
qwen3-235b-a22b
74.31
99.28
49.34
36.262
1.04
439.38 ms
45,468,824
791.67 MB
#51
x-ai
grok-4-fast
74.12
98.10
50.15
10.611
1.00
762.122 ms
50,156,062
1,088.51 MB
#52
mistralai
mistral-medium-3.1
74.12
99.42
48.81
2.088
1.04
666.02 ms
53,051,447
878.95 MB
#53
qwen
qwen3-next-80b-a3b-thinking
73.96
97.96
49.96
17.344
1.02
720.837 ms
54,897,195
1,106.61 MB
#54
meta-llama
llama-4-maverick
73.96
99.61
48.30
3.095
1.04
410.78 ms
40,161,866
793.26 MB
#55
deepseek
deepseek-chat-v3.1
73.90
99.58
48.22
3.914
1.04
608.681 ms
45,651,463
322.46 MB
#56
deepcogito
cogito-v2-preview-llama-70b
73.86
99.20
48.51
2.851
1.10
847.14 ms
42,613,427
792.70 MB
#57
qwen
qwen-2.5-coder-32b-instruct
73.82
99.16
48.49
2.456
1.08
732.878 ms
46,841,414
767.00 MB
#58
google
gemini-2.5-pro-preview-06-05
73.70
99.09
48.32
20.782
1.04
579.36 ms
38,815,820
806.77 MB
#59
mistralai
codestral-2508
73.44
99.66
47.22
0.855
1.00
775.14 ms
42,657,411
620.15 MB
#60
google
gemini-2.0-flash-lite-001
73.17
98.38
47.95
1.286
1.02
1,285.714 ms
50,589,629
969.49 MB
#61
qwen
qwen3-30b-a3b-thinking-2507
73.16
99.03
47.30
19.002
1.02
602.48 ms
45,928,106
890.50 MB
#62
google
gemini-2.5-flash
73.01
99.94
46.09
2.126
1.02
337.4 ms
36,295,667
262.45 MB
#63
qwen
qwen3-coder-flash
73.01
99.46
46.56
2.453
1.02
556.74 ms
42,185,121
868.81 MB
#64
anthropic
claude-3.5-haiku
72.91
99.90
45.93
2.731
1.08
522.38 ms
47,370,988
297.58 MB
#65
x-ai
grok-code-fast-1
72.58
98.71
46.46
6.570
1.02
830.98 ms
51,488,460
1,077.86 MB
#66
thedrummer
cydonia-24b-v4.1
72.55
99.42
45.68
1.966
1.10
412.306 ms
36,265,794
823.24 MB
#67
meta-llama
llama-4-scout
72.46
99.97
44.96
2.048
1.04
289.875 ms
39,101,618
134.66 MB
#68
google
gemini-3.1-flash-lite-preview
72.44
93.48
51.39
1.517
1.00
7,438.06 ms
53,824,558
965.34 MB
#69
openai
gpt-4o-mini
72.38
99.94
44.83
2.145
1.04
690.28 ms
54,131,214
193.58 MB
#70
openai
gpt-oss-120b
72.33
97.98
46.68
3.205
1.09
533.957 ms
41,234,766
980.50 MB
#71
openai
gpt-5.1
72.33
97.30
47.36
9.993
1.02
1,506.08 ms
53,506,834
1,119.95 MB
#72
google
gemini-2.5-flash-image
72.32
99.91
44.74
1.301
1.00
408.34 ms
39,751,249
369.30 MB
#73
google
gemma-3-27b-it
72.25
98.80
45.70
5.021
1.06
694.878 ms
41,212,019
869.17 MB
#74
openai
gpt-5.4-mini
72.20
97.59
46.80
1.300
1.02
1,981.02 ms
36,763,125
1,070.25 MB
#75
x-ai
grok-3-beta
71.95
99.14
44.76
1.701
1.04
633.612 ms
42,572,577
720.40 MB
#76
anthropic
claude-sonnet-4.6
71.83
93.83
49.83
3.549
1.00
5,336.72 ms
46,682,298
961.64 MB
#77
z-ai
glm-4.6:exacto
71.67
96.66
46.68
10.752
1.04
838.771 ms
48,329,302
1,130.98 MB
#78
openai
gpt-5
71.59
98.53
44.64
25.613
1.04
643.3 ms
61,356,069
1,161.59 MB
#79
anthropic
claude-haiku-4.5
71.52
99.46
43.57
1.671
1.12
553.44 ms
49,735,847
918.65 MB
#80
google
gemini-2.5-flash-lite-preview-09-2025
71.14
98.71
43.57
1.480
1.08
1,463.2 ms
43,979,391
864.07 MB
#81
deepseek
deepseek-chat-v3-0324:free
71.07
96.13
46.01
5.875
1.11
383.682 ms
38,010,973
813.72 MB
#82
google
gemini-2.0-flash-001
70.99
99.94
42.04
1.426
1.02
350.146 ms
44,547,543
181.54 MB
#83
mistralai
devstral-medium
70.84
99.61
42.08
1.405
1.08
420.714 ms
44,380,748
715.11 MB
#84
mistralai
voxtral-small-24b-2507
70.75
99.38
42.12
0.902
1.02
610.163 ms
41,840,418
837.25 MB
#85
anthropic
claude-sonnet-4.5
70.74
93.30
48.18
3.491
1.00
6,180.14 ms
43,114,181
899.14 MB
#86
google
gemma-4-31b-it
70.73
90.63
50.82
5.276
1.00
7,081.24 ms
56,772,354
1,050.70 MB
#87
nvidia
nemotron-nano-9b-v2
70.70
98.80
42.59
12.717
1.33
483.347 ms
40,823,966
813.29 MB
#88
qwen
qwen3-30b-a3b-instruct-2507
70.57
99.69
41.44
2.806
1.09
308 ms
31,184,916
374.88 MB
#89
mistralai
magistral-small-2506
70.03
99.10
40.96
12.425
1.18
522.531 ms
39,072,130
681.69 MB
#90
openai
gpt-5.4-nano
69.88
97.32
42.45
1.792
1.14
1,843.9 ms
45,378,797
1,138.96 MB
#91
google
gemini-2.5-flash-preview
69.77
99.64
39.90
1.622
1.00
384.551 ms
42,309,547
735.32 MB
#92
openai
gpt-5-nano
69.71
97.30
42.12
27.104
1.08
1,015.46 ms
45,844,074
1,126.98 MB
#93
mistralai
devstral-small
69.18
98.78
39.58
2.412
1.09
372.696 ms
44,597,846
757.05 MB
#94
google
gemini-2.5-flash-lite
68.35
99.87
36.83
1.512
1.04
850.96 ms
39,578,736
261.53 MB
#95
anthropic
claude-opus-4
67.85
98.62
37.07
4.243
1.97
352.457 ms
28,250,540
112.14 MB
#96
openai
gpt-5-mini
67.36
97.52
37.20
20.092
1.02
906.12 ms
61,641,565
1,386.39 MB
#97
mistralai
ministral-8b
67.29
99.73
34.86
0.925
1.00
385.911 ms
40,043,041
257.63 MB
#98
anthropic
claude-3-haiku
66.14
90.04
42.24
1.544
1.31
4,766.958 ms
34,045,851
774.89 MB
#99
mistralai
mistral-small-3.1-24b-instruct
65.97
87.92
44.02
1.809
1.00
376.5 ms
37,893,118
912.60 MB
#100
mistralai
mistral-nemo
64.74
99.48
30.00
3.307
1.09
680.644 ms
48,641,279
222.69 MB
#101
openai
gpt-4.1-nano
63.93
99.93
27.93
1.538
1.06
445.694 ms
52,428,071
239.26 MB
#102
meta-llama
llama-3.3-70b-instruct
56.31
77.07
35.56
3.501
1.21
445.242 ms
38,658,489
992.39 MB
#103
openai
gpt-5-pro
55.97
79.10
32.84
288.492
1.05
660.364 ms
47,670,686
1,025.31 MB
#104
openrouter
andromeda-alpha
54.66
94.32
15.00
28.858
1.82
358.176 ms
37,959,195
240.96 MB
#105
moonshotai
kimi-linear-48b-a3b-instruct
53.81
82.34
25.27
1.305
1.57
348.686 ms
18,091,302
755.15 MB
#106
liquid
lfm-2.2-6b
52.41
98.82
6.00
1.393
1.07
428.357 ms
77,197,005
224.10 MB
#107
liquid
lfm2-8b-a1b
50.73
99.46
2.00
1.389
1.08
294.462 ms
37,560,298
133.52 MB
#108
qwen
qwen2.5-coder-7b-instruct
50.46
98.62
2.30
5.664
1.00
264 ms
29,050,068
205.58 MB
#109
deepseek
deepseek-v3.1-terminus:exacto
50.28
58.65
41.91
3.206
1.56
950.439 ms
53,138,787
1,064.91 MB
#110
inclusionai
ring-1t
50.05
98.10
2.00
26.389
1.00
293 ms
7,321,600
116.24 MB
#112
openrouter
sherlock-dash-alpha
45.73
53.21
38.24
1.237
1.22
932.811 ms
56,016,230
1,309.17 MB
#113
alibaba
tongyi-deepresearch-30b-a3b
45.05
57.82
32.29
20.128
1.13
553.026 ms
57,570,886
1,153.90 MB
#114
baidu
ernie-4.5-21b-a3b-thinking
34.63
38.57
30.68
54.468
1.15
847.854 ms
48,091,065
1,108.87 MB
#115
perplexity
sonar-pro-search
20.33
5.71
34.95
3.717
2.45
683.182 ms
56,964,923
1,347.49 MB
#116
nvidia
nemotron-nano-12b-v2-vl:free
9.01
0.00
18.03
23.357
1.86
487.333 ms
57,185,740
1,558.26 MB