points by simonw 22 hours ago Here's a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
nikcub 18 hours ago Real test here would be using tinker to fine tune a tinker model to generate pelicans
dozerly 19 hours ago I’m afraid you’re going to have to start randomizing your benchmarks somehow. I’m sure these models are trained on this problem by now. tyre 19 hours ago If they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models. tstrimple 19 hours ago Wait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?! argee 10 hours ago That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark. 0-_-0 12 hours ago It's not a benchmark, it's a meme OrangeMusic 9 hours ago To be fair, the "they're trained on this benchmark" response is also a meme.
tyre 19 hours ago If they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models.
tstrimple 19 hours ago Wait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?! argee 10 hours ago That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
argee 10 hours ago That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
0-_-0 12 hours ago It's not a benchmark, it's a meme OrangeMusic 9 hours ago To be fair, the "they're trained on this benchmark" response is also a meme.
m3kw9 9 hours ago How is this still a valid test if there is a good chance they will now specifically train on it?
Real test here would be using tinker to fine tune a tinker model to generate pelicans
well it's good they didn't train on the test!
Gist is returning 403
I’m afraid you’re going to have to start randomizing your benchmarks somehow. I’m sure these models are trained on this problem by now.
If they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models.
Wait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?!
That’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
It's not a benchmark, it's a meme
To be fair, the "they're trained on this benchmark" response is also a meme.
How is this still a valid test if there is a good chance they will now specifically train on it?