tristanj 8 hours ago

It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:

1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.

2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.

3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.

  • fsh 7 hours ago

    4. Google did a worse job than Anthropic and OpenAI.

    • dpe82 7 hours ago

      Well sure, but parent is trying to reason about why.

      • andy_ppp 6 hours ago

        It’s unlikely to be deliberate tbh, it’s more likely the data, algorithms and the training environments aren’t quite as good as Anthropic, Open AI is a lot closer to Google than people want to admit.

        • exe34 3 hours ago

          Why would data and training environment lead to frontier intelligence but slow?

          • andy_ppp 3 hours ago

            Because the environment did not reward being as efficient as other models?

            • exe34 3 hours ago

              Oh so it's using more tokens to do the same work. Make sense, thanks!

        • LoganDark 3 hours ago

          Why would a search engine that's had nearly the entire internet for decades not have the data of a lab that's only been scraping for a few years?

          • andy_ppp 3 hours ago

            Because Google are probably more hamstrung by bureaucracy and little things like the law.

            • LoganDark 2 hours ago

              Since when? Google recently disobeyed some legal requirements because "haha, couldn't catch me / too late now bitch". (This is when they cut down a forest for a datacenter)

              • tibbar 2 hours ago

                Internet data is only good for some parts of model training. That is why there is now a huge cottage industry that creates custom datasets, RL envs, etc.

                Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.

                • andy_ppp 42 minutes ago

                  Yes I'm very much talking about data used in the RL section, I'm sure Google are indeed very good at pre-training.

          • PaulHoule 1 hour ago

            The web is mostly trash. Go look at commoncrawl!

  • teknologist 7 hours ago

    Also, Google uses their own TPU architecture rather than GPU/Nvidia.

  • zarzavat 7 hours ago

    It's 1. Carbon was announced the day of Google's earnings call. It's a PR stunt.

    • kyrra 38 minutes ago

      Googlers had been testing carbon for some time prior to the announcement.

  • aberoham 6 hours ago

    Google's frontend web servers routinely time out connections from agy (the antigravity cli) when using Carbon, wasting tons of work. Like any effort level more than "low" is completely unusable.

    It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.

    One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.

    Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.

    • p_l 5 hours ago

      This has been acknowledged by recent "agy" release notes, it was timing out in client not in backend

    • kingleopold 4 hours ago

      this happened even before agy btw, their CLI was subpar level for months even before 3.1 pro release and now too

    • yard2010 2 hours ago

      I never got any google model to work using their infra - 429, billing errors and timeouts. As someone wrote here a few months ago - it's an exercise in managing frustration. So I had to resort to openrouter.

    • embedding-shape 2 hours ago

      Lots of Google tooling is the same for me, simply unusable whenever I try even simple stuff.

      I think no one tests this as a "non-Google employee" or something, they all have specific testing backends or something, and once deployed to the public, there is no one responsible for the experience of non-Googlers and no one testing it, that's why most of the Google stuff has such poor UX because of hundreds of small cuts all over the experience.

      • antihipocrat 1 hour ago

        Google never had the mandate to align design standards across internal and external tooling. BigQuery and some of the other analytics tools are great though, I'd be surprised if staff didn't use the same internally - maybe someone here can confirm?

      • kyrra 39 minutes ago

        What specific things are like this? If you have a complaint and it's something I can reproduce, I can attempt to file a bug for it.

  • vintermann 5 hours ago

    So in short, as is usually claimed about Google products from bazel to tensorflow to Kubernetes: they may work better inside Google than outside of it.

  • npn 1 hour ago

    4. they are so behind that they can not create effective datasets like other frontier labs do. So instead they do all the tricks the Chinese labs do: using RL with the single purpose of benchmaxxing the AA score.

    Look how all Chinese models, or haiku/luna class US models behave, you can see the same pattern: they think too much and only work well with a subset of tasks

    • Doohickey-d 1 hour ago

      Which is weird, because Google is in the best position for getting training data: they have YouTube (which I guess is why Google's models are good at multimedia), as well as the Google crawler. Which effectively websites cannot opt out of if they want to be Google searchable. So Google's bot probably is the _only_ bot (scraper) that hasn't been blocked from a large portion of the internet.

      • giancarlostoro 1 hour ago

        Not only that, they pioneered some of the key technologies... Who isn't still on transformer models? Also funny when you consider Meta is in a weird similar boat, introduced key things, so far behind the frontier.

        • aswegs8 36 minutes ago

          The innovator's dilemma. Large companies are structurally worse at innovation.

      • efromvt 1 hour ago

        I think raw base text training data may be decreasingly important vs good RL environments/gyms, and maybe less clear that Google wins by default there?

    • owkman 19 minutes ago

      Seeing how every new benchmark that has come out has Carbon doing well, I'd have expected the benchmaxxing allegations to die out. But they've been ok incredibly persistent. It's jagged like all frontier models. You could argue about it's warts, but benchmaxxing is unfounded. Did it benchmaxx every public and private dataset? Is that what RL is perhaps?

  • moffkalast 1 hour ago

    Maybe they should be more efficient and sell fewer ads.

  • Danywalery 1 hour ago

    Google has a huge flow of user data; they were the first to figure out about the Transformer, yet they’re lagging behind in the race. This is very strange, although their flash models are quite good, they’re expensive compared to the Chinese ones.

  • kyrra 16 minutes ago

    1) "Gemini 4 Hydrogen is adapted from Gemini 4 Carbon." From https://deepmind.google/models/model-cards/gemini-4-hydrogen... . Also, the Agy team is pushing is that Carbon is a great planner, and then use hydrogen as the worker. They want people to use it.

    2) lots of people inside Google use carbon. If they didn't care about it, it wouldn't have been pushed on all of Google.

    3) Deepmind is competing with cloud and others. they are the only Frontier Lab that actually has to compete with other business units within an org for compute. GCP and others want that compute as well.

    (Googler, opinions are my own)

horacemorace 8 hours ago

I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.

  • weatherlite 5 hours ago

    So it's slow but good? Reasonable trade off I think

    • glimshe 5 hours ago

      It depends on how much you're willing to wait. I used Argon a few times and while it did some interesting things, it was so slow that I couldn't fit it in my workflow. GPT Sol was great and 5-10x faster.

      I feel that right now Hydrogen is the only viable model Google has in cost/speed-benefit. It's cheap and makes sense for general use in the $20 month plan with Youtube and storage, but not as a prime coding model.

    • antupis 4 hours ago

      it depends but IMO Carbon is best at the moment with code reviewing and documentation writing. Opus is still my go-to model for code generation but at moment Carbon does better job for those auxiliary jobs.

      • eknkc 4 hours ago

        Why not Fable? I've mostly been using GPT side models and have recently been dipping my toes into Claude models. I went big with Fable 5.2 as it was "the best" the offered. Apart from token usage, is there any particular reason to use Opus?

        • antupis 3 hours ago

          Our organization still blocks Fable because reasons probably something related to storing prompts.

  • artdigital 2 hours ago

    My thinking as well! I really like Carbon

    It’s hella slow for sure and agy harness makes this worse by telling it to read a ton of files before making any changes!! But once it starts, its solutions are very good and it usually continues until the task is fully completed! The output was so good at times, I was honestly surprised by how thought-through and properly implemented everything was. It implemented edge cases I would have missed and made decisions that I would call sensible. For example adding a —-flag that absolutely improved the product.

    Code review still catches medium priority issues but that’s the same with all of the models.

    I’m now putting Carbon on bigger tasks that I don’t care how long they take. As I am writing this I’m having dinner while Carbon slowly chops away at a task queue on my server.

    All in all, Carbon is great, Hydrogen is great, Google is killing it!

    (agy is still bad though, but you can tweak the “read a million files first” behavior with your system prompt)

    • joshuanapoli 1 hour ago

      My experience is similar: Carbon found better ways to solve open-ended problems than Fable. In these cases, where I don’t have a solid design ahead of time, it’s OK to wait.

  • nautilus12 24 minutes ago

    I agree, the best way i can describe Carbon is that it is "cautious". Very very refreshing attitude for a model to have.

hetspookjee 1 hour ago

Might be the only one in the thread but ever since Gemini 4 Argon. I’ve become a huge fan of the Google models. I do use Carbon most of the time since it’s become available but notice that i also am falling back more and more on Argon again without much difference. For security purposes argon has far fewer triggers. I only use the Google models as orchestrator for how they talk and analyse. I let it delegate everything to Astra / Fable / Opus models

  • moffkalast 1 hour ago

    That's right, you are probably the only one :)

  • xnx 1 hour ago

    There's dozens of us!

lmeyerov 41 minutes ago

It's been jarring & confusing to go from a year and a half of Claude Code max plan, and equivalent of Codex, to Google agy & their models

My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy

While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models

My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem

timtam 5 hours ago

Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED). Gemini couldn’t finish this response I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.

I only got around the issue by strictly setting an effort budget or a stopping rule.

Lerc 7 hours ago

A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.

It really looks like something is broken more than just being inefficient.

  • teknologist 7 hours ago

    Theo seems to dislike anything that isn't Anthropic Claude. Suspicious, to say the least. On CursorBench, a good measure of real-world coding tasks, Gemini 4 Carbon comes out top: https://cursor.com/evals

    • LiamPowell 7 hours ago

      Every time I see a Theo tweet he's judging something on the most absurd metrics. Today for example, he was saying Astra 6.2 is bad because when he gave it a single line prompt telling it to create a new level for a game it used a name for the level that he'd seen before. Judging a model on how often it reuses a name is one of the most ridiculous benchmarks I've ever seen.

      As far as I can tell, all his judgements are based on similarly absurd metrics, I don't know why people treat his preferences as valuable.

    • knollimar 2 hours ago

      I’m not sure Claude favoritism is fair; he loved OpenAI for a while. He just has really arbitrary and frankly bad takes imo

    • ywvcbk 1 hour ago

      They have excluded all OpenAI models, though? Of course I doubt they'd have the highest scores but likely would be most price efficient.

  • 96khz 7 hours ago

    are we talking about token prefill here?

    • Lerc 6 hours ago

      11 seconds worth?

  • xnx 1 hour ago

    How recent is that video? Google just reduced time to first token (https://x.com/_mohansolo/status/2122821676249625014): "Time to first token (TTFT) for Carbon is now 1.5x faster on Antigravity. Will follow soon with Argon too.

    Lots of work went into this so you all could have a faster experience. More to come "

dada216 1 hour ago

ITT it's that huge max output token of 1mln tokens.

It's pretty far beyond other models (Opus 5.5 is at 128k).

timcobb 1 hour ago

really glad to see _this_ kind of analysis of these models. These type of analysis is much more practically useful than some context-less (especially with regard to time) number of "how good it is".

SubiculumCode 8 hours ago

That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.

  • ahsillyme 8 hours ago

    Uhm, I'm not a programmer but why isn't this a good look? I would want my engineers to fix problems that they notice in passing. (Edit: the "unnecessary work" angle is what irked me in the article)

    • horacemorace 7 hours ago

      Good SWEs get a feel for the minimum amount of changes to fix something. Fewer changes usually minimizes the chance for new problems to pop up. Junior SWEs usually try to do too much.

      • bot403 6 hours ago

        This. There should be 2 PRs. The core fix, the the unrelated fixes. Because I guarantee sometimes those unrelated fixes will cause regressions so they have to be scrutinized too.

    • shwetanshu21 5 hours ago

      Sometimes that problem might have downstream other effects which causes bigger problems which is why it is left as it is.

      Most of the times it has to do with refactoring downstream services or dependencies or just unexpected behaviour that shows up.

lp92 8 hours ago

>Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.

With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.

mengqijin 5 hours ago

I tested with long and complicated tasks Carbon did perform better and took almost the same time as Hydrogen, roughly 50% more cost than Hydrogen.

But with simple tasks, Carbon is way too slow, and much higher cost.

gdevenyi 1 hour ago

I just got a 6 month max subscription via the antigravity science program. I've tried to start using it and found both the harness to be terrible and the model to be very slow.

It might do good work but I'm having a hard time figuring out if it is.

ahmadyan 5 hours ago

i'm having all sort of problems with the gemini carbon lag. using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.

cybice 5 hours ago

Interesting that it looks like behaviour of cheap models, like deep seek flash overuse tool calls

5555watch 1 hour ago

Carbon is extremely good for particular research questions I'm doing, and the only hope at the moment is that they don't eventually nerf it down, especially if topics like these about the speed might raise concerns.

In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.

So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!

Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.

Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.

  • joshuanapoli 1 hour ago

    Yeah, Carbon is good when the problem needs some imagination, when a better solution is worth waiting for. Fable 5.2 moved towards faster or more efficient responses. So it’s more practical for most tasks, but tends to be less creative.

oytis 1 hour ago

They claimed a precious domain I should say, shame that they are not using it for anything kernel-related

jrflo 8 hours ago

Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?

  • murderfs 6 hours ago

    I'd guess that it's because Google eats its own dogfood, and if they didn't train the models to be overly meticulous, the existing codebase would implode under a tsunami of slop.

resters 3 hours ago

i wanted to test carbon but ran out of quota before i could get a definitive result.

user_7832 8 hours ago

I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.

I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.

Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).

  • incognition 8 hours ago

    Oddly flash 3.8 for me is good. they really pushed carbon to some other extreme

    • p_l 5 hours ago

      Heard a rumour that Carbon is ~10x the size of Opus 5.5

  • auyez 7 hours ago

    At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.

    • londons_explore 6 hours ago

      The Human mind so far has shown remarkably little ability to redesign other human minds to be smarter.

      The existence of AI able to design a smarter AI I would not say is a given.

  • wellcolormesurp 7 hours ago

    They are cracking software and math, not finance, law or medicine or whatever it is you do.

    Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.

    • oblio 3 hours ago

      > The other dominos will fall automatically once you got the engineering automated.

      Based on...? Wishful thinking?

  • SyneRyder 7 hours ago

    Are you able to give hints to some of those use cases? Recent Opus and Fable seem to have made big leaps in art and music, and I don't think all of that is the longer context window since Sonnet 3.x. But those are both things that can be created with code and mathematical thinking as well. I definitely wouldn't want to use recent models for writing though (not that I ever wanted them to write for me anyway).

    • knollimar 2 hours ago

      Opus 5.5 trashes everything not fable and astra for my knowledge work. It’s an insufferable pedant though.

      I haven’t used a lot of Carbon though.

  • vintermann 5 hours ago

    I also use LLMs for a difficult non-coding task (historical HTR), and my experience is that Gemini Argon is much better than its competitors on that.

    Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.

    Still, those are much better failure modes than hallucinating, which powerful models are still prone to.

nandanadileep29 7 hours ago

yeah i tried using it its not efficient at all. just burns up the token and not get the work done.

logicchains 6 hours ago

Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.

kristjansson 6 hours ago

Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?

NoahZuniga 7 hours ago

Google pretty clearly says in their product documentation that carbon is just a version of carbon that thinks way longer.

  • lukax 7 hours ago

    You probably meant a version of Argon?

  • kzrdude 6 hours ago

    Then that's "test-time scaling" again and Google then showed that they can reach frontier level benchmark results using it. It's pretty cool as a technology actually (the demonstrated capability), even if the model is not so useful when it's slow.