> But I would really prefer if Search just worked the way it did in the first place.
Which is one thing technologists often screw up: They misinterpret how people react to change; and they misinterpret how people really use their products.
Google Search used to be (and still is) a way to find places on the internet. As much as I like AI in searches, I agree; when I want to find the thing, I don't want AI getting in the way.
Their inability to adapt to fix the problems caused by AI and SEO have caused the product to fall apart and users went somewhere else. I do think however their decision to make search tedious for users by moving the correct answer deep into the page (or at least favouring pages that did that) was a brilliant play for advertising but has killed a lot of their core use, making it easier to leave.
Making a product worse for consumers and better for the provider by dropping quality is everywhere even in physical world companies. It's not even just a matter of h to e product, but support. Two decades ago, Home Depot had a push to save on labor at the store that led to having nobody there that understood what they were selling, or could help anyone with any project. Lowes ate them alive for a while, and it took them years to regain enough expertise to just not lose in the market straight up. We also see this happening to restaurants all the time.
The issue with someone like Google, or Microsoft, is that they have a larger moat in many products, so the amount of degradation that internal actors will be happy to push in search for profit is much deeper.
Of course Google is disincentivized from fixing the SEO spam problem because those sites are usually full of ads from Google's ad network. Changing the algorithm to favor noncommercial sites with no ads would hurt their revenue, so they don't do it.
> when I want to find the thing, I don't want AI getting in the way.
This is the reverse of my desires. Most pages on the current web contain a few useful nuggets buried in a shit sandwich of overly verbose text and aggressive advertising. AI extraction and summarization distills the mess to concentrate the useful bits.
I want the small web back. I want more people to produce quality ad-free content for the love of doing so and the desire to make the world a better place. And then I want a classic search engine to find stuff; one that heavily downranks content with ads.
I don't think it's that simple. In the early 2000s, Google won by showing up with an algorithmic solution to web spam. The algorithm could be gamed if you had sufficient resources, but it worked because there wasn't all that much money at stake.
Then, they essentially bankrolled their enemy by giving websites a way to monetize spammy content. By the late 2010s, they started losing the battle for many types of organic queries - although many HNers forget that the most common and "important" search queries are things like "Nike shoes", not "an implementation of a Bloom filter in Rust". And then, in the 2020s, they pioneered LLMs and made the technology public, and drove a final nail in that coffin by making it possible to produce SEO spam on any topic, including Bloom filters, practically for free.
Now, they see AI answers as their salvation, steamrolling a variety of non-monetizable use cases that never really mattered to them. And this will probably work only until AI-poisoning SEO is in full swing, at which point Google will find themselves with dissatisfied users and diminished ad revenue.
Up until around 2015 desktop traffic dominated, google pre 2015 tended to favour big comprehensive articles that covered not just the question being asked by the searcher but also everything surrounding it. Reading such content would inform you not only of the answer to the question you were asking but also would give you enough context to understand if you were even asking the right question to begin with.
To me this was an absolute golden age driven by the ability for website users to create good comprehensive content and monetise it via ads.
Post 2015 google shifted to prioritising content that directly answered the question being asked and little else. I assume because mobile users just wanted their answers which ultimately lead to the "snippets" that directly answered the query in google itself.
Google invented the tech for LLM's but did basically nothing with it until OpenAI forced their hands because they knew that it would kill the ad driven web. Once OpenAI did force their hands we end up with search as it is now.
Pre 2015 there was a point to creating good information on the web, now there isn't, it'll just be stolen by LLM's, there's literally no point beyond your own personal altruism.
Thankfully for google it owns youtube and there is still an incentive to upload useful information there, so that's increasingly where new information will live, but the web as we have known it most of our lives, is done.
Eh, I think the "golden era" you're referring to was mostly about altruism too. The vast majority of people creating helpful content on the internet (Wikipedia, Stack Exchange, Reddit, personal blogs, YT videos) have not seen a penny of ad revenue. YouTube is often held as an example of how the internet rewards content creators, but you have a couple of celebs and then a massive long tail of people who don't even clear the threshold to be eligible for monetization. Sure, 3Blue1Brown made some money. A million of people posting in-depth videos about woodworking or appliance repair have not.
(The requirement for monetization on YT is 3k "qualified watch hours" per month. If you upload 5-minute videos, and 10% viewers make it to the end, you need hundreds of thousands of views.)
But now, it's worse than altruism. The work your doing doesn't directly benefit the masses, it benefits OpenAI & co. You're not doing charitable work, you're a total chump.
> Pre 2015 there was a point to creating good information on the web, now there isn't, it'll just be stolen by LLM's, there's literally no point beyond your own personal altruism.
If you're someone providing information for free, because information wants to be free or whatever, then you should be happy when a copyright agnostic (ignorant?) AI company scoops it up for their next model. In a way, you're directly shaping the zeitgeist for everyone that uses those models.
You don't have to wait for that one lone person to stumble upon your site. Now everyone gets an answer from an LLM directly shaped in some small way by small changes to a few of a few billion weights.
People didn't provide their information because "information wants to be free or whatever", they provided it in the hope it would help people but also help fund their further exploration of their particular niche, creating a beneficial cycle where they could be funded for their hobbies and in return create useful information for others who followed in their footsteps.
The entire premise of this thread is that search is fucked in large part because there's no incentive to upload any information any more beyond altruism.
The LLM's can figure their own shit out, I'm not feeding them for free.
When I was putting information out on the internet, I was trying to find a way to unleash my passion for a subject. I wasn't trying to monetize it. At best I was hoping to cover server or DNS name costs. This expectation of monetization for every damn thing is probably my biggest gripe with this whole world we live in today. Ads are bad enough, but it pains me when someone is sharing something they are passionate about on YouTube but still decide they have to ask me to like and subscribe and scrounge for sponsors with products to pitch. It makes everything into a financial transaction. A hot take in today's world, I guess, but it does bother me.
Of course YT creators can do whatever the hell they want. They don't have to listen to me.
From their point of view they didn't misunderstand or screw anything up, they changed it in a way to increase how much money they make from it. That's the purpose it serves them. And they'll do the same with AI when they feel able to, same as any other provider.
I havent made the leap yet, but it's pushing me closer and closer to a paid service like kagi. The thought of paying a subscription for search is abhorrent but it might be the only option for sanity.
While I'm no longer a customer, using Kagi showed me just how much the Google experience can be improved. The $10/month is tiny for the quality of life improvement.
I ended up building my own search instead - whoever controls your search results also controls how you find information and what you see, so I felt I needed to start building my own index. If you don't want to go down that rabbit hole and don't mind Kagi's politics, $10/month is a small price to pay to get a better search experience.
I started with a metasearch, since that's (mostly) all Kagi is. I send prepaid-API calls to Brave, Mojeek, Marginalia & Yep for every query, and get about 200 results back on every search. The cost is far less than I paid Kagi. My first version was a single PHP file written by Claude Sonnet 3.6 (!), any modern AI would one-shot something dramatically better. Obviously you can iterate & improve it over time yourself. Grouping the results by domain and ranking by how many search engines return the same page/domain will get you ~60% of the ranking quality.
Over the years I added more ranking signals and had Claude build a ranking optimizer. I provide "golden queries" ranked as I would prefer, and the optimizer runs experiments to optimize the weights for each ranking signal. It isn't perfect, especially since I only use on-page signals. But it has been good enough that I don't miss Google.
My own index is just a couple of SQLite databases. It is ridiculous how far you can get with SQLite, FTS Full Text Search & its built-in BM25. Each webpage takes up 10KB of index storage on average, once trimmed down to just the indexable text. I never clear my Firefox history, so I went through my history cache to find which sites I actually visit, then made a command line indexer to run on my laptop and index them in the mornings. Most sites offer a sitemap or RSS file, so you don't need to worry about crawling the site. My tiny index doesn't answer every query, and that's what the metasearch is for. But surprisingly often, that tiny subset of the internet does have the answer I need, and it saves me a paid search API fee every time it does.
My system is held together with duct-tape and hope, so for most people it's easier just to use Kagi (or Uruky.com). If you do want to build it out, the blog at marginalia.nu has far more useful technical details than anything I can offer. But if you do go down the rabbit hole, I think you'll start questioning how a giant enterprise like Google can manage to screw up search so much.
Do you worry that your methodologies will get you stuck in a local maxima (or minima?). By using combined output of metasearch and perhaps missing overlooked gems? (It's something I worry about in my own data, as those out of the norm things are sometimes very important to me in the past)
I treat my Firefox History the same way though the way I use it is a bit different right now. I would like to harness it like yours, but first I think I need to pipe it into ArchiveBox or something
I use Kagi and have considered moving to Fastmail.
Aside from the obvious larger issues with Gmail (privacy, trusting that Google won't just deactivate your account, etc), is there anything in the day-to-day usage of Fastmail that you find to be an improvement over Gmail, in the same way that Kagi is an improvement over Google search?
I don't feel like I'm a huge email user, but for me the privacy and lack of trust with Google was enough. Losing access to an email account at the whim of some Google algorithm, with nobody to contact about it is just a problem I don't need.
One thing i like with fastmail is the masked emails & its integration with 1password. It's really nice to have a separate masked email per account, and with 1password browser extension generating them when signing up is trivially easy.
This massively improves your security.
You can keep the actual email that you use to log into fastmail completely secret and never use it anywhere which makes it harder for people to attack your fastmail account.
I've been using Fastmail for a few years with my own domain, and it's great. I can't speak to Gmail and if they have this feature, but I use custom aliases with any company or service I do business with (for example, I'll do southwestairlines112026@[domain].net). When the company is hacked and my email address is leaked, or if the company decides to not respect the "Unsubscribe" button, I disable that alias and force it to do a bounce back. This is a feature I only recently discovered and love it.
One thing that stands out is IMAP access. While the fastmail web application is good, it also works great with thunderbird. Gmail on the other hand seems to apply some heavy throttling to IMAP access - folder operations, moving items, and calendar updates are much slower with thunderbird + gmail than thunderbird + fastmail.
I recently switched to Kagi after years of using ddg and have been very happy with it. You get 100 free searches. Before I made the leap, I spent a period of time with Firefox set up to use kagi if I typed `k <query>`. I found myself using kagi for the vast majority of my searches pretty quickly, the signal:noise ratio is substancially better, especially for technical subjects.
> The thought of paying a subscription for search is abhorrent
I've found myself paying for more and more products that I used to get for free. Given how expensive running a search engine is ad revenue is necessary for a free product. Advertisers pay more if they are getting clicks and conversions... Obviously this ends up impacting search results. Additionally, the tradition SEO model went from bad to worse with LLMs.
Anyways, up until very recently I would've agreed with you, but ultimately this was a pretty easy expense to justify after a short trial.
Just as a counter, I tried Kagi and was unimpressed. I didn’t get any better results than normal google stuff, I couldn’t see what would be worth paying for.
I don't see why it's "abhorrent". In fact, paying for a useful service that somebody else provides is common sense.
I've been a happy customer of Kagi since 2022, and for me it's well worth the money. It gets better results than Google or DuckDuckGo, and I trust the results more because they're not pay-to-play like Google.
AI web search is really only as good as the index it is relying on. Maybe it can generate stuff without needing to search but...
Just hope that the index quality does not get further deprioritized... It might make sense to make a public indexes if corporations aren't interested in maintaining them...
Google brought this situation on themselves by making their product worse in order to pump up ad revenue. Many of their users were already looking for an excuse to jump ship. I would have done the same myself, but my clients aren’t exactly looking for help ranking on Kagi, and I need to be where their customers are.
"Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket"
- Eric Hoffer
I've been playing the Ocarina of Time remake and whenever I get stuck trying to get a certain item or progress on a quest, I'll fire up Claude voice chat and ask it for guidance, and it regularly gets confused and gives me wrong answers (somewhere around ~30% of the time).
Given how long these models have been out and that it's a highly constrained game world from almost 30 years ago which is documented top to bottom, I struggle to understand how Claude is still so bad at straightforward questions with straightforward answers. This was a solved problem (albeit with a less modern UX) in the legacy internet and it doesn't actually feel like getting the right answer is necessarily easier.
Google used to return these documents, or at least forums and communities who knew where these guides were. Google no longer connects communities together however.
LLMs don't either but the hope was that they at least read all of the old internet and have it somewhat memorized.
Haven’t we moved past an assumption/need that these (frontier) llm (-backed services) have memorized their training data now that tool calling has proven to be so effective? The model knows enough about the question to be able to search for sources that contain the answer and then summarize the findings back? I’m shocked Claude voice ever makes a mistake on this, does it not have agentic capabilities backing its responses?
> I struggle to understand how Claude is still so bad
Claude is absolutely killing it for me. Man-years and man-years of backlog work, hobby projects, optimizations, research, experiments, finally implemented. Bloated apps finally optimized to fit on micro instances, slashing my infra costs. Recovery from operations incidents in a third of the time.
If I could make Claude forget everything it every knew about Ocarina of Time to make it 1% better at what it already does so well, I'd press that button in a heartbeat.
This feels like a very odd use for Claude as opposed to any LLM focused on web search, but more to the point 30 years ago you could hit up Gamefaqs and get a plain-text document to download and manually search it locally to your heart's content in milliseconds. A lot of LLMs are hitting the mountains of garbage in, tiny pile of garbage out problem more than being bad LLMs.
I've been playing through Ship of Harkinion and have also been using just standard brave search with cheap brave AI that's free and been asking questions when I get stuck in my playthrough and it's nailed it every single time. Probably two dozen questions so far in my playthrough.
Claude should be an order of magnitude better than that. So, I can't say I understand how your experience has been so bad. Very surprising.
In general anecdotes aren't great for much unless you understand what the person is doing.
For example in the earlier days of the net lots of people just sucked with the search engines that existed while I found what I wanted quickly looking for in the same topic.
How you ask the question is still important, and will probably always be important.
I remember in around 2011 realizing that ~85% of the first page of Google search results was ads. I haven’t used Google search much in the last 5 years, mostly due to this but also because of how they rewarded blog spam.
Google's non US/EN results MUST be absolute shit. That's the only take-away I've ever been able to get out of comments like this.
I don't know if it's the language piece, the geography and routing, maybe something else in the backend user profile... Idk
There is no other search engine that seems to come remotely close for me for actually showing me what I'm looking for, and on the first page. I've tried dozens probably over the years based on random comments from people, but never find anything else close to the results quality Google search gives me.
I wish I knew for sure what the difference is! Maybe query verbage, search modifiers, etc?
Maybe Google search is like an LLM - if you’re asking for a code reference it works, but if you’re asking for something else (music theory?) it just doesn’t?
Point is maybe it depends on topic maybe?
These days I use a couple different search sources both paid and free and it’s been that way for a couple years.
I am wondering if something in their user profile or identifier is tainting the results.
I've had people on this very website tell me they searched for XYZ and couldn't find it, so then out of curiosity I will try and replicate it.
I have flipped on a vpn, in incognito mode with tons of anti-fingerprinting tools, and the XYZ thing will be the first or second result on the page.
IDK, I just find it weird as hell and would love to know why.
> Maybe Google search is like an LLM - if you’re asking for a code reference it works, but if you’re asking for something else (music theory?) it just doesn’t?
Possibly.
Maybe it is the way they structure their query? I've been using it since release and the death of altavista and others, so I may not even be aware of how the two may differ. I have heard some say that natural language search was what they always /expected/ from google search, so maybe there is something to that.
The thing about it is before Google killed search from a Product standpoint, they were killing search from a policy standpoint by rewarding SEO hackers and other junk behavior. It's been over 10 years now that most content returned from a google search is gigantic catch-all pages with sections like:
"What time is the football game on tonight? People often ask what time the football game is on tonight, and to find out what time the football game is on tonight. Tonight the football game is on at a time when the football game is on tonight".
People started to use LLMs for search because they actually return the desired content in a form the user wants to read. I unfortunately actually like AI mode because of this even if i recognize relying on AI summaries for search has its problems.
You can only scale moderation so far, and non-neutrality exposes you to liability (not moderating something can be construed as permitting it)
If Google really wanted they could've let us self moderate / let us install crowdsource block lists .. but I'm betting they decided against, as it'd help people block ad-funded sites their monopoly depends on.
Google gives away so much value for free. Google Maps for example. Free, and great.
Now they're giving away their market position ;) halting all the things inside Google that led to good products. Thanks Google!
We haven't seen it manifest yet because alternatives need to emerge to jump to.
But Google is rolling over right as it gets easier for a small hungry team to build an amazing thing. Here comes a new era of great and useful stuff for us all.
It'll take a moment. Monopolies don't crumble overnight.
If you believe this is how it should go, then wherever you can, do your part. Help alternatives get so good that they stop being the "alternative".
It feels pretty duplicitous that the staff technology writer for The New Yorker is publishing their work behind a paywall, but doesn't see the value in paying for great search results (via kagi).
Paywalls contributed to this too. Maybe it’s covered in the article? This reader will never know. In the past, these institutions held a lot of sway with me, but now none aside from the frustration of the paywall.
LLMs (and people by extension) need powerful search tools more than ever.
The internet is going to close up as content becomes increasingly valuable. The free ride ended a while ago with paywalls everywhere, but even social networks have raised shields.
I think that primary thing that wont survive is people writing stuff for public - exactly what was found on blogs, web pages, when institutions and companies gave out.
Since no one will find that, you might as well write for yourself. And if you are wroting for just yourself, you will keep it for yourself or just wont bother.
Oh, people will still write in public. I do. It's the people that wanted credit for what they wrote in public that will stop doing it. But I think everybody else can just keep on doing what they were doing.
I’ve been using Kagi search for a while, and had no real complaints, as the search does what I’ve always expected it to do. For a while I completely ignored Kagi assistant because why the fuck would I want a slower and verbose version of a search that already worked fine?
In the past few months though, I’ve been heavily using the assistant set to use deepseek to assist with putting together a Minecraft 26.2 mod pack. It’s saved me so much time and frustration it’s unreal. Turns out there are certain things that are complex enough that regular search really isn’t all that great for.
For example, I can now do things like ask if there’s a mod that allows torch placement from the hotbar while holding something else, via middle click, compatible with Minecraft Java 26.2 neoforge, as the one I previously used is stuck on 1.21.1. And instead of spending 2-3 hours trying a bunch of keywords and combing through poorly named and documented mods, I can quickly get fairly reasonable confirmation that this doesn’t exist, along with some alternatives.
I can also hand it a GitHub source and asking it what needs to get changed to update a mod to newer versions, giving me an idea if doing so will get within my extremely limited coding capability. A real game changer is basically asking it for a tutorial, like what’s involved with fixing a model from an outdated resource pack for a recent version and having it work with Fresh Animations. This would previously have required many days of painful and frustrating searching.
I can also ask it for basic scripts, rather than spending time hoping to find something someone else already made. Such as taking ratings from my navidrome database, and writing them into tags in the related songs. Just the other day I asked it for a violentmonkey script that can take a Reddit comment with a video, and give me a url that yt-dlp can handle as is without having to dig through a bunch of nonsense manually. I would have never have had the expertise or time to learn to implement these things myself.
Of course, one must keep in mind that the knowledge the llm has has a time cutoff, but I’m frankly have so much of a better time than before.
I can now see myself using the next update of Claude to generate mods (the current version does not have much knowledge of Minecraft 26.2) with features I want. While I’m reasonably competent technically, actual coding is beyond me, and is far beyond my free time to learn.
While AI doesn’t replace my use of classic search wholesale, it does take a set of problems I was trying to poorly solve with classic search, and actually gives me what I want.
"The largest tech companies are investing so much capital in A.I. infrastructure chips, data centers, intensively trained models that they must make the bet pay off, one way or another."
It sounds like it's working at least with respect to this author
He seems to have no alternaative but to keep feeding training data to a chatbot/agent
I am still using traditional search, done from the command line in textmode
No browser, no Javascript, no cookies
There's even way to still get "10+ blue links" from Google, it's "hiding" in plain sight, unchanged since the mid 90's, though I expect this will be shut off eventually
"But I would really prefer if Search just worked the way it did in the first place."
It's possible but the user might have to stop using Google, Anthropic (Google is an investor), Spear Street Technology, Inc., and so on
I think this author like so many other "tech" journalists will go whichever way the wind blows, whilst making futile complaints into the ether
"I find myself using A.I. to fight against aggressive A.I."
Here is what one security researcher said about "Instinct"
> But I would really prefer if Search just worked the way it did in the first place.
Which is one thing technologists often screw up: They misinterpret how people react to change; and they misinterpret how people really use their products.
Google Search used to be (and still is) a way to find places on the internet. As much as I like AI in searches, I agree; when I want to find the thing, I don't want AI getting in the way.
Their inability to adapt to fix the problems caused by AI and SEO have caused the product to fall apart and users went somewhere else. I do think however their decision to make search tedious for users by moving the correct answer deep into the page (or at least favouring pages that did that) was a brilliant play for advertising but has killed a lot of their core use, making it easier to leave.
Making a product worse for consumers and better for the provider by dropping quality is everywhere even in physical world companies. It's not even just a matter of h to e product, but support. Two decades ago, Home Depot had a push to save on labor at the store that led to having nobody there that understood what they were selling, or could help anyone with any project. Lowes ate them alive for a while, and it took them years to regain enough expertise to just not lose in the market straight up. We also see this happening to restaurants all the time.
The issue with someone like Google, or Microsoft, is that they have a larger moat in many products, so the amount of degradation that internal actors will be happy to push in search for profit is much deeper.
Of course Google is disincentivized from fixing the SEO spam problem because those sites are usually full of ads from Google's ad network. Changing the algorithm to favor noncommercial sites with no ads would hurt their revenue, so they don't do it.
> when I want to find the thing, I don't want AI getting in the way.
This is the reverse of my desires. Most pages on the current web contain a few useful nuggets buried in a shit sandwich of overly verbose text and aggressive advertising. AI extraction and summarization distills the mess to concentrate the useful bits.
I want the small web back. I want more people to produce quality ad-free content for the love of doing so and the desire to make the world a better place. And then I want a classic search engine to find stuff; one that heavily downranks content with ads.
There is https://marginalia-search.com/
100% agree, but I don't know how to turn back the clock to browsing weird little sites of back text on gray background in NCSA Mosaic.
Build a search engine that prioritizes that content, and let folks figure the rest out?
It probably won't be huge but isn't that kind of the point?
But that still won't give you all the capability. Often the info I'm looking for isn't just in one place.
Kagi Small Web
What wikis do you use?
"Siri, please spell out those URLs for me?”
I don't think it's that simple. In the early 2000s, Google won by showing up with an algorithmic solution to web spam. The algorithm could be gamed if you had sufficient resources, but it worked because there wasn't all that much money at stake.
Then, they essentially bankrolled their enemy by giving websites a way to monetize spammy content. By the late 2010s, they started losing the battle for many types of organic queries - although many HNers forget that the most common and "important" search queries are things like "Nike shoes", not "an implementation of a Bloom filter in Rust". And then, in the 2020s, they pioneered LLMs and made the technology public, and drove a final nail in that coffin by making it possible to produce SEO spam on any topic, including Bloom filters, practically for free.
Now, they see AI answers as their salvation, steamrolling a variety of non-monetizable use cases that never really mattered to them. And this will probably work only until AI-poisoning SEO is in full swing, at which point Google will find themselves with dissatisfied users and diminished ad revenue.
I see a different timeline...
Up until around 2015 desktop traffic dominated, google pre 2015 tended to favour big comprehensive articles that covered not just the question being asked by the searcher but also everything surrounding it. Reading such content would inform you not only of the answer to the question you were asking but also would give you enough context to understand if you were even asking the right question to begin with.
To me this was an absolute golden age driven by the ability for website users to create good comprehensive content and monetise it via ads.
Post 2015 google shifted to prioritising content that directly answered the question being asked and little else. I assume because mobile users just wanted their answers which ultimately lead to the "snippets" that directly answered the query in google itself.
Google invented the tech for LLM's but did basically nothing with it until OpenAI forced their hands because they knew that it would kill the ad driven web. Once OpenAI did force their hands we end up with search as it is now.
Pre 2015 there was a point to creating good information on the web, now there isn't, it'll just be stolen by LLM's, there's literally no point beyond your own personal altruism.
Thankfully for google it owns youtube and there is still an incentive to upload useful information there, so that's increasingly where new information will live, but the web as we have known it most of our lives, is done.
> there's literally no point beyond your own personal altruism.
Or for your own reference later. Or for the fun of it (some of us are that type of person!). But yeah, little else.
Eh, I think the "golden era" you're referring to was mostly about altruism too. The vast majority of people creating helpful content on the internet (Wikipedia, Stack Exchange, Reddit, personal blogs, YT videos) have not seen a penny of ad revenue. YouTube is often held as an example of how the internet rewards content creators, but you have a couple of celebs and then a massive long tail of people who don't even clear the threshold to be eligible for monetization. Sure, 3Blue1Brown made some money. A million of people posting in-depth videos about woodworking or appliance repair have not.
(The requirement for monetization on YT is 3k "qualified watch hours" per month. If you upload 5-minute videos, and 10% viewers make it to the end, you need hundreds of thousands of views.)
But now, it's worse than altruism. The work your doing doesn't directly benefit the masses, it benefits OpenAI & co. You're not doing charitable work, you're a total chump.
> Pre 2015 there was a point to creating good information on the web, now there isn't, it'll just be stolen by LLM's, there's literally no point beyond your own personal altruism.
If you're someone providing information for free, because information wants to be free or whatever, then you should be happy when a copyright agnostic (ignorant?) AI company scoops it up for their next model. In a way, you're directly shaping the zeitgeist for everyone that uses those models.
You don't have to wait for that one lone person to stumble upon your site. Now everyone gets an answer from an LLM directly shaped in some small way by small changes to a few of a few billion weights.
People didn't provide their information because "information wants to be free or whatever", they provided it in the hope it would help people but also help fund their further exploration of their particular niche, creating a beneficial cycle where they could be funded for their hobbies and in return create useful information for others who followed in their footsteps.
The entire premise of this thread is that search is fucked in large part because there's no incentive to upload any information any more beyond altruism.
The LLM's can figure their own shit out, I'm not feeding them for free.
When I was putting information out on the internet, I was trying to find a way to unleash my passion for a subject. I wasn't trying to monetize it. At best I was hoping to cover server or DNS name costs. This expectation of monetization for every damn thing is probably my biggest gripe with this whole world we live in today. Ads are bad enough, but it pains me when someone is sharing something they are passionate about on YouTube but still decide they have to ask me to like and subscribe and scrounge for sponsors with products to pitch. It makes everything into a financial transaction. A hot take in today's world, I guess, but it does bother me.
Of course YT creators can do whatever the hell they want. They don't have to listen to me.
From their point of view they didn't misunderstand or screw anything up, they changed it in a way to increase how much money they make from it. That's the purpose it serves them. And they'll do the same with AI when they feel able to, same as any other provider.
I havent made the leap yet, but it's pushing me closer and closer to a paid service like kagi. The thought of paying a subscription for search is abhorrent but it might be the only option for sanity.
I have used Kagi for a few years now, worth the cash
While I'm no longer a customer, using Kagi showed me just how much the Google experience can be improved. The $10/month is tiny for the quality of life improvement.
I ended up building my own search instead - whoever controls your search results also controls how you find information and what you see, so I felt I needed to start building my own index. If you don't want to go down that rabbit hole and don't mind Kagi's politics, $10/month is a small price to pay to get a better search experience.
You made your own index, or just your own frontend? Do you have an advice on methodology?
I did both, but my index is tiny.
I started with a metasearch, since that's (mostly) all Kagi is. I send prepaid-API calls to Brave, Mojeek, Marginalia & Yep for every query, and get about 200 results back on every search. The cost is far less than I paid Kagi. My first version was a single PHP file written by Claude Sonnet 3.6 (!), any modern AI would one-shot something dramatically better. Obviously you can iterate & improve it over time yourself. Grouping the results by domain and ranking by how many search engines return the same page/domain will get you ~60% of the ranking quality.
Over the years I added more ranking signals and had Claude build a ranking optimizer. I provide "golden queries" ranked as I would prefer, and the optimizer runs experiments to optimize the weights for each ranking signal. It isn't perfect, especially since I only use on-page signals. But it has been good enough that I don't miss Google.
My own index is just a couple of SQLite databases. It is ridiculous how far you can get with SQLite, FTS Full Text Search & its built-in BM25. Each webpage takes up 10KB of index storage on average, once trimmed down to just the indexable text. I never clear my Firefox history, so I went through my history cache to find which sites I actually visit, then made a command line indexer to run on my laptop and index them in the mornings. Most sites offer a sitemap or RSS file, so you don't need to worry about crawling the site. My tiny index doesn't answer every query, and that's what the metasearch is for. But surprisingly often, that tiny subset of the internet does have the answer I need, and it saves me a paid search API fee every time it does.
My system is held together with duct-tape and hope, so for most people it's easier just to use Kagi (or Uruky.com). If you do want to build it out, the blog at marginalia.nu has far more useful technical details than anything I can offer. But if you do go down the rabbit hole, I think you'll start questioning how a giant enterprise like Google can manage to screw up search so much.
Do you worry that your methodologies will get you stuck in a local maxima (or minima?). By using combined output of metasearch and perhaps missing overlooked gems? (It's something I worry about in my own data, as those out of the norm things are sometimes very important to me in the past)
I treat my Firefox History the same way though the way I use it is a bit different right now. I would like to harness it like yours, but first I think I need to pipe it into ArchiveBox or something
Fastmail and Kagi. You won't look back.
I use Kagi and have considered moving to Fastmail.
Aside from the obvious larger issues with Gmail (privacy, trusting that Google won't just deactivate your account, etc), is there anything in the day-to-day usage of Fastmail that you find to be an improvement over Gmail, in the same way that Kagi is an improvement over Google search?
I don't feel like I'm a huge email user, but for me the privacy and lack of trust with Google was enough. Losing access to an email account at the whim of some Google algorithm, with nobody to contact about it is just a problem I don't need.
One thing i like with fastmail is the masked emails & its integration with 1password. It's really nice to have a separate masked email per account, and with 1password browser extension generating them when signing up is trivially easy.
This massively improves your security.
You can keep the actual email that you use to log into fastmail completely secret and never use it anywhere which makes it harder for people to attack your fastmail account.
In addition to the reasons you stated:
- Bring-your-own domain name with lots of useful features for aliases and routing messages around based on that
- Real human customer service, and with agents that actually want to help you to boot
- Everything follows common standards (IMAP, CalDAV, CardDAV), no need to deal with Google/Apple-specific weird stuff
- Masked email
- 60 GB storage per seat
- Actually usable and good web UI
I've been using Fastmail for a few years with my own domain, and it's great. I can't speak to Gmail and if they have this feature, but I use custom aliases with any company or service I do business with (for example, I'll do southwestairlines112026@[domain].net). When the company is hacked and my email address is leaked, or if the company decides to not respect the "Unsubscribe" button, I disable that alias and force it to do a bounce back. This is a feature I only recently discovered and love it.
One thing that stands out is IMAP access. While the fastmail web application is good, it also works great with thunderbird. Gmail on the other hand seems to apply some heavy throttling to IMAP access - folder operations, moving items, and calendar updates are much slower with thunderbird + gmail than thunderbird + fastmail.
I recently switched to Kagi after years of using ddg and have been very happy with it. You get 100 free searches. Before I made the leap, I spent a period of time with Firefox set up to use kagi if I typed `k <query>`. I found myself using kagi for the vast majority of my searches pretty quickly, the signal:noise ratio is substancially better, especially for technical subjects.
> The thought of paying a subscription for search is abhorrent
I've found myself paying for more and more products that I used to get for free. Given how expensive running a search engine is ad revenue is necessary for a free product. Advertisers pay more if they are getting clicks and conversions... Obviously this ends up impacting search results. Additionally, the tradition SEO model went from bad to worse with LLMs.
Anyways, up until very recently I would've agreed with you, but ultimately this was a pretty easy expense to justify after a short trial.
Just as a counter, I tried Kagi and was unimpressed. I didn’t get any better results than normal google stuff, I couldn’t see what would be worth paying for.
Kagi is great. My first search felt just like when google was new and SEO didn't exist.
I switched to Kagi about 6 months ago. Can't see myself ever going back to Google/DDG. Feels like what I remember Google being 15 years ago.
I don't see why it's "abhorrent". In fact, paying for a useful service that somebody else provides is common sense.
I've been a happy customer of Kagi since 2022, and for me it's well worth the money. It gets better results than Google or DuckDuckGo, and I trust the results more because they're not pay-to-play like Google.
Using Kagi for about 3 months now. Very pleased with it. You know get 50 search free without even making an account, so worth a try.
its less the paying for a sub, and more that the whole financial backing of the internet (advertising) is reaching a point where its just too much.
AI web search is really only as good as the index it is relying on. Maybe it can generate stuff without needing to search but...
Just hope that the index quality does not get further deprioritized... It might make sense to make a public indexes if corporations aren't interested in maintaining them...
Google brought this situation on themselves by making their product worse in order to pump up ad revenue. Many of their users were already looking for an excuse to jump ship. I would have done the same myself, but my clients aren’t exactly looking for help ranking on Kagi, and I need to be where their customers are.
https://archive.ph/6dZS9
https://kagi.com
have been a paying user for years, very happy with the service.
Related: https://www.wheresyoured.at/the-men-who-killed-google/
https://archive.ph/6dZS9
"Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket"
- Eric Hoffer
I've been playing the Ocarina of Time remake and whenever I get stuck trying to get a certain item or progress on a quest, I'll fire up Claude voice chat and ask it for guidance, and it regularly gets confused and gives me wrong answers (somewhere around ~30% of the time).
Given how long these models have been out and that it's a highly constrained game world from almost 30 years ago which is documented top to bottom, I struggle to understand how Claude is still so bad at straightforward questions with straightforward answers. This was a solved problem (albeit with a less modern UX) in the legacy internet and it doesn't actually feel like getting the right answer is necessarily easier.
Why would an LLM be good at giving you answers about specifically the ocarina of time remake? it seems like the worst case scenario for confusion
Because documents like this exist and it's an implicit assumption that LLMs have read them
https://gamefaqs.gamespot.com/n64/197771-the-legend-of-zelda...
Google used to return these documents, or at least forums and communities who knew where these guides were. Google no longer connects communities together however.
LLMs don't either but the hope was that they at least read all of the old internet and have it somewhat memorized.
God I hope all that lot is backed up somewhere. Maybe I need to take my own backup, all that ascii should be cheap.
Haven’t we moved past an assumption/need that these (frontier) llm (-backed services) have memorized their training data now that tool calling has proven to be so effective? The model knows enough about the question to be able to search for sources that contain the answer and then summarize the findings back? I’m shocked Claude voice ever makes a mistake on this, does it not have agentic capabilities backing its responses?
> I struggle to understand how Claude is still so bad
Claude is absolutely killing it for me. Man-years and man-years of backlog work, hobby projects, optimizations, research, experiments, finally implemented. Bloated apps finally optimized to fit on micro instances, slashing my infra costs. Recovery from operations incidents in a third of the time.
If I could make Claude forget everything it every knew about Ocarina of Time to make it 1% better at what it already does so well, I'd press that button in a heartbeat.
llms seem really bad at giving information about existing video game for whatever reason. it just makes shit up half the time.
This feels like a very odd use for Claude as opposed to any LLM focused on web search, but more to the point 30 years ago you could hit up Gamefaqs and get a plain-text document to download and manually search it locally to your heart's content in milliseconds. A lot of LLMs are hitting the mountains of garbage in, tiny pile of garbage out problem more than being bad LLMs.
You can still do that I think, although there's a lot of HTML faqs these days
I've been playing through Ship of Harkinion and have also been using just standard brave search with cheap brave AI that's free and been asking questions when I get stuck in my playthrough and it's nailed it every single time. Probably two dozen questions so far in my playthrough.
Claude should be an order of magnitude better than that. So, I can't say I understand how your experience has been so bad. Very surprising.
In general anecdotes aren't great for much unless you understand what the person is doing.
For example in the earlier days of the net lots of people just sucked with the search engines that existed while I found what I wanted quickly looking for in the same topic.
How you ask the question is still important, and will probably always be important.
the greater struggle for me to understand is why you are asking an llm this. thank you also for contributing to global warming for video game tips.
I remember in around 2011 realizing that ~85% of the first page of Google search results was ads. I haven’t used Google search much in the last 5 years, mostly due to this but also because of how they rewarded blog spam.
Google's non US/EN results MUST be absolute shit. That's the only take-away I've ever been able to get out of comments like this.
I don't know if it's the language piece, the geography and routing, maybe something else in the backend user profile... Idk
There is no other search engine that seems to come remotely close for me for actually showing me what I'm looking for, and on the first page. I've tried dozens probably over the years based on random comments from people, but never find anything else close to the results quality Google search gives me.
I wish I knew for sure what the difference is! Maybe query verbage, search modifiers, etc?
Maybe Google search is like an LLM - if you’re asking for a code reference it works, but if you’re asking for something else (music theory?) it just doesn’t?
Point is maybe it depends on topic maybe?
These days I use a couple different search sources both paid and free and it’s been that way for a couple years.
> Point is maybe it depends on topic maybe?
I am wondering if something in their user profile or identifier is tainting the results.
I've had people on this very website tell me they searched for XYZ and couldn't find it, so then out of curiosity I will try and replicate it.
I have flipped on a vpn, in incognito mode with tons of anti-fingerprinting tools, and the XYZ thing will be the first or second result on the page.
IDK, I just find it weird as hell and would love to know why.
> Maybe Google search is like an LLM - if you’re asking for a code reference it works, but if you’re asking for something else (music theory?) it just doesn’t?
Possibly.
Maybe it is the way they structure their query? I've been using it since release and the death of altavista and others, so I may not even be aware of how the two may differ. I have heard some say that natural language search was what they always /expected/ from google search, so maybe there is something to that.
The thing about it is before Google killed search from a Product standpoint, they were killing search from a policy standpoint by rewarding SEO hackers and other junk behavior. It's been over 10 years now that most content returned from a google search is gigantic catch-all pages with sections like:
"What time is the football game on tonight? People often ask what time the football game is on tonight, and to find out what time the football game is on tonight. Tonight the football game is on at a time when the football game is on tonight".
People started to use LLMs for search because they actually return the desired content in a form the user wants to read. I unfortunately actually like AI mode because of this even if i recognize relying on AI summaries for search has its problems.
Neutrality and moderation are conflicting goals.
You can only scale moderation so far, and non-neutrality exposes you to liability (not moderating something can be construed as permitting it)
If Google really wanted they could've let us self moderate / let us install crowdsource block lists .. but I'm betting they decided against, as it'd help people block ad-funded sites their monopoly depends on.
Google gives away so much value for free. Google Maps for example. Free, and great.
Now they're giving away their market position ;) halting all the things inside Google that led to good products. Thanks Google!
We haven't seen it manifest yet because alternatives need to emerge to jump to.
But Google is rolling over right as it gets easier for a small hungry team to build an amazing thing. Here comes a new era of great and useful stuff for us all.
It'll take a moment. Monopolies don't crumble overnight.
If you believe this is how it should go, then wherever you can, do your part. Help alternatives get so good that they stop being the "alternative".
> Not to be left behind, Reddit is introducing [...] A.I.-generated short-form videos based on its content.
Is this truly a value-add? Like will Reddit make more money doing this? If so, I weep for the future.
There are so many tiktok and YouTube channels that read interesting threads in TTS. Really not surprising that they are going for this
It feels pretty duplicitous that the staff technology writer for The New Yorker is publishing their work behind a paywall, but doesn't see the value in paying for great search results (via kagi).
Paywalls contributed to this too. Maybe it’s covered in the article? This reader will never know. In the past, these institutions held a lot of sway with me, but now none aside from the frustration of the paywall.
LLMs (and people by extension) need powerful search tools more than ever.
The internet is going to close up as content becomes increasingly valuable. The free ride ended a while ago with paywalls everywhere, but even social networks have raised shields.
I think that primary thing that wont survive is people writing stuff for public - exactly what was found on blogs, web pages, when institutions and companies gave out.
Since no one will find that, you might as well write for yourself. And if you are wroting for just yourself, you will keep it for yourself or just wont bother.
Oh, people will still write in public. I do. It's the people that wanted credit for what they wrote in public that will stop doing it. But I think everybody else can just keep on doing what they were doing.
People who wanted credit and people who wanted someone to read it will stop.
Which is significant anount of people who wrote. People who dont care about either are a minority of those who wrote.
I’ve been using Kagi search for a while, and had no real complaints, as the search does what I’ve always expected it to do. For a while I completely ignored Kagi assistant because why the fuck would I want a slower and verbose version of a search that already worked fine?
In the past few months though, I’ve been heavily using the assistant set to use deepseek to assist with putting together a Minecraft 26.2 mod pack. It’s saved me so much time and frustration it’s unreal. Turns out there are certain things that are complex enough that regular search really isn’t all that great for.
For example, I can now do things like ask if there’s a mod that allows torch placement from the hotbar while holding something else, via middle click, compatible with Minecraft Java 26.2 neoforge, as the one I previously used is stuck on 1.21.1. And instead of spending 2-3 hours trying a bunch of keywords and combing through poorly named and documented mods, I can quickly get fairly reasonable confirmation that this doesn’t exist, along with some alternatives.
I can also hand it a GitHub source and asking it what needs to get changed to update a mod to newer versions, giving me an idea if doing so will get within my extremely limited coding capability. A real game changer is basically asking it for a tutorial, like what’s involved with fixing a model from an outdated resource pack for a recent version and having it work with Fresh Animations. This would previously have required many days of painful and frustrating searching.
I can also ask it for basic scripts, rather than spending time hoping to find something someone else already made. Such as taking ratings from my navidrome database, and writing them into tags in the related songs. Just the other day I asked it for a violentmonkey script that can take a Reddit comment with a video, and give me a url that yt-dlp can handle as is without having to dig through a bunch of nonsense manually. I would have never have had the expertise or time to learn to implement these things myself.
Of course, one must keep in mind that the knowledge the llm has has a time cutoff, but I’m frankly have so much of a better time than before.
I can now see myself using the next update of Claude to generate mods (the current version does not have much knowledge of Minecraft 26.2) with features I want. While I’m reasonably competent technically, actual coding is beyond me, and is far beyond my free time to learn.
While AI doesn’t replace my use of classic search wholesale, it does take a set of problems I was trying to poorly solve with classic search, and actually gives me what I want.
"The largest tech companies are investing so much capital in A.I. infrastructure chips, data centers, intensively trained models that they must make the bet pay off, one way or another."
It sounds like it's working at least with respect to this author
He seems to have no alternaative but to keep feeding training data to a chatbot/agent
I am still using traditional search, done from the command line in textmode
No browser, no Javascript, no cookies
There's even way to still get "10+ blue links" from Google, it's "hiding" in plain sight, unchanged since the mid 90's, though I expect this will be shut off eventually
"But I would really prefer if Search just worked the way it did in the first place."
It's possible but the user might have to stop using Google, Anthropic (Google is an investor), Spear Street Technology, Inc., and so on
I think this author like so many other "tech" journalists will go whichever way the wind blows, whilst making futile complaints into the ether
"I find myself using A.I. to fight against aggressive A.I."
Here is what one security researcher said about "Instinct"
https://www.businessinsider.com/former-meta-security-chief-a...