I actually read this article for some random reason (it happens sometimes) and HN has a similar problem - there are third party apps that charge money. HN has always taken a friendly approach to third-party projects as long as they make it clear they aren't official and include links to the original HN comments/submissions, but the charging money part doesn't feel right to me. I'd rather that everything HN-related be free and in the spirit of just-because.
Since Google started charging us quite a bit for the Firebase API a year or two ago, we're in a similar position to what the article describes - we're paying for https://github.com/HackerNews/API and thus indirectly funding the commercial apps.
Btw this is not to take a side on any of the Reddit stuff - that would be ultra dumb of me and anyway I'm succeeding at staying ignorant of the details. But I think it's worth saying something publicly about the analogous issue on HN because we might have to do something about it at some point.
The current API isn't particularly usable (you have to request individual items to get an entire thread) and our long term plan is to replace it with a more usable one that just gives a JSON representation of any HN URL and is hosted on our server. I've been posting about that for an embarrassing number of years (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...) but you all know how slow we are. Anyhow, whenever we get there, we'll probably do proper things like have API keys and terms of service and whatnot. If the commercial app thing is still a thing at that point, maybe we can find some decent way to address it.
I agree, charging money for an app built atop a free service is sketchy and doesn't feel right. But how much are they actually making? HN only has a few 10's of thousands of DAUs, and HN's population is notoriously difficult to monetize. Still, it wouldn't bother me one bit if the HN policies changed to forbid such parasitic behavior.
The real story is that this is all a sideshow and low-value distraction compared to the commercial entities scraping the fuck out of HN, Reddit, StackOverflow, and the Internet at large purely for their own gain without giving anything back to the data providers. The real parasites being the data-hungry LLM makers who drive zero external traffic and don't credit sources in any way.
I've actually taken all of my non-commercial websites offline for the first time in 22 years, because I don't wish to continue feeding this beast. I can't see any upside in continuing to donate to them.
Edit:
p.s. @dang: Your message was better with the burning bridges Reddit joke ;)
(responding to just the HN part here) - hopefully all so small as not to matter much. HN's lesser order of magnitude has advantages.
HN helped pg make $1B, so...
i guess that's called leverage
Why is charging money for the app sketchy? If users are willing to pay for the app then clearly it’s providing real value to them.
As long it’s clear to the user that they could get the experience for free by visiting the website I don’t see what the problem is.
With that said, I think it’s totally fair for HN to ask the app/user to pay for their API access too.
About parasitic behavior: I think if Reddit had said “in 6 months the API will no longer be free. An average user creating W posts, X comments, Y searches and Z votes will incur $1 in monthly API costs” then there would’ve been very little drama. 99% of people will be sympathetic to a company charging for a thing that costs them money to provide.
It’s the fact that Reddit is using this event to get rid of third party apps by charging ridiculous prices for the API, not honestly owning this decision and finally giving less than a month of transition before it comes into effect that has made all these people angry.
To represent the angry people as just freeloaders who demand to get stuff that costs money for free would be dishonest.
The new API price is not ridiculous. There's been misinformation about the pricing.
Here's the standard API prices of three websites:
(Price for 50 million API calls)
Imgur - $3,333
Reddit - $12,000
Twitter - $42,000
Twitter's API used to be free until Elon Musk recently started charging exorbitant prices for it.
Without a deeper understanding of what the nature of the API is, and how many requests are needed to service a given use case, the comparison doesn’t mean much.
The Apollo developer claimed that it would cost him on average $2.50 per user per month, to use the API.
That is pretty nuts for a free app, how many people would pay for that?
I have to say including Twitter — who wanted to kill of third party apps — to make Reddit seem more level headed is an interesting way of approaching it.
The gist is this: if I am the average user, not on Reddit Premium and using the native app with ads, the available numbers indicate Reddit earns $X on me. Their API pricing is at least $2X per user. So either they are seeing API users as cash cows, or they want to price them out.
And like I indicated, there’s a loting things Reddit could’ve done that would’ve led to less drama:
- Longer transition period - Outright banning third party apps - Include ads in the API - Make API access part of Reddit Premium
To me it seems that every step of the way, Reddit made the most confusing and most rage inducing choice available to them. That’s either incompetence or malice.
I don't see a problem with charging for a product (I feel like this sentiment isn't exactly unique here on HN). Just like I don't see a problem in Reddit charging these 3rd party apps to play ball. But the pricing was not grounded in reality, certainly not with but a 30-day notice. All of the developers whose apps will be shutdown at the end of the month are on record saying they were more than willing to pay a fee to access the API, but that the cost and timeline is absolutely impossible.
Particularly, with Apollo, many of his customers were on annual subs, and even if he started charging more to pay API price increase, if his customers would even bear it, it will still take much longer than a month to make that transition.
There really is no defense for the timeline Reddit put forward for this sweeping change.
So, having done some recent scraping of HN (historical front-page views only, completed in a couple of days, with only incremental updates now) ...
... and looked at the API and finding myself somewhat underwhelmed ...
... the JSON option would be interesting (much easier to parse than raw HTML, particularly HN raw HTML). Given that there are performance concerns with the primary server already, I'd suggest:
- Put JSON on a secondary server. (Automatic redirects for, say, id=?<ID>&json=1 requests being an option to transparently serve via requests made to the main site.)
- Use some sort of account-based mechanism to regulate access. I'm thinking more in terms of "how much" and "how fast" than "who" and "what", though that might also come into play.
I also suspect that a JSON delivery would also make alternate app / front-end design even more viable, so you might want to consider that in design and provisioning considerations.
Oh believe me it's never going to happen until the performance problem is fixed. That's the other thing we've been waiting on for years. It's coming though.
Rewrite in APL?
COBOL
For something like this, you need a COBOL framework you can trust.
COBOL on cogs
http://www.coboloncogs.org/INDEX.HTM
It's robust, ... performant... well robust.... Fully conformant to COBOL 85...
As a former user of Relay Pro and Reddit I paid DBrady for how his app used the free API for ME.
Me. The user. If Reddit wants to change my behavior... API away but don't go after these people who are just facilitating API access. What's next Dang? HTML browsers?
Motivation + Reddit's handling of the situation has been the main driver of the negativity. Very few people took issue with the idea of Reddit charging commercial users of it's API.
Reddit's initial announcement was that 3rd party apps were costing them money, that API access would be paid for soon™, and that NSFW content would no longer be available through the API.
This raised immediate concerns from a few different groups. Moderators rely on free 3rd party tools to deal with a lack of functionality in Reddit's own, the removal of NSFW content from the API would cause issues even if the tools themselves were exempted. Visually impaired users rely on accessibility clients to deal with Reddit's poor experience with screen readers/magnification.
Reddit scrambled to respond by stating there would be exemptions for non-commercial mod and accessibility tools. This dealt with some of the issues.
3rd party app developers then began shutting down and it was revealed that, in addition to removing NSFW content and exorbitant pricing, Reddit had given developers only 30 days to respond to the price changes.
During all of this Reddit's CEO was openly, and privately, making disparaging comments about the userbase, 3rd party app developers, and moderators.
Reddit's actions throughout have proved that their intention was to kill off the 3rd party applications and that, without community outrage, moderation and accessibility tools would have been collateral damage. It doesn't take a genius to comprehend that 30 days is not enough time for the 3rd party developers to transition their paying users to Reddit's new payment model, or that users would not have been willing to pay for a version with censored content.
There's a major difference between you guys and Reddit though - this is not your main gig. Reddit is Reddit's only thing. HN is a side thing for YC. That makes a massive difference to how much cost is reasonable to eat.
The peculiar viewpoint you’re demonstrating in this comment is similar to Reddit’s. It’s not objectively wrong, but the calculus involved definitely looks weird from the outside.
How would you quantify the value of Hacker News to Y Combinator? You’re talking about it like it’s a cost centre, a charitable contribution to society; what proportion of Y Combinator’s revenue can be attributed to the brand and content marketing benefits which flow from the (almost) entirely unpaid contributions of HN users? The Firebase API now costs “quite a bit”; is that amount comparable to the income of a YC partner? How much should a third-party developer be compensated for the work of building a native client which brings HN to a wider audience or increases the engagement of its existing audience? How much should an HN user be compensated for their submissions and comments? Which people deserve to make money here and why? Who is “indirectly funding” whom?
You're bringing in so many variables there that everyone will basically just respond with however they feel and fit their own von Neumann elephant to it: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....
That works for me - I'm not claiming to be unpeculiar and used the word "feel" in my comment above.
Spitballing here: keep unauthenticated (as in, cached) access free; and add a button on /user to pay something like $10 for write access in 3P apps.
This is the best solution, and really what Reddit should have done.
There’s no reason they couldn’t have required Reddit Premium to access the API from third-party apps. Then the app developers don’t have to deal with the burden of payment processing, Reddit theoretically earns more than it would under the API plan (the Apollo dev estimated the monthly API cost for an average user would be $2.50 (but would be much higher for actual customers due to needing to pay the Apple tax and for development costs) while Premium is $5 per month right now and already removes ads), and nobody gets angry.
Plus then you have people’s credit card information, and thus you can avoid the regulatory issues that Reddit is claiming they’re encountering with NSFW content.
The fact that Reddit seemingly refuses to even consider this option really confirms that this is being done purely out of spite over both not being able to monetize OpenAI’s training data collection plus spite over most of the third-party apps being more financially stable than Reddit currently is.
I agree with pretty much your entire post, but do want to disagree with a part.
> The fact that Reddit seemingly refuses to even consider this option really confirms that this is being done purely out of spite over both not being able to monetize OpenAI’s training data collection plus spite over most of the third-party apps being more financially stable than Reddit currently is.
I think it's a combination of spite for the LLMs (which IMO is short sighted, they will just turn to scraping HTML), but also they assume that they can make more off the data vacuum and ads serving to those users than even Reddit Gold would pay them every month.
I don't think it's spite, I think it's rational. (Not that I approve.)
They're widely expected to IPO imminently. One of the most critical metrics in the S-1 will be mobile app DAUs/MAUs. They can't credibly include the 3P app users in those figures. They also don't want to be in a situation where most of their paying subscribers are insulated from future "features" and UX "evolutions" that investors will demand.
So, it makes sense to me that they would induce a scenario where users are forced to choose between migrating to the official app or quitting reddit.
At the very least it is reasonable to separate out high volume usage versus front end replacement and paywalling the former. Since you don't monetize, my intuition is that alternate front ends shouldn't actually be changing your cost. Though I can understand wanting some sort of revenue or cost sharing if revenue is being generated - or these front ends just being free in the spirit of hn.
It's the last bit that matters to me - the sprit of the thing. The currency of HN is curiosity, not money. Though much of the curiosity is about money.
That's economics. There's no such thing as a free lunch. If HN gives out free API access and holds the line, them I don't see a problem with it. Personally I do think you should charge some minimal amount so people don't abuse it. Development for new features could also be funded from this income.
What's the order of magnitude of requests the API serve per day? I've build a demo on top of HN data and I had a feeling that most of it is quite static and could easily be cached behind something like cloudflare
I don't know. If someone wants to tell me how to look it up in the byzantine labyrinth of google's cloud admin ui, I could give it a try, but might not make it back out.
Yes to the other bit.
I built a GraphQL API to cache the responses and dataload them like 2 years ago on top of HN API so it can give nicer responses and don't hit the endpoint as much through batching. It can load entire HN thread in one request.
https://github.com/searchableguy/hackernews_graphql
It is still incomplete though because I remember you said I could not scrape the site beyond robot.txt limits which would be too less for the use case because I wanted to provide write functionality so I gave up.
Maybe people can host a caching relay and a layer like graphql on top as an unofficial way to lessen the burden on the firebase API (it's a pain to use anyway).
IDK about HN's internals but replacing that JSON API is not even a weekend of work and firebase is allegedly expensive.
> replacing that JSON API is not even a weekend of work
I have a lot of unfinished weekends of work to get done