It seems that Meta didn't mention one aspect of their "next-generation" AI training system: their incompetent web crawler hammers websites while completely ignoring robots.txt. Meta's 57.141.0.0/24 made up about 90% of all requests to my server this past week, and it didn't stop after I blocked it. They continue to make a request every 2 seconds despite me responding with a 403 every time. Truly an engineering marvel that Meta has bestowed upon us.
It seems that Meta didn't mention one aspect of their "next-generation" AI training system: their incompetent web crawler hammers websites while completely ignoring robots.txt. Meta's 57.141.0.0/24 made up about 90% of all requests to my server this past week, and it didn't stop after I blocked it. They continue to make a request every 2 seconds despite me responding with a 403 every time. Truly an engineering marvel that Meta has bestowed upon us.
nft add table inet filter
nft add set inet filter meta_crawl '{ type ipv4_addr; flags interval; }'
nft add element inet filter meta_crawl '{ 57.141.0.0/24 }'
nft add chain inet filter input '{ type filter hook input priority 0; }'
nft add rule inet filter input ip saddr @meta_crawl tcp dport {80,443} reject with tcp reset
> incompetent
I think that's not incompetence, I think that's by design.
Is this one of the pilot models they claim R&D tax credits for?
https://news.ycombinator.com/item?id=49921118
Actual title: ”Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster”
Posting this two days before earnings seems to indicate that there is going to be some bad capex news.
Imagine your job is just replacing millions of Dollars worth of hardware every day. I mean there must be a reason the hold 8 GPUs per rack spare.
shell glob rather than RE? what's the world coming to!