We have around 20 servers in a colo center down the street.
At this number of servers we can still host websites that have millions of users (but not tens of millions). They are not exotic servers either. In fact by now they are, on average, around 11 years old. And costed anywhere from 2k to 8k at the time of purchase. Some are as old as 19 years. Hell, when we bought some of them - with 32GB of memory each - AWS had no concept of "high memory" instances and you had to completely pay out your ass for a 32GB server, despite ram being fairly cheap at the time.
We have no dedicated hardware person. Between myself and the CTO, we average maybe a day per month thinking about or managing the hardware. If we need something special setup that we have no experience in, we have a person we know that we contract, and he walks us through how and why he set it up as he did. We've used him twice in the last 13 years.
The last time one of us had to visit the colocation center was months ago. The last time one of us had to go there in an emergency was years ago. It's a 5 minute drive from each of our homes.
So, why exactly should we use the cloud? We have servers we already paid for. We rent 3 cabinets - I don't recall the exact cost, but I think its around $1k per month. We spend practically no time managing them. In our time being hosted in a colo center - the past 19 years - we've had a total of 3 outages that were the fault of our colo center. They all lasted on the order of minutes.
I think people who have no experience managing servers dramatically overestimate how much time it takes to manage servers. Depending on your team, it can definitely be easier to manage your own hardware than to manage your cloud infrastructure.
Yep. A lot of issues come from legacy products and configurations. You can do a greenfield infrastructure with OpenStack, VMware, or oVirt/RHEV and it's a pleasure to work with.
In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.
This is accurate in my experience. Also keeping things up and being there in case shit hits the fan is a full-time job. You can't write features and also manage servers equally well. Unless you have no life I guess.
Totally. I've had personal colo'd servers for 20 years at this point. But I'm tired of knowing that at any point I might have to wake up and haul my ass down to San Jose to swear at some piece of failing gear. I'm excitedly moving it all into the cloud.
How far are you from you collocation? Is moving that closer an option?
Colocation in SF proper is very spendy, which is how we ended up in San Jose. But waking up to go anywhere at 3 am to swear at gear is no longer on my list of fun activities. And there's no colo that will move the gear along with me when I go on vacation.
> there's no colo that will move the gear along with me when I go on vacation
This is actually the main one for me. I've managed our own servers for a decade with almost zero downtime, and very little time spent at the colo. But you cannot safely go out of town without having someone else around who is familiar enough with your setup to deal with an outage.
So moving stuff to AWS now almost entirely for that reason.
There is a middle ground. There are smaller companies where you can rent a bare metal server, usually with unmetered connection (priced by bandwidth). There is an on-call support 24x7 to replace any failing part when you call them. They can build custom servers and also give long term or bulk discounts.
I have a good experience with [1] (a smaller local company) and Hetzner [2], a bigger provider. Compare the prices with cloud. Especially if you need something RAM intensive.
[1]: https://www.superhosting.net/dedicated-servers
[2]: https://www.hetzner.com/dedicated-rootserver
I live in a software engineering world professionally but my background is in traditional "neckbeard" Linux system administration. This ends up making me "DevOps" but honestly a lot of what I've ended up doing in my career is basic sysadmin for organizations that get a remarkably long ways before realizing they need it - things like telephony and video surveillance become really unreasonably expensive when you end up relying on a cloud service because you don't have the skillset to manage them in-house.
This is purely my opinion, but I think that 1) there is a strange shortage of IT professionals (people who are not software engineers but instead understand systems) in much of the industry today, and 2) a lot of tech companies, even those that are currently well functioning, might be able to save a lot of money if they hired someone with a conventional IT background. This is a little self-serving of course, but it really does astound me when I see the bills that some companies are paying cloud services to do something that is traditionally done in-house by an IT department. And not everything can readily be outsourced to some "aaS" provider, so on top of that you end up with things like software companies with multi-million budgets running an office network that consists of a consumer WiFi router someone picked up at Fry's - not realizing that they are losing a lot of time to dealing with how poorly that ends up working.
I think part of the problem rests in academia - at least in my area a lot of universities seem to have really backed off on IT programs in favor of CS. I went through an undergraduate program that involved project management, decision analysis, and finance courses because these were considered by the college (I would say accurately) critical skills for the IT field. But that program had an incredible two students and was widely considered inferior to the CS program with hundreds.
Another part of the problem though seems to rest in industry. The salary differential between "DevOps Engineer" and "IT Analyst" is incredible when in practice they end up doing mostly the same thing in a lot of small orgs. So I end up walking sort of an odd line of "I have a long background in IaC but I also know about conference room equipment." And I'm not saying that everything with a Cisco/Tandberg badge isn't overpriced, but Zoom rooms can end up costing just as much and seem to be less reliable - not surprising for a platform which, by practical necessity of the lack of IT support in many orgs, is built on the Silicon Valley time-tested architecture of "five apple consumer products taped together."
From my experience, large enterprises sabotage the effectiveness of internal IT with bureaucracy and politics in a misguided attempt to eliminate all possibility of mistakes being made.
It's usually done with the "let's pretend it is ITIL" process.
Let me give two examples where if I had been the client then I would absolutely have sprinted for the cloud if I could, or at the very least start talking it up as much better.
1) System outage, time to fix 5 hours and 3 minutes. The 5 hours was me sitting in front of my computer with screens open showing the problem and waiting for various managers/decision-makers to fly by and take a look as they were ping-ponging around the office panicking about what would be impacted by the fix. Everything that was going to get impacted was already impacted by the system not working, and I had to explain that to them multiple times. Towards the end of the day, I eventually got the go-ahead to do the 3 minutes of work to fix the system. This system being down had prevented another team from doing any work for the entire afternoon.
2) Two full days of politics and paperwork to get approval to do 30 minutes of work, all while the client was impatiently asking "is it done yet" every few hours.
Learn it. Just like a new framework.
It's really not that hard.
There has also been an incredible propaganda campaign by the cloud providers to make this seem incredibly hard.
Agreed, it's just become part of the group think of our times.
The only hard part is finding a provider who won't screw you over. But then again this is the same as if you are to buy a car.
You buy rack space, they give you a network cable and IP details. You configure the OS with that IP and your online.
Counterpoint: The OP is saying they spend 1 day a month managing servers. It's ridiculously little. The servers might as well be unmanaged.
That's not enough time to keep OS or software up to date, or to monitor hardware usage, or to replace failing hardware and deal with spare parts.
This is just not true. A properly set up system requires very little manual work. Updated, monitoring, and alerting are all automated.
I help manage ~50 on prem servers plus NAS, switches, etc. Over the past three years, I have had to replace one hard drive and one power supply. And neither of those brought anything down because of redundancy. We get alerts when something goes wrong, but that is a rare occurrence.
This is my original point: servers won't just explode the minute you look away. It is not a full time job to "manage" them, unless you are at a real datacenter scale where a 1/1000 event happens twice a day. You set them up and then they continue to do what they do for a very long time.
Do you have any resources you could link that go through buying and setting up servers?
We have about 170 servers in a colo about 30 minutes from my home. I can go months without going to the colo. Usually when I go out there it is to replace a disk.
We have a bunch of hardware coming up on end of life. I looked at moving us to cloud. The new hardware will cost about what 3 years of AWS would cost. Considering that the hardware I am replacing is over 6 years old, we are better off sticking with on-prem.