đŸ Archived View for crm114.space âș reading-list âș adfdebadd0afe60110fe035bb2f1d6d2.gmi captured on 2023-01-29 at 02:55:40. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
â©ïž Go back to the list of web bookmarks
†℠†℠†℠†℠†℠†â„
DevOps started as a well-intentioned set of practices and culture. Over the years, it has devolved into an unholy beast of division and tunnel vision. Why did we stop dreaming bigger? What happened to tearing down silos, increasing engineering velocity, and adding value? Remember? The things DevOps was supposed to do?
But the reality is, outside of FAANG and the most well-funded companies, your team is probably doing one of the following:
Congrats, thatâs not DevOps. Iâd wager most of what they are doing is using Terraform and YAML to do menial tasks for the engineering team.
Need a database? File a ticket with DevOps.
Need an IAM role? File a ticket with DevOps.
Before long, your massive team of engineers has fully saturated your understaffed âDevOpsâ teamâs backlog.
This approach does not scale.
DevOps, the free-for-all where each engineer does the necessary ops to do their job.
Best practices? Weâll invent them along the way!
Secure? Iâm too busy increasing conversion rates!
Naming conventions? Nah. NAH. nah-prod. PROD-NAH. PRODUCTION-NAH
Cost management? Deleting unused cloud resources? Nope, weâve got AWS credits to burn! đ„
The problem is most engineers donât want to do operations work.
They want to build the product. They want to add tangible value. But organizations will force this upon them, and engineers will either work with what theyâve already got, guess their way through some new cloud service, or the DevOps tasks will slowly become the responsibility of a few individuals that become the âDevOpsâ team.
The worst part of this scenario is that deadlines rule, and product stakeholders don't see ops.
When was the last time you saw a product manager high-five the ops person and say, âfast fucking autoscaler!â?
Slowly your team feels the malleable nature of DevOps stagnate into rigid infrastructure as teams stop making the hard operations decisions and just bend their software around what they have.
In the end, all infrastructure eventually becomes a platform - how easy is yours to change?
Iâve been on the operations side of engineering for a long time. Iâve been developing in Ruby since Rails 1.0. Before that, I wrote some of the trashiest PHP the world has seen. I had the opportunity to migrate from a data center to AWS EC2 when it was launched in 2006. As Iâve become more experienced in operations, I was âpigeonholedâ into the âDevOpsâ role.
To date, Iâve deployed over 200 production-grade Kubernetes clusters across a handful of companies.
Want to know a secret?
Iâve copied and pasted the same damn Terraform modules for every single one. My job felt like a scam, but companies pay me for my expertise in Kubernetes, not for writing Terraform. I had built a copy-paste-driven, single-user platform-as-a-service and no one cared so long as nothing was broken.
Youâve made it this far, so Iâm just going to say it:
The companies building âDevOpsâ teams are going in the right direction, but they need to be moving away from infrastructure configuration management and towards platform engineering and enabling developer self-service.
The knowledge silos are good. The silos are a feature, not a bug.
Expertise is a good thing.
DevOps is bullshit.
Continuous advancement through teamwork, efficiency in communication, elevated morale, leaving the door open to suggestions for betterment⊠sign me up, right?
DevOps loop, obviously bullshit đ·
This model worked when the cloud was simple. It made sense, even. Back when we had a few virtual machines, maybe an S3 bucket and a couple of queues, this loop was achievable. What threw a wrench in everything was the growing complexity of our systems, the operational burden of those systems, and compliance. For many organizations, operations teams inevitably become DevOps gatekeepers at every phase of the loop.
Planning today requires knowledge and trade-offs of cloud services. As the cloud has gotten more complex, teams are left with a few options: work with what theyâve got or sync with ops to see what their backlog looks like.
Organizations that have invested in platform engineering have either a golden path already identified or enable teams to research and develop new golden paths rapidly. This flexibility is key to a great platform. We arenât building âgolden roadsâ, they are paths. Golden paths should provide direction and guardrails but be adaptable when the business needs change.
Coding is becoming more and more of a hodge-podge of cloud APIs. This is a good thing! We should want to write and run less code.
Do you know what's worse than waiting through an ops backlog during the planning phase? Missing your deadline and working late because youâre waiting for someone on the ops team to update IAM policies and create KMS keys because you didnât realize the SNS Topics were in a different region than your SQS Queues.
A great internal developer platform has conventions in place to handle the small tedious bits of the cloud, like IAM and KMS, that take the most amount of time to address. They focus on an engineer's intentions, not making engineers worry over implementation details.
Ah yes, CI/CD. The fuzzy boundary between development and production. These phases are so fraught with bullshit that theyâve become meme-worthy:
But it's not an âops problemâ, it's an entire organization problem. Iâm convinced that the CI/CD phase is the root of where frustration and division grow between operations and engineering teams.
Take this fairly common example:
In production, we are running in containers, but developing in the container is too slow, so the team leans towards asdf and a README full of stuff to copy, paste, and pray. During a sprint, an engineer adds convert (ImageMagick) to the mix to support manipulating images and forgets to update the Dockerfile, and then production goes down.
This is where trust between developers and operations begins to erode.
Internal developer platforms should allow an engineer to run their application where they want, how they want. Lambda and Docker, sure. K8s + Buildpacks, sure. VMs and a tarball, sure.
Unfortunately, many of the platforms today force developers to run on Kubernetes. A great developer platform meets engineers where they are. It should be flexible in architecture but opinionated in self-service.
The final stages of this âperfectâ loop model are arguably where Ops should be. Thatâd be great, except now weâve got SREs to cover that. Huh.
If the âDevOpsâ team ships a Postgres RDS instance it will run fine forever, that is until an application starts using it. All of a sudden a cascade of N+1s hit, the CPU spikes, and queries grind to a halt. Who is woken up? And why does this always happen at 2 AM? In this scenario, there is nothing for operations personnel to do, yet here they are.
Do you still think DevOps is doing what itâs supposed to do?
The expertise that operations and SRE teams have is critical to developing secure, scalable systems, but the old idea of âDevOpsâ and the hurdles our industry has turned it into is holding us back.
Iâve spent a good portion of the last two years talking to teams about their cloud infrastructure and DevOps processes/culture. Iâve heard the above rant in some form from team after team. What's more concerning about the state of DevOps today are some of the other things Iâve heardâŠ
Show of handsâhow many people in your organization think CI/CD is DevOps?
Show of handsâhow many people in your organization think they donât need DevOps because they run serverless?
Show of handsâhow many of you think the above two interpretations are a problem?
To most, the term âDevOpsâ has completely lost its meaning.
If youâre really really doing DevOps, do you think reinventing all of that commodity is really worth your engineering teamâs time? Is that a good investment for the business?
âYou build it, you run it.â Pffffft. More like: You build it, but it takes longer than your sprint estimate, and you cut corners on the ops bit that doesnât fit your âdefinition of done.â
Knowledge silos and expertise are two sides of the same coin. From full stack engineering to DevOps practitioner, our industry loves to pretend everyone can do everything. Weâre an industry of hobbyists. We love to tinker. I don't know if we are fooling ourselves or if the industry has been exploiting our hobby-driven nature, but it's time for DevOps to get thrown out of an airlock.
The growing zeitgeist is that âplatform engineering is the future.â[1][2] And given that I co-founded a product in the space, I sure hope so! Unfortunately, organizations canât get there by expecting the âDevOpsâ team to do it. Iâm sorry, but copy/pasting some Terraform modules between your git repos is a terrible âplatform.â Your engineers donât want to deal with it, and inevitably your ops team will be on the hook for supporting it. Hell, even HashiCorp is jumping on the âno codeâ provisioning train for their âplease contact salesâ plan. Huh, seems like all these companies with enterprise bucks are struggling too.
âno codeâ provisioning train
So how does the average organization get to the promised land of Platform Engineering?
Simple, just hire some more frontend and backend engineers to develop a great internal PaaS with all the golden paths your operations team is architecting while you are trying to build your actual product that runs on top of it.
Obviously easier said than done.
To get there, many organizations will need a reality check.
For every operations person without software development skills, there are FORTY engineers without cloud operations skills. If you are going to build an internal platform, youâll need experts with overlapping experience in both fields working together.
You are also going to need a shitload of time and budget. This isnât a hackathon project. It isnât the pet project of the new CTO with a wild hair up their ass to plant their flag on the business. You can give it a cute âskunkworksâ name to show everyone you have access to Wikipedia too, but building an internal platform is a startup within your primary business. Building an internal development platform is like changing the tires and engine of a car while it is hurtling toward a cliff.
Stay agile.
Migrate an auxiliary service to it quickly.
Get feedback from your engineering customers.
Yeah. ENGINEERING CUSTOMERS. They arenât your team anymore. They are your business's second set of customers, but if these customers arenât buying it, you end up with morale problems, engineers pining for âthe old way,â a boatload of debt, and a bunch of wasted time and effort.
Chin up! You or the new CTOâs next team can get it done.
If you are planning to build a platform, there are five core components that are required, but in our experience, there are a few more attributes that make for a great platform.
Extendable with open-source tooling. Teams and workloads are different and will have different golden paths. If an IDP has one opinionated way to run your workload, it is no better than a PaaS. An abstraction over Kubernetes is not enough. IDPs must work for all workload types, whether containerized, serverless, or virtualized.
Anti-Lock-In, you should be able to walk away from a bad build-or-buy decision without risking production or having an arduous migration process.
Security, compliance, and guardrails must be built in. Without them, you are not enabling self-service, you are enabling disservice. The fastest way to derail self-service is a CISO worrying about a breach. A simple web form or YAML abstraction over an AWS or GCP API is not enough. Expertise in practices and security must be included.
Powerful building blocks to increase engineering velocity. We have an industry-wide shortage of expertise in the cloud space, a great IDP should have safe, dependable building blocks for designing cloud services quickly.
Enable experimentation through flexibility and extensibility. If you have to reach for another tool or platform to see how your app would work in a serverless container or with a different Pub/Sub system, you are going to receive pushback from your stakeholders in your IDP. IDPs must enable experimentation so the right decisions can be made for our applications.
Ephemeral environments for applications and infrastructure must be supported. Our applications are becoming so heavily dependent on cloud services. If you canât provision dependencies like buckets, queues, or databases, when you open a pull request, then are you really getting a close approximation to production?
Configurable alerting and monitoring for provisioned infrastructure and applications with good defaults. If engineers can deploy their own resources, they must be monitored without requiring an extra tool to configure. Otherwise, the likelihood that those resources have alerts configured will decrease significantly.
Platform engineering is possible, and it is the future. Our systems are getting more complicated and higher scale earlier as more and more parts of the world get online. We're creating great new engineers daily out of boot camps, but we arenât excelling in operational maturity as an industry. We have a lot of data to protect. We are stewards of personal information. Customers assume we have a fiduciary responsibility, but we mostly act like hobbyists. We need to make sure âplatform engineeringâ is the next bullshit buzzword.
1. Sid Palas (@sidpalas) on Twitter
2. Charity Majors - The Future of Ops Is Platform Engineering
