💾 Archived View for tilde.pink › ~maria › log › 2022-05-17_cloud_insanity.gmi captured on 2022-06-03 at 23:21:36. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
does bare-metal still exist? i suppose it does. somewhere. does it get less? pretty sure it does. i base this on telltales, and none ever end with "...we moved from cloud to bare-metal"
the cloud promise is flexibility, reduced costs, less maintenance, less overhead, easier everything. whether it can live up to those promises is highly dependent on factors. maybe it's comparable to the agile promise "oh if you do it right, it definitely works" and when it doesn't, you probably haven't done it right!
unfortunately my workplace has started to migrate into this cloud thing. due to constraints of the health sector, we're not allowed to use any company based outside of the EU. this rules out... uhm... everyone. this mostly is a political decision to begin with, but small companies always have less space for arguments as big players. and so here we are, on the open telekom cloud. about 6 years ago i was part of another cloud move, on the software end. after the cloud came kubernetes, again on the software side of things. when i think back, all it did was cause a ton of dismay and annoyance. my main product was a very specific tcp proxy that had a whole proxy farm behind it and was making smart decisions which proxy to use when and how often. highly cpu and network intensive, very little ram usage, yet unoptimized code caused a lot of ripple effects.
the open telekom cloud ain't bad. it really ain't. it's based on openstack and they created a ton of extensions on top that work nicely. it's pretty raw, like aws was during my last cloud move. nothing wrong with raw. back then we wrote 80% of the glue code ourselves. our own deploy, our own provisioning, our own logging, our own everything. even for kubernetes (which was self-hosted because aws didn't have the managed cluster available in europe) we had our own helm replacement that used kubectl underneath. since i looked at it from an application side, i cared little for the details, but moving to smaller, dedicated containerized services felt like it moved complexity out of my application. at times i was even enthusiastic about it.
seeing this move unfold from the operations side makes me cringe. we have a completely viable setup on filoo. it's overprovisioned as fuck. we run application kvms and provision them with maas and ansible. consul handles all of our discovery and dns needs, and the product monolith is running nicely in all of this. peripheral services are spread across the thing and either have a dedicated bare-metal machine or their own kvm. there is a lot of optimization potential here, and i am not arguing that we're bleeding edge or solved this smart in any way. no definitely not! but it works and the amount of wake-up calls i get from alarms are neglectable. our scaling option is to buy another big-ass machine and ... well, yes that's it.
the incentive to migrate was not triggered by a consultant agency, not by sales representatives of any major provider, but by the cto who has a hard time leaving the developer days behind. imagine everything running on kubernetes, it'd solve so many pain points we have right now. imagine the scaling options we'd have. imagine how easy logging and monitoring would be. imagine how....
we ended up heeding the call (or rather the decision was a management one and we just went along or quit). the new team lead has done this before, once. so we're adopting "industry standards" and doing our best to use managed services where possible to avoid having to rebuild our infrastructure from scratch. we've been at it for months now. puzzled with questions like "how do you run this on an auto-scaling group?" and "do you even need discovery on the cloud?" and "how can you tie helmchart and terraform together so you can guarantee secure layer 4 network flows?", we sustain the fight against the false prophets and their cloud promise.
when we're done we won't be doing this much infrastructure and move up in the ladder. we'll have time to do other things
or so they sow hope. a hope that is just as believable as flying camels. in a company that needs to self-host everything from logging down to application crash monitoring due to data security, i am sure that the cloud will solve the hamster wheel problem, of course it will. it's the cloud! it's not opinionated, it's not complicated, and it doesn't make anything harder to add between 2 and 6 new layers of complexity beneath.
so why do i make it sound so bad? well. i am on gemini. my main computer is an rpi4. i like my xterm and other optimized programs. i am convinced that solving scaling with computing resources is a mistake. i think removing layers of managed software is a good thing. there's three elements to it and you can only have two of them. complexity, time and sustainability. the cloud tries to pick time and sustainability. bare-metal is more flexible, but it does eat up way more time than the alternatives. there is nothing wrong with spending time on this though.
you cannot reach enlightenment by outsourcing complexity
and i wish someone listened to me and we could have started to remove unnecessary complexity. instead we're dragging it to the cloud, making everything more complex and convoluted. i guess this isn't how i can get enthusiastic about something.