As mentioned in the past[1], this site is hosted on my own Kubernetes cluster that lives in a corner of my home. This is nice for learning/experimentation, but it needs a connection to the internet that a domestic broadband subscription doesn't really provide (static IPs and so on.)
To address this, I use Cloudflare[2]'s Argo Tunnel product to automatically create and connect via encrypted tunnel my Kubernete's deployments to the Cloudflare CDN.
In particular, I use the Cloudflare Ingress Controller[3]. It's super neat and does the job well, but it isn't supported any more because Cloudflare have a new preferred model that separates the tunnels from the provisioning (which removes most of the convenience, but which is nevertheless entirely understandable from their point of view.)
While unsupported for a bit over a year, this has nevertheless worked fine up until now. Well, up until about two days ago that is.
At that point, Cloudflare had a problem[4]. A problem which caused all the tunnels to stop working, and which took sites like mine which depend on it down. The exact details are unclear, but in simple terms it seems as if the root certificate distributed with their tunnel clients expired, leaving them unable to connect to Cloudflare.
Naturally, Cloudflare fixed this fairly rapidly. For their supported clients... To fix the problem, you just needed to download a newer version of the client and deploy.
Except the Ingress Controller isn't updated any more. And completely re-architecting the ingress for my cluster in an emergency doesn't sound like a lot of fun, particularly at a time when the day job is keeping me busy for 70+ hours a week...
Fortunately, this isn't just a post to whine about my problems. It's a post to say "hurrah for Open Source", and also to give pointers to anyone with the same problem.
Because, thankfully, Cloudflare were good enough to open source the Ingress Controller. So while they don't support it any more, you can at least go about fixing it yourself - or if you're very lucky, by the time you realise what's wrong you discover someone else already did.
In this case thanks to chungthuang[5] on Github, who has forked the cloudflare-ingress-controller[6] repo with a version with updated CA certificate. Phew!
From there, it's easy to build a new Docker image for the ingress controller. So easy in fact, that I've done it and pushed it to the public Docker Hub so anyone else can use it too - `snowgoons/argo-tunnel:0.6.5-5-cacerts2021`.
If your Kubernetes logfiles are full of:
22.02.2021 20:20:27 time="2021-02-22T18:20:27Z" level=error msg="Unable to dial edge" error="Handshake with edge error: x509: certificate has expired or is not yet valid" 22.02.2021 20:20:28 time="2021-02-22T18:20:28Z" level=error msg="Unable to dial edge" error="Handshake with edge error: x509: certificate has expired or is not yet valid" 22.02.2021 20:20:30 time="2021-02-22T18:20:30Z" level=error msg="Unable to dial edge" error="Handshake with edge error: x509: certificate has expired or is not yet valid"
Then find your deployment of the Kubernetes ingress controller (if you used a Helm chart, you want to update the `values.yaml`), and change the image:
┌─────────────────────┬────────────────────────────────────────────────┐ │ Image │ Reference │ ╞═════════════════════╪════════════════════════════════════════════════╡ │ Old image: │ `gcr.io/cloudflare-registry/argo-tunnel:0.6.5` │ ├─────────────────────┼────────────────────────────────────────────────┤ │ New, working image: │ `snowgoons/argo-tunnel:0.6.5-5-cacerts2021` │ └─────────────────────┴────────────────────────────────────────────────┘
Redeploy, and hey-presto, all is good.
Hope that helps someone out of a hole!
1: /posts/2020-05-11-snowgoonsplatform.gmi
3: https://github.com/cloudflare/cloudflare-ingress-controller
4: https://www.cloudflarestatus.com/incidents/jvydcx604vv9
5: https://github.com/chungthuang
6: https://github.com/chungthuang/cloudflare-ingress-controller
--------------------