💾 Archived View for foo.zone › gemfeed › atom.xml captured on 2024-05-26 at 14:20:44.
View Raw
More Information
⬅️ Previous capture (2024-05-12)
➡️ Next capture (2024-07-09)
🚧 View Differences
-=-=-=-=-=-=-
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<updated>2024-05-18T13:15:08+03:00</updated>
<title>foo.zone feed</title>
<subtitle>To be in the .zone!</subtitle>
<link href="gemini://foo.zone/gemfeed/atom.xml" rel="self" />
<link href="gemini://foo.zone/" />
<id>gemini://foo.zone/</id>
<entry>
<title>Projects I currently don't have time for</title>
<link href="gemini://foo.zone/gemfeed/2024-05-03-projects-i-currently-dont-have-time-for.gmi" />
<id>gemini://foo.zone/gemfeed/2024-05-03-projects-i-currently-dont-have-time-for.gmi</id>
<updated>2024-05-03T16:23:03+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Art by Laura Brown</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='ProjectsIcurrentlydonthavetimefor'>Projects I currently don't have time for</h1><br />
<br />
<span class='quote'>Published at 2024-05-03T16:23:03+03:00</span><br />
<br />
<pre>
Art by Laura Brown
.'`~~~~~~~~~~~`'.
( .'11 12 1'. )
| :10 \ 2: |
| :9 @-> 3: |
| :8 4; |
'. '..7 6 5..' .'
~-------------~ ldb
</pre>
<br />
<h2 style='display: inline' id='Introduction'>Introduction</h2><br />
<br />
<span>Over the years, I have collected many ideas for my personal projects and noted them down. I am currently in the process of cleaning up all my notes and reviewing those ideas. I don’t have time for the ones listed here and won’t have any soon due to other commitments and personal projects. So, in order to "get rid of them" from my notes folder, I decided to simply put them in this blog post so that those ideas don't get lost. Maybe I will pick up one or another idea someday in the future, but for now, they are all put on ice in favor of other personal projects or family time.</span><br />
<br />
<pre>
Table of contents
=================
Projects I currently don't have time for
Introduction
Hardware projects I don't have time for
I use Arch, btw!
OpenBSD home router
Pi-Hole server
Infodash
Reading station
Retro station
Sound server
Project Freekat
Programming projects I don't have time for
CLI-HIVE
Enhanced KISS home photo albums
KISS file sync server with end-to-end encryption
A language that compiles to `bash`
A language that compiles to `sed`
Renovate VS-Sim
KISS ticketing system
A domain-specific language (DSL) for work
Self-hosting projects I don't have time for
My own Matrix server
Ampache music server
Librum eBook reader
Memos - Note-taking service
Bepasty server
Books I don't have time to read
Fluent Python
Programming Ruby
Peter F. Hamilton science fiction books
New websites I don't have time for
Create a "Why Raku Rox" site
Research projects I don't have time for
Project secure
CPU utilisation is all wrong
</pre>
<br />
<h2 style='display: inline' id='HardwareprojectsIdonthavetimefor'>Hardware projects I don't have time for</h2><br />
<br />
<h3 style='display: inline' id='IuseArchbtw'>I use Arch, btw!</h3><br />
<br />
<span>The idea was to build the ultimate Arch Linux setup on an old ThinkPad X200 booting with the open-source LibreBoot firmware, complete with a tiling window manager, dmenu, and all the elite tools. This is mainly for fun, as I am pretty happy (and productive) with my Fedora Linux setup. I ran EndeavourOS (close enough to Arch) on an old ThinkPad for a while, but then I switched back to Fedora because the rolling releases were annoying (there were too many updates).</span><br />
<br />
<h3 style='display: inline' id='OpenBSDhomerouter'>OpenBSD home router</h3><br />
<br />
<span>In my student days, I operated a 486DX PC with OpenBSD as my home DSL internet router. I bought the setup from my brother back then. The router's hostname was <span class='inlinecode'>fishbone</span>, and it performed very well until it became too slow for larger broadband bandwidth after a few years of use.</span><br />
<br />
<span>I had the idea to revive this concept, implement <span class='inlinecode'>fishbone2</span>, and place it in front of my proprietary ISP router to add an extra layer of security and control in my home LAN. It would serve as the default gateway for all of my devices, including a Wi-Fi access point, would run a DNS server, Pi-hole proxy, VPN client, and DynDNS client. I would also implement high availability using OpenBSD's CARP protocol.</span><br />
<br />
<a class='textlink' href='https://openbsdrouterguide.net'>https://openbsdrouterguide.net</a><br />
<a class='textlink' href='https://pi-hole.net/'>https://pi-hole.net/</a><br />
<a class='textlink' href='https://www.OpenBSD.org'>https://www.OpenBSD.org</a><br />
<a class='textlink' href='https://www.OpenBSD.org/faq/pf/carp.html'>https://www.OpenBSD.org/faq/pf/carp.html</a><br />
<br />
<span>However, I am putting this on hold as I have opted for an OpenWRT-based solution, which was much quicker to set up and runs well enough.</span><br />
<br />
<a class='textlink' href='https://OpenWRT.org/'>https://OpenWRT.org/</a><br />
<br />
<h3 style='display: inline' id='PiHoleserver'>Pi-Hole server</h3><br />
<br />
<span>Install Pi-hole on one of my Pis or run it in a container on Freekat. For now, I am putting this on hold as the primary use for this would be ad-blocking, and I am avoiding surfing ad-heavy sites anyway. So there's no significant use for me personally at the moment.</span><br />
<br />
<a class='textlink' href='https://pi-hole.net/'>https://pi-hole.net/</a><br />
<br />
<h3 style='display: inline' id='Infodash'>Infodash</h3><br />
<br />
<span>The idea was to implement my smart info screen using purely open-source software. It would display information such as the health status of my personal infrastructure, my current work tracker balance (I track how much I work to prevent overworking), and my sports balance (I track my workouts to stay within my quotas for general health). The information would be displayed on a small screen in my home office, on my Pine watch, or remotely from any terminal window.</span><br />
<br />
<span>I don't have this, and I haven't missed having it, so I guess it would have been nice to have it but not provide any value other than the "fun of tinkering."</span><br />
<br />
<h3 style='display: inline' id='Readingstation'>Reading station</h3><br />
<br />
<span>I wanted to create the most comfortable setup possible for reading digital notes, articles, and books. This would include a comfy armchair, a silent barebone PC or Raspberry Pi computer running either Linux or *BSD, and an e-Ink display mounted on a flexible arm/stand. There would also be a small table for my paper journal for occasional note-taking. There are a bunch of open-source software available for PDF and ePub reading. It would have been neat, but I am currently using the most straightforward solution: a Kobo Elipsa 2E, which I can use on my sofa.</span><br />
<br />
<h3 style='display: inline' id='Retrostation'>Retro station</h3><br />
<br />
<span>I had an idea to build a computer infused with retro elements. It wouldn't use actual retro hardware but would look and feel like a retro machine. I would call this machine HAL or Retron.</span><br />
<br />
<span>I would use an old ThinkPad laptop placed on a horizontal stand, running NetBSD, and attaching a keyboard from ModelFkeyboards. I use WindowMaker as a window manager and run terminal applications through Retro Term. For the monitor, I would use an older (black) EIZO model with large bezels.</span><br />
<br />
<a class='textlink' href='https://www.NetBSD.org'>https://www.NetBSD.org</a><br />
<a class='textlink' href='https://www.modelfkeyboards.com'>https://www.modelfkeyboards.com</a><br />
<a class='textlink' href='https://github.com/Swordfish90/cool-retro-term)'>https://github.com/Swordfish90/cool-retro-term)</a><br />
<br />
<span>The computer would occasionally be used to surf the Gemini space, take notes, blog, or do light coding. However, I have abandoned the project for now because there isn't enough space in my apartment, as my daughter will have a room for herself.</span><br />
<br />
<h3 style='display: inline' id='Soundserver'>Sound server</h3><br />
<br />
<span>My idea involved using a barebone mini PC running FreeBSD with the Navidrome sound server software. I could remotely connect to it from my phone, workstation/laptop to listen to my music collection. The storage would be based on ZFS with at least two drives for redundancy. The app would run in a Linux Docker container under FreeBSD via Bhyve.</span><br />
<br />
<a class='textlink' href='https://github.com/navidrome/navidrome'>https://github.com/navidrome/navidrome</a><br />
<a class='textlink' href='https://wiki.freebsd.org/bhyve'>https://wiki.freebsd.org/bhyve</a><br />
<br />
<h3 style='display: inline' id='ProjectFreekat'>Project Freekat</h3><br />
<br />
<span>My idea involved purchasing the Meerkat mini PC from System76 and installing FreeBSD. Like the sound-server idea (see previous idea), it would run Linux Docker through Bhyve. I would self-host a bunch of applications on it:</span><br />
<br />
<ul>
<li>Wallabag</li>
<li>Ankidroid</li>
<li>Miniflux & Postgres</li>
<li>Audiobookshelf</li>
<li>...</li>
</ul><br />
<span>All of this would be within my LAN, but the services would also be accessible from the internet through either Wireguard or SSH reverse tunnels to one of my OpenBSD VMs, for example:</span><br />
<br />
<ul>
<li><span class='inlinecode'>wallabag.awesome.buetow.org</span></li>
<li><span class='inlinecode'>ankidroid.awesome.buetow.org</span></li>
<li><span class='inlinecode'>miniflux.awesome.buetow.org</span></li>
<li><span class='inlinecode'>audiobookshelf.awesome.buetow.org</span></li>
<li>...</li>
</ul><br />
<span>I am abandoning this project for now, as I am currently hosting my apps on AWS ECS Fargate under <span class='inlinecode'>*.cool.buetow.org</span>, which is "good enough" for the time being and also offers the benefit of learning to use AWS and Terraform, knowledge that can be applied at work.</span><br />
<br />
<a class='textlink' href='./2024-02-04-from-babylon5.buetow.org-to-.cloud.html'>My personal AWS setup</a><br />
<br />
<h2 style='display: inline' id='ProgrammingprojectsIdonthavetimefor'>Programming projects I don't have time for</h2><br />
<br />
<h3 style='display: inline' id='CLIHIVE'>CLI-HIVE</h3><br />
<br />
<span>This was a pet project idea that my brother and I had. The concept was to collect all shell history of all servers at work in a central place, apply ML/AI, and return suggestions for commands to type or allow a fuzzy search on all the commands in the history. The recommendations for the commands on a server could be context-based (e.g., past occurrences on the same server type). </span><br />
<br />
<span>You could decide whether to share your command history with others so they would receive better suggestions depending on which server they are on, or you could keep all the history private and secure. The plan was to add hooks into zsh and bash shells so that all commands typed would be pushed to the central location for data mining.</span><br />
<br />
<h3 style='display: inline' id='EnhancedKISShomephotoalbums'>Enhanced KISS home photo albums</h3><br />
<br />
<span>I don't use third-party cloud providers such as Google Photos to store/archive my photos. Instead, they are all on a ZFS volume on my home NAS, with regular offsite backups taken. Thus, my project would involve implementing the features I miss most or finding a solution simple enough to host on my LAN:</span><br />
<br />
<ul>
<li>A feature I miss presents me with a random day from the past and some photos from that day. This project would randomly select a day and generate a photo album for me to view and reminisce about memories.</li>
<li>Another feature I miss is the ability to automatically deduplicate all the photos, as I am sure there are tons of duplicates on my NAS.</li>
<li>Auto-enhancing the photos (perhaps using ImageMagick?)</li>
<li>I already have a simple <span class='inlinecode'>photoalbum.sh</span> script that generates an album based on an input directory. However, it would be great also to have a timeline feature to enable browsing through different dates.</li>
</ul><br />
<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br />
<br />
<h3 style='display: inline' id='KISSfilesyncserverwithendtoendencryption'>KISS file sync server with end-to-end encryption</h3><br />
<br />
<span>I aimed to have a simple server to which I could sync notes and other documents, ensuring that the data is fully end-to-end encrypted. This way, only the clients could decrypt the data, while an encrypted copy of all the data would be stored on the server side. There are a few solutions (e.g., NextCloud), but they are bloated or complex to set up. </span><br />
<br />
<span>I currently use Syncthing for encrypted file sync across all my devices; however, the data is not end-to-end encrypted. It's a good-enough setup, though, as my Syncthing server is in my home LAN on an encrypted file system.</span><br />
<br />
<a class='textlink' href='https://syncthing.net'>https://syncthing.net</a><br />
<br />
<span>I also had the idea of using this as a pet project for work and naming it <span class='inlinecode'>Cryptolake</span>, utilizing post-quantum-safe encryption algorithms and a distributed data store.</span><br />
<br />
<h3 style='display: inline' id='Alanguagethatcompilestobash'>A language that compiles to <span class='inlinecode'>bash</span></h3><br />
<br />
<span>I had an idea to implement a higher-level language with strong typing that could be compiled into native Bash code. This would make all resulting Bash scripts more robust and secure by default. The project would involve developing a parser, lexer, and a Bash code generator. I planned to implement this in Go.</span><br />
<br />
<span>I had previously implemented a tiny scripting language called Fype (For Your Program Execution), which could have served as inspiration.</span><br />
<br />
<a class='textlink' href='./2010-05-09-the-fype-programming-language.html'>The Fype Programming Language</a><br />
<br />
<h3 style='display: inline' id='Alanguagethatcompilestosed'>A language that compiles to <span class='inlinecode'>sed</span></h3><br />
<br />
<span>This is similar to the previous idea, but the difference is that the language would compile into a sed script. Sed has many features, but the brief syntax makes scripts challenging to read. The higher-level language would mimic sed but in a form that is easier for humans to read.</span><br />
<br />
<h3 style='display: inline' id='RenovateVSSim'>Renovate VS-Sim</h3><br />
<br />
<span>VS-Sim is an open-source simulator programmed in Java for distributed systems. VS-Sim stands for "Verteilte Systeme Simulator," the German translation for "Distributed Systems Simulator." The VS-Sim project was my diploma thesis at Aachen University of Applied Sciences.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/vs-sim'>https://codeberg.org/snonux/vs-sim</a><br />
<br />
<span>The ideas I had was:</span><br />
<br />
<ul>
<li>Translate the project into English.</li>
<li>Modernise the Java codebase to be compatible with the latest JDK.</li>
<li>Make it compile to native binaries using GraalVM.</li>
<li>Distribute the project using AppImages.</li>
</ul><br />
<span>I have put this project on hold for now, as I want to do more things in Go and fewer in Java in my personal time.</span><br />
<br />
<h3 style='display: inline' id='KISSticketingsystem'>KISS ticketing system</h3><br />
<br />
<span>My idea was to program a KISS (Keep It Simple, Stupid) ticketing system for my personal use. However, I am abandoning this project because I now use the excellent Taskwarrior software. You can learn more about it at:</span><br />
<br />
<a class='textlink' href='https://taskwarrior.org/'>https://taskwarrior.org/</a><br />
<br />
<h3 style='display: inline' id='AdomainspecificlanguageDSLforwork'>A domain-specific language (DSL) for work</h3><br />
<br />
<span>At work, an internal service allocates storage space for our customers on our storage clusters. It automates many tasks, but many tweaks are accessible through APIs. I had the idea to implement a Ruby-based DSL that would make using all those APIs for ad-hoc changes effortless, e.g.:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">Cluster </font><font color="#F3E651">:</font><font color="#ff0000">UK</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#F3E651">:</font><font color="#ff0000">uk01 </font><b><font color="#ffffff">do</font></b>
<font color="#ff0000"> Customer</font><font color="#F3E651">.</font><font color="#ff0000">C1A1</font><font color="#F3E651">.</font><font color="#ff0000">segments</font><font color="#F3E651">.</font><font color="#ff0000">volumes</font><font color="#F3E651">.</font><font color="#ff0000">each </font><b><font color="#ffffff">do</font></b><font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000">volume</font><font color="#F3E651">|</font>
<font color="#ff0000"> puts volume</font><font color="#F3E651">.</font><font color="#ff0000">usage_stats</font>
<font color="#ff0000"> volume</font><font color="#F3E651">.</font><font color="#ff0000">move_off! </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> volume</font><font color="#F3E651">.</font><font color="#ff0000">over_subscribed?</font>
<font color="#ff0000"> </font><b><font color="#ffffff">end</font></b>
<b><font color="#ffffff">end</font></b>
</pre>
<br />
<span>I am abandoning this project because my workplace has stopped the annual pet project competition, and I have other more important projects to work on at the moment.</span><br />
<br />
<a class='textlink' href='./2022-04-10-creative-universe.html'>Creative universe (Work pet project contests)</a><br />
<br />
<h2 style='display: inline' id='SelfhostingprojectsIdonthavetimefor'>Self-hosting projects I don't have time for</h2><br />
<br />
<h3 style='display: inline' id='MyownMatrixserver'>My own Matrix server</h3><br />
<br />
<span>I value privacy. It would be great to run my own Matrix server for communication within my family. I have yet to have time to look into this more closely.</span><br />
<br />
<a class='textlink' href='https://matrix.org'>https://matrix.org</a><br />
<br />
<h3 style='display: inline' id='Ampachemusicserver'>Ampache music server</h3><br />
<br />
<span>Ampache is an open-source music streaming server that allows you to host and manage your music collection online, accessible via a web interface. Setting it up involves configuring a web server, installing Ampache, and organising your music files, which can be time-consuming. </span><br />
<br />
<h3 style='display: inline' id='LibrumeBookreader'>Librum eBook reader</h3><br />
<br />
<span>Librum is a self-hostable e-book reader that allows users to manage and read their e-book collection from a web interface. Designed to be a self-contained platform where users can upload, organise, and access their e-books, Librum emphasises privacy and control over one's digital library.</span><br />
<br />
<a class='textlink' href='https://github.com/Librum-Reader/Librum'>https://github.com/Librum-Reader/Librum</a><br />
<br />
<span>I am using my Kobo devices or my laptop to read these kinds of things for now.</span><br />
<br />
<h3 style='display: inline' id='MemosNotetakingservice'>Memos - Note-taking service</h3><br />
<br />
<span>Memos is a note-taking service that simplifies and streamlines information capture and organisation. It focuses on providing users with a minimalistic and intuitive interface, aiming to enhance productivity without the clutter commonly associated with more complex note-taking apps.</span><br />
<br />
<a class='textlink' href='https://www.usememos.com'>https://www.usememos.com</a><br />
<br />
<span>I am abandoning this idea for now, as I am currently using plain Markdown files for notes and syncing them with Syncthing across my devices.</span><br />
<br />
<h3 style='display: inline' id='Bepastyserver'>Bepasty server</h3><br />
<br />
<span>Bepasty is like a Pastebin for all kinds of files (text, image, audio, video, documents, binary, etc.). It seems very neat, but I only share a little nowadays. When I do, I upload files via SCP to one of my OpenBSD VMs and serve them via vanilla httpd there, keeping it KISS.</span><br />
<br />
<a class='textlink' href='https://github.com/bepasty/bepasty-server'>https://github.com/bepasty/bepasty-server</a><br />
<br />
<h2 style='display: inline' id='BooksIdonthavetimetoread'>Books I don't have time to read</h2><br />
<br />
<h3 style='display: inline' id='FluentPython'>Fluent Python</h3><br />
<br />
<span>I consider myself an advanced programmer in Ruby, Bash, and Perl. However, Python seems to be ubiquitous nowadays, and most of my colleagues prefer Python over any other languages. Thus, it makes sense for me to also learn and use Python. After conducting some research, "Fluent Python" appears to be the best book for this purpose.</span><br />
<br />
<span>I don't have time to read this book at the moment, as I am focusing more on Go (Golang) and I know just enough Python to get by (e.g., for code reviews). Additionally, there are still enough colleagues around who can review my Ruby or Bash code.</span><br />
<br />
<h3 style='display: inline' id='ProgrammingRuby'>Programming Ruby</h3><br />
<br />
<span>I've read a couple of Ruby books already, but "Programming Ruby," which covers up to Ruby 3.2, was just recently released. I would like to read this to deepen my Ruby knowledge further and to revisit some concepts that I may have forgotten.</span><br />
<br />
<span>As stated in this blog post, I am currently more eager to focus on Go, so I've put the Ruby book on hold. Additionally, there wouldn't be enough colleagues who could "understand" my advanced Ruby skills anyway, as most of them are either Java developers or SREs who don't code a lot.</span><br />
<br />
<h3 style='display: inline' id='PeterFHamiltonsciencefictionbooks'>Peter F. Hamilton science fiction books</h3><br />
<br />
<span>I am a big fan of science fiction, but my reading list is currently too long anyway. So, I've put the Hamilton books on the back burner for now. You can see all the novels I've read here:</span><br />
<br />
<a class='textlink' href='https://paul.buetow.org/novels.html'>https://paul.buetow.org/novels.html</a><br />
<a class='textlink' href='gemini://paul.buetow.org/novels.gmi'>gemini://paul.buetow.org/novels.gmi</a><br />
<br />
<br />
<h2 style='display: inline' id='NewwebsitesIdonthavetimefor'>New websites I don't have time for</h2><br />
<br />
<h3 style='display: inline' id='CreateaWhyRakuRoxsite'>Create a "Why Raku Rox" site</h3><br />
<br />
<span>The website "Why Raku Rox" would showcase the unique features and benefits of the Raku programming language and highlight why it is an exceptional choice for developers. Raku, originally known as Perl 6, is a dynamic, expressive language designed for flexible and powerful software development.</span><br />
<br />
<span>This would be similar to the "Why OpenBSD rocks" site:</span><br />
<br />
<a class='textlink' href='https://why-openbsd.rocks'>https://why-openbsd.rocks</a><br />
<a class='textlink' href='https://raku.org'>https://raku.org</a><br />
<br />
<span>I am not working on this for now, as I currently don’t even have time to program in Raku.</span><br />
<br />
<h2 style='display: inline' id='ResearchprojectsIdonthavetimefor'>Research projects I don't have time for</h2><br />
<br />
<h3 style='display: inline' id='Projectsecure'>Project secure</h3><br />
<br />
<span>For work: Implement a PoC that dumps Java heaps to extract secrets from memory. Based on the findings, write a Java program that encrypts secrets in the kernel using the <span class='inlinecode'>memfd_secret()</span> syscall to make it even more secure.</span><br />
<br />
<a class='textlink' href='https://lwn.net/Articles/865256/'>https://lwn.net/Articles/865256/</a><br />
<br />
<span>Due to other priorities, I am putting this on hold for now. The software we have built is pretty damn secure already!</span><br />
<br />
<h3 style='display: inline' id='CPUutilisationisallwrong'>CPU utilisation is all wrong</h3><br />
<br />
<span>This research project, based on Brendan Gregg's blog post, could potentially significantly impact my work.</span><br />
<br />
<a class='textlink' href='https://brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html'>https://brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html</a><br />
<br />
<span>The research project would involve setting up dashboards that display actual CPU usage and the cycles versus waiting time for memory access.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Related and maybe interesting:</span><br />
<br />
<a class='textlink' href='./2022-06-15-sweating-the-small-stuff.html'>Sweating the small stuff - Tiny projects of mine</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>'Slow Productivity' book notes</title>
<link href="gemini://foo.zone/gemfeed/2024-05-01-slow-productivity-book-notes.gmi" />
<id>gemini://foo.zone/gemfeed/2024-05-01-slow-productivity-book-notes.gmi</id>
<updated>2024-04-27T14:18:51+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>These are my personal takeaways after reading 'Slow Productivity - The lost Art of Accomplishment Without Burnout' by Cal Newport.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='SlowProductivitybooknotes'>"Slow Productivity" book notes</h1><br />
<br />
<span class='quote'>Published at 2024-04-27T14:18:51+03:00</span><br />
<br />
<span>These are my personal takeaways after reading "Slow Productivity - The lost Art of Accomplishment Without Burnout" by Cal Newport.</span><br />
<br />
<span>The case studies in this book were a bit long, but they appeared to be well-researched. I will only highlight the interesting, actionable items in the book notes.</span><br />
<br />
<span>These notes are mainly for my own use, but you may find them helpful.</span><br />
<br />
<pre>
,.......... ..........,
,..,' '.' ',..,
,' ,' : ', ',
,' ,' : ', ',
,' ,' : ', ',
,' ,'............., : ,.............', ',
,' '............ '.' ............' ',
'''''''''''''''''';''';''''''''''''''''''
'''
</pre>
<br />
<span>"Slow productivity" does not mean being less productive. Cal Newport wants to point out that you can be much more productive with "slow productivity" than you would be without it. It is a different way of working than most of us are used to in the modern workplace, which is hyper-connected and always online.</span><br />
<br />
<h2 style='display: inline' id='PseudoproductivityandShallowwork'>Pseudo-productivity and Shallow work</h2><br />
<br />
<span>People use visible activity instead of real productivity because it's easier to measure. This is called pseudo-productivity.</span><br />
<span>Pseudo-productivity is used as a proxy for real productivity. If you don't look busy, you are dismissed as lazy or lacking a work ethic.</span><br />
<br />
<span>There is a tendency to perform shallow work because people will otherwise dismiss you as lazy. A lot of shallow work can cause burnout, as multiple things are often being worked on in parallel. The more you have on your plate, the more stressed you will be.</span><br />
<br />
<span>Shallow work usually doesn't help you to accomplish big things. Always have the big picture in mind. Shallow work can't be entirely eliminated, but it can be managed—for example, plan dedicated time slots for certain types of shallow work.</span><br />
<br />
<h2 style='display: inline' id='Accomplishmentswithoutburnout'>Accomplishments without burnout</h2><br />
<br />
<span>The overall perception is that if you want to accomplish something, you must put yourself on the verge of burnout. Cal Newport writes about "The lost Art of Accomplishments without Burnouts", where you can accomplish big things without all the stress usually involved.</span><br />
<br />
<span>There are three principles for the maintenance of a sustainable work life:</span><br />
<br />
<ul>
<li>Do fewer things</li>
<li>Work at a natural pace</li>
<li>Obsess over quality</li>
</ul><br />
<h2 style='display: inline' id='Dofewerthings'>Do fewer things</h2><br />
<br />
<span>There will always be more work. The faster you finish it, the quicker you will have something new on your plate.</span><br />
<br />
<span>Reduce the overhead tax. The overhead tax is all the administrative work to be done. With every additional project, there will also be more administrative stuff to be done on your work plate. So, doing fewer things leads to more and better output and better quality for the projects you are working on.</span><br />
<br />
<span>Limit the things on your plate. Limit your missions (personal goals, professional goals). Reduce your main objectives in life. More than five missions are usually not sustainable very easily, so you have to really prioritise what is important to you and your professional life.</span><br />
<br />
<span>A mission is an overall objective/goal that can have multiple projects. Limit the projects as well. Some projects need clear endings (e.g., work in support of a never-ending flow of incoming requests). In this case, set limits (e.g., time box your support hours). You can also plan "office hours" for collaborative work with colleagues to avoid ad hoc distractions.</span><br />
<br />
<span>The key point is that after making these commitments, you really deliver on them. This builds trust, and people will leave you alone and not ask for progress all the time.</span><br />
<br />
<span>Doing fever things is essential for modern knowledge workers. Breathing space in your work also makes you more creative and happier overall.</span><br />
<br />
<span>Pushing workers more work can make them less productive, so the better approach is the pull model, where workers pull in new work when the previous task is finished.</span><br />
<br />
<span>If you can quantify how busy you are or how many other projects you already work on, then it is easier to say no to new things. For example, show what you are doing, what's in the roadmap, etc. Transparency is the key here. </span><br />
<br />
<span>You can have your own simulated pull system if the company doesn't agree to a global one: </span><br />
<br />
<ul>
<li>State which additional information you would need.</li>
<li>Create a rough estimate of when you will be able to work on it</li>
<li>Estimate how long the project would take. Double that estimate, as humans are very bad estimators.</li>
<li>Respond to the requester and state that you will let him know when the estimates change.</li>
</ul><br />
<span>Sometimes, a little friction is all that is needed to combat incoming work, e.g., when your manager starts seeing the reality of your work plate, and you also request additional information for the task. If you already have too much on your plate, then decline the new project or make room for it in your calendar. If you present a large task list, others will struggle to assign more to you.</span><br />
<br />
<span>Limit your daily goals. A good measure is to focus on one goal per day. You can time block time for deep work on your daily goal. During that time, you won't be easily available to others.</span><br />
<br />
<span>The battle against distractions must be fought to be the master of your time. Nobody will fight this war for you. You have to do it for yourself. (Also, have a look at Cal Newport's "time block planning" method).</span><br />
<br />
<span>Put tasks on autopilot (regular recurring tasks).</span><br />
<br />
<h2 style='display: inline' id='Workatanaturalpace'>Work at a natural pace</h2><br />
<br />
<span>We suffer from overambitious timelines, task lists, and business. Focus on what matters. Don't rush your most important work to achieve better results.</span><br />
<br />
<span>Don't rush. If you rush or are under pressure, you will be less effective and eventually burn out. Our brains work better then not rushy. The stress heuristic usually indicates too much work, and it is generally too late to reduce workload. That's why we all typically have dangerously too much to do.</span><br />
<br />
<span>Have the courage to take longer to do things that are important. For example, plan on a yearly and larger scale, like 2 to 5 years.</span><br />
<br />
<span>Find a reasonable time for a project and then double the project timeline against overconfident optimism. Humans are not great at estimating. They gravitate towards best-case estimates. If you have planned more than enough time for your project, then you will fall into a natural work pace. Otherwise, you will struggle with rushing and stress.</span><br />
<br />
<span>Some days will still be intense and stressful, but those are exceptional cases. After those exceptions (e.g., finalizing that thing, etc.), calmer periods will follow again.</span><br />
<br />
<span>Pace yourself over modest results over time. Simplify and reduce the daily task lists. Meetings: Certain hours are protected for work. For each meeting, add a protected block to your calendar, so you attend meetings only half a day max.</span><br />
<br />
<span>Schedule slow seasons (e.g., when on vacation). Disconnect in the slow season. Doing nothing will not satisfy your mind, though. You could read a book on your subject matter to counteract that.</span><br />
<br />
<h2 style='display: inline' id='Obsessoverquality'>Obsess over quality </h2><br />
<br />
<span>Obsess over quality even if you lose short-term opportunities by rejecting other projects. Quality demands you slow down. The two previous two principles (do fewer things and work at a natural pace) are mandatory for this principle to work:</span><br />
<br />
<ul>
<li>Focus on the core activities of your work for your obsession - you will only have the time to obsess over some things.</li>
<li>Deliver solid work with good quality.</li>
<li>Sharpen the focus to do the best work possible.</li>
</ul><br />
<span>Go pro to save time, and don't squeeze everything out that you can from freemium services. Professional software services eliminate administrative work:</span><br />
<br />
<ul>
<li>Pay people who know what they are doing and focus on your stuff. </li>
<li>For example, don't repair that car if you know the mechanic can do that much better than you. </li>
<li>Or don't use the free version of the music streaming service if it interrupts you with commercials, hindering your ability to concentrate on your work.</li>
<li>Hire an accountant for your yearly tax returns. He knows much more about that stuff than you do. And in the end, he will even be cheaper as he knows all the tax laws.</li>
<li>...</li>
</ul><br />
<span>Adjust your workplace to what you want to accomplish. You could have dedicated places in your home for different things, e.g., a place where you read and think (armchair) and a place where you collaborate (your desk or whiteboard). Surround yourself with things that inspire you (e.g., your favourite books on your shelf next to you, etc.).</span><br />
<br />
<span>There is the concept of quiet quitting. It doesn't mean quitting your job, but it means that you don't go beyond and above the expectations people have of you. Quiet quitting became popular with modern work, which is often meaningless and full of shallow tasks. If you obsess over quality, you enjoy your craft and want to go beyond and above.</span><br />
<br />
<span>Implement rituals and routines which shift you towards your goals:</span><br />
<br />
<ul>
<li>For example, if you want to be a good Software Engineer, you also have to put in the work regularly. For instance, progress a bit every day in your project at hand, even if it is only one hour daily. Also, a little quality daily work will be more satisfying over time than many shallow tasks.</li>
<li>Do you want to be lean and/or healthy? Schedule your daily walks and workouts. They will become habits over time.</li>
<li>There's the compounding effect where every small effort made every day will yield significant results in the long run</li>
</ul><br />
<span>Deciding what not to do is as important as deciding what to do. </span><br />
<br />
<span>It appears to be money thrown out of the window, but you get a $50 expensive paper notebook (and also a good pen). Unconsciously, it will make you take notes more seriously. You will think about what to put into the notebooks more profoundly and have thought through the ideas more intensively. If you used very cheap notebooks, you would scribble a lot of rubbish and wouldn't even recognise your handwriting after a while anymore. So choosing a high-quality notebook will help you to take higher-quality notes, too.</span><br />
<br />
<span>Slow productivity is actionable and can be applied immediately.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other book notes of mine are:</span><br />
<br />
<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 "The Pragmatic Programmer" book notes</a><br />
<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 "Never split the difference" book notes</a><br />
<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 "The Obstacle is the Way" book notes</a><br />
<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes</a><br />
<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 "Mind Management" book notes</a><br />
<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 "Slow Productivity" book notes (You are currently reading this)</a><br />
<br />
<a class='textlink' href='../resources.html'>More books and other resources I found useful.</a><br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>KISS high-availability with OpenBSD</title>
<link href="gemini://foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi" />
<id>gemini://foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi</id>
<updated>2024-03-30T22:12:56+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Art by Michael J. Penick (mod. by Paul B.)</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='KISShighavailabilitywithOpenBSD'>KISS high-availability with OpenBSD</h1><br />
<br />
<span class='quote'>Published at 2024-03-30T22:12:56+02:00</span><br />
<br />
<pre>
Art by Michael J. Penick (mod. by Paul B.)
ACME-sky
__________
/ nsd tower\ (
/____________\ (\) awk-ward
|:_:_:_:_:_| )) plant
|_:_,--.:_:| dig-bubble (\// )
|:_:|__|_:_| relayd-castle _ ) )) ((
_ |_ _ :_:| _ _ _ (_) (((( /)\`
| |_| |_| | _| | |_| |_| | o \\)) (( (
\_:_:_:_:/|_|_|_|\:_:_:_:_/ . (( ))))
|_,-._:_:_:_:_:_:_:_.-,_| )) ((//
|:|_|:_:_:,---,:_:_:|_|:| ,-. )/
|_:_:_:_,'puffy `,_:_:_:_| _ o ,;'))((
|:_:_:_/ _ | _ \_:_:_:| (_O (( ))
_____|_:_:_| (o)-(o) |_:_:_|--'`-. ,--. ksh under-water (((\'/
', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\ /,---.~ goat \`))
. ` |_:_:_| \`-'/ |_:_:_|. ` . ` /()\.__( ) .,-----'`-\(( sed-root
', ;|:_:_:| `-' |:_:_:| ', ; ', ; `--'| \ ', ; ', ; ',')).,--
. ` MJP ` . ` . ` . ` . httpd-soil ` . . ` . ` . ` . ` . `
', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ;
</pre>
<br />
<pre>
Table of contents
=================
KISS high-availability with OpenBSD
My auto-failover requirements
My HA solution
Only OpenBSD base installation required
Fairly cheap and geo-redundant
Failover time and split-brain
Failover support for multiple protocols
Let's encrypt TLS certificates
Monitoring
Rex automation
More HA
</pre>
<br />
<span>I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like (in unsorted and slightly unrelated order) BGP, LVS/IPVS, ldirectord, Pacemaker, STONITH, scripted VIP failover via ARP, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work. </span><br />
<br />
<span>But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space - something as KISS (keep it simple and stupid) as possible.</span><br />
<br />
<span>It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.</span><br />
<br />
<span class='quote'>PS: ASCII-art reflects an OpenBSD under-water world with all the tools available in the base system.</span><br />
<br />
<h2 style='display: inline' id='Myautofailoverrequirements'>My auto-failover requirements</h2><br />
<br />
<ul>
<li>Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible. </li>
<li>Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month already!).</li>
<li>It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.</li>
<li>It should be geo-redundant. </li>
<li>It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.</li>
<li>Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.</li>
<li>Let's Encrypt TLS certificates should always work (before and after a failover).</li>
<li>Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.</li>
<li>Don't configure everything manually. The configuration should be automated and reproducible.</li>
</ul><br />
<h2 style='display: inline' id='MyHAsolution'>My HA solution</h2><br />
<br />
<h3 style='display: inline' id='OnlyOpenBSDbaseinstallationrequired'>Only OpenBSD base installation required</h3><br />
<br />
<span>My HA solution for Web and Gemini is based on DNS (OpenBSD's <span class='inlinecode'>nsd</span>) and a simple shell script (OpenBSD's <span class='inlinecode'>ksh</span> and some little <span class='inlinecode'>sed</span> and <span class='inlinecode'>awk</span> and <span class='inlinecode'>grep</span>). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.</span><br />
<br />
<a class='textlink' href='https://man.OpenBSD.org/nsd.8'>https://man.OpenBSD.org/nsd.8</a><br />
<a class='textlink' href='https://man.OpenBSD.org/ksh'>https://man.OpenBSD.org/ksh</a><br />
<a class='textlink' href='https://man.OpenBSD.org/awk'>https://man.OpenBSD.org/awk</a><br />
<a class='textlink' href='https://man.OpenBSD.org/sed'>https://man.OpenBSD.org/sed</a><br />
<a class='textlink' href='https://man.OpenBSD.org/dig'>https://man.OpenBSD.org/dig</a><br />
<a class='textlink' href='https://man.OpenBSD.org/ftp'>https://man.OpenBSD.org/ftp</a><br />
<a class='textlink' href='https://man.OpenBSD.org/cron'>https://man.OpenBSD.org/cron</a><br />
<br />
<span>I also used the <span class='inlinecode'>dig</span> (for DNS checks) and <span class='inlinecode'>ftp</span> (for HTTP/HTTPS checks) programs. </span><br />
<br />
<span>The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The <span class='inlinecode'>ksh</span> script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM: </span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/bin/ksh</font></i>
<font color="#ff0000">ZONES_DIR</font><font color="#F3E651">=</font><font color="#ff0000">/var/nsd/zones/master</font><font color="#F3E651">/</font>
<font color="#ff0000">DEFAULT_MASTER</font><font color="#F3E651">=</font><font color="#ff0000">fishfinger</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#ff0000">DEFAULT_STANDBY</font><font color="#F3E651">=</font><font color="#ff0000">blowfish</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#7bc710">determine_master_and_standby ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">master</font><font color="#F3E651">=</font><font color="#ff0000">$DEFAULT_MASTER</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">standby</font><font color="#F3E651">=</font><font color="#ff0000">$DEFAULT_STANDBY</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -i </font><font color="#ff0000">health_ok</font><font color="#F3E651">=</font><font color="#bb00ff">1</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> ftp -</font><font color="#bb00ff">4</font><font color="#ff0000"> -o - https</font><font color="#F3E651">://</font><font color="#ff0000">$master</font><font color="#ff0000">/index</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">|</font><font color="#ff0000"> grep -q </font><font color="#bb00ff">"Welcome to $master"</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"https://$master/index.txt IPv4 health check failed"</font>
<font color="#ff0000"> </font><font color="#ff0000">health_ok</font><font color="#F3E651">=</font><font color="#bb00ff">0</font>
<font color="#ff0000"> </font><b><font color="#ffffff">elif</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> ftp -</font><font color="#bb00ff">6</font><font color="#ff0000"> -o - https</font><font color="#F3E651">://</font><font color="#ff0000">$master</font><font color="#ff0000">/index</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">|</font><font color="#ff0000"> grep -q </font><font color="#bb00ff">"Welcome to $master"</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"https://$master/index.txt IPv6 health check failed"</font>
<font color="#ff0000"> </font><font color="#ff0000">health_ok</font><font color="#F3E651">=</font><font color="#bb00ff">0</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#ff0000">$health_ok</font><font color="#ff0000"> -eq </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">tmp</font><font color="#F3E651">=</font><font color="#ff0000">$master</font>
<font color="#ff0000"> </font><font color="#ff0000">master</font><font color="#F3E651">=</font><font color="#ff0000">$standby</font>
<font color="#ff0000"> </font><font color="#ff0000">standby</font><font color="#F3E651">=</font><font color="#ff0000">$tmp</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000">}</font>
</pre>
<br />
<span>The failover scripts looks for the <span class='inlinecode'> ; Enable failover</span> string in the DNS zone files and swaps the <span class='inlinecode'>A</span> and <span class='inlinecode'>AAAA</span> records of the DNS entries accordingly:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">fishfinger$ grep failover /var/nsd/zones/master/foo</font><font color="#F3E651">.</font><font color="#ff0000">zone</font><font color="#F3E651">.</font><font color="#ff0000">zone</font>
<font color="#ff0000"> </font><font color="#bb00ff">300</font><font color="#ff0000"> IN A </font><font color="#bb00ff">46.23</font><font color="#F3E651">.</font><font color="#bb00ff">94.99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000"> </font><font color="#bb00ff">300</font><font color="#ff0000"> IN AAAA 2a03</font><font color="#F3E651">:</font><font color="#bb00ff">6000</font><font color="#F3E651">:</font><font color="#ff0000">6f67</font><font color="#F3E651">:</font><font color="#bb00ff">624</font><font color="#F3E651">::</font><font color="#bb00ff">99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">www </font><font color="#bb00ff">300</font><font color="#ff0000"> IN A </font><font color="#bb00ff">46.23</font><font color="#F3E651">.</font><font color="#bb00ff">94.99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">www </font><font color="#bb00ff">300</font><font color="#ff0000"> IN AAAA 2a03</font><font color="#F3E651">:</font><font color="#bb00ff">6000</font><font color="#F3E651">:</font><font color="#ff0000">6f67</font><font color="#F3E651">:</font><font color="#bb00ff">624</font><font color="#F3E651">::</font><font color="#bb00ff">99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">standby </font><font color="#bb00ff">300</font><font color="#ff0000"> IN A </font><font color="#bb00ff">23.88</font><font color="#F3E651">.</font><font color="#bb00ff">35.144</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">standby </font><font color="#bb00ff">300</font><font color="#ff0000"> IN AAAA 2a01</font><font color="#F3E651">:</font><font color="#ff0000">4f8</font><font color="#F3E651">:</font><font color="#ff0000">c17</font><font color="#F3E651">:</font><font color="#ff0000">20f1</font><font color="#F3E651">::</font><font color="#bb00ff">42</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#7bc710">transform ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> sed -E </font><font color="#bb00ff">'</font>
<font color="#bb00ff"> /IN A .*; Enable failover/ {</font>
<font color="#bb00ff"> /^standby/! {</font>
<font color="#bb00ff"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN A '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/master_a</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> /^standby/ {</font>
<font color="#bb00ff"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN A '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/standby_a</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> /IN AAAA .*; Enable failover/ {</font>
<font color="#bb00ff"> /^standby/! {</font>
<font color="#bb00ff"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN AAAA '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/master_aaaa</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> /^standby/ {</font>
<font color="#bb00ff"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN AAAA '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/standby_aaaa</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> / ; serial/ {</font>
<font color="#bb00ff"> s/^( +) ([0-9]+) .*; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> '</font><font color="#ff0000">$(</font><font color="#ff0000">date </font><font color="#F3E651">+%</font><font color="#ff0000">s</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> '</font>
<font color="#ff0000">}</font>
</pre>
<br />
<span>After the failover, the script reloads <span class='inlinecode'>nsd</span> and performs a sanity check to see if DNS still works. If not, a rollback will be performed:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#! Race condition !#</font></i>
<font color="#ff0000"> </font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> -f </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> mv </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak </font><font color="#ff0000">$zone_file</font>
<b><font color="#ffffff">fi</font></b>
<font color="#ff0000">cat </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000"> transform </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font>
<font color="#ff0000">grep -v </font><font color="#bb00ff">' ; serial'</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp</font>
<font color="#ff0000">grep -v </font><font color="#bb00ff">' ; serial'</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">old</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp</font>
<font color="#ff0000">echo </font><font color="#bb00ff">"Has zone $zone_file changed?"</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> diff -u </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">old</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"The zone $zone_file hasn't changed"</font>
<font color="#ff0000"> rm </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.*.</font><font color="#ff0000">tmp</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#bb00ff">0</font>
<b><font color="#ffffff">fi</font></b>
<font color="#ff0000">cp </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak</font>
<font color="#ff0000">mv </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font><font color="#ff0000">$zone_file</font>
<font color="#ff0000">rm </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.*.</font><font color="#ff0000">tmp</font>
<font color="#ff0000">echo </font><font color="#bb00ff">"Reloading nsd"</font>
<font color="#ff0000">nsd-control reload</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> zone_is_ok </font><font color="#ff0000">$zone</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"Rolling back $zone_file changes"</font>
<font color="#ff0000"> cp </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">invalid</font>
<font color="#ff0000"> mv </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak </font><font color="#ff0000">$zone_file</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"Reloading nsd"</font>
<font color="#ff0000"> nsd-control reload</font>
<font color="#ff0000"> zone_is_ok </font><font color="#ff0000">$zone</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#bb00ff">3</font>
<b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">for</font></b><font color="#ff0000"> cleanup </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> invalid bak</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> -f </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">$cleanup</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> rm </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">$cleanup</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">done</font></b>
<font color="#ff0000">echo </font><font color="#bb00ff">"Failover of zone $zone to $MASTER completed"</font>
<b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#bb00ff">1</font>
</pre>
<br />
<span>A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.</span><br />
<br />
<span>The authorative nameserver for my domains runs on both VMs, and both are configured to be a "master" DNS server so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.</span><br />
<br />
<span>Check out the whole script here:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh'>dns-failover.ksh</a><br />
<br />
<h3 style='display: inline' id='Fairlycheapandgeoredundant'>Fairly cheap and geo-redundant</h3><br />
<br />
<span>I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).</span><br />
<br />
<a class='textlink' href='https://OpenBSD.Amsterdam'>https://OpenBSD.Amsterdam</a><br />
<a class='textlink' href='https://www.Hetzner.cloud'>https://www.Hetzner.cloud</a><br />
<br />
<span>I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.</span><br />
<br />
<span>A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.</span><br />
<br />
<h3 style='display: inline' id='Failovertimeandsplitbrain'>Failover time and split-brain</h3><br />
<br />
<span>A DNS failover doesn't happen immediately. I've configured a DNS TTL of <span class='inlinecode'>300</span> seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).</span><br />
<br />
<span>A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.</span><br />
<br />
<h3 style='display: inline' id='Failoversupportformultipleprotocols'>Failover support for multiple protocols</h3><br />
<br />
<span>With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's <span class='inlinecode'>httpd</span> (OpenBSD's HTTP server) and <span class='inlinecode'>relayd</span> (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entries, which VM receives the requests.</span><br />
<br />
<a class='textlink' href='https://man.OpenBSD.org/httpd.8'>https://man.OpenBSD.org/httpd.8</a><br />
<a class='textlink' href='https://man.OpenBSD.org/relayd.8'>https://man.OpenBSD.org/relayd.8</a><br />
<br />
<span>For example, the master is responsible for the <span class='inlinecode'>https://www.foo.zone</span> and <span class='inlinecode'>https://foo.zone</span> hosts, whereas the standby can be reached via <span class='inlinecode'>https://standby.foo.zone</span> (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. <span class='inlinecode'>irregular.ninja</span>, <span class='inlinecode'>paul.buetow.org</span> and so on. The same applies to my Gemini capsules for <span class='inlinecode'>gemini://foo.zone</span>, <span class='inlinecode'>gemini://standby.foo.zone</span>, <span class='inlinecode'>gemini://paul.buetow.org</span> and <span class='inlinecode'>gemini://standby.paul.buetow.org</span>.</span><br />
<br />
<span>On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!</span><br />
<br />
<h3 style='display: inline' id='LetsencryptTLScertificates'>Let's encrypt TLS certificates</h3><br />
<br />
<span>All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.</span><br />
<br />
<span>If the master always serves <span class='inlinecode'>foo.zone</span> and the standby always <span class='inlinecode'>standby.foo.zone</span>, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for <span class='inlinecode'>foo.zone</span> and the new standby wouldn't have a valid certificate for <span class='inlinecode'>standby.foo.zone</span> which would lead to TLS errors on the clients.</span><br />
<br />
<span>As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:</span><br />
<br />
<ul>
<li>In an odd week number, the first server is the default master</li>
<li>In an even week number, the second server is the default master.</li>
</ul><br />
<span>Which translates to:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab"># Weekly auto-failover for Let's Encrypt automation</font></i>
<b><font color="#ffffff">local</font></b><font color="#ff0000"> -i -r </font><font color="#ff0000">week_of_the_year</font><font color="#F3E651">=</font><font color="#ff0000">$(</font><font color="#ff0000">date </font><font color="#F3E651">+%</font><font color="#ff0000">U</font><font color="#F3E651">)</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#ff0000">$(</font><font color="#F3E651">(</font><font color="#ff0000"> week_of_the_year </font><font color="#F3E651">%</font><font color="#ff0000"> </font><font color="#bb00ff">2</font><font color="#ff0000"> </font><font color="#F3E651">))</font><font color="#ff0000"> -eq </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">tmp</font><font color="#F3E651">=</font><font color="#ff0000">$master</font>
<font color="#ff0000"> </font><font color="#ff0000">master</font><font color="#F3E651">=</font><font color="#ff0000">$standby</font>
<font color="#ff0000"> </font><font color="#ff0000">standby</font><font color="#F3E651">=</font><font color="#ff0000">$tmp</font>
<b><font color="#ffffff">fi</font></b>
</pre>
<br />
<span>This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.</span><br />
<br />
<span>The ACME automation is yet another daily CRON script <span class='inlinecode'>/usr/local/bin/acme.sh</span>. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt. </span><br />
<br />
<span>Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/acme.sh.tpl'><span class='inlinecode'>acme.sh.tpl</span> - Rex template for the <span class='inlinecode'>acme.sh</span> script of mine.</a><br />
<a class='textlink' href='https://man.OpenBSD.org/acme-client.1'>https://man.OpenBSD.org/acme-client.1</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br />
<br />
<h3 style='display: inline' id='Monitoring'>Monitoring</h3><br />
<br />
<span>CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed. </span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/gogios'>https://codeberg.org/snonux/gogios</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>KISS server monitoring with Gogios</a><br />
<br />
<span>Gogios, as I developed it by myself, isn't part of the OpenBSD base system. </span><br />
<br />
<h3 style='display: inline' id='Rexautomation'>Rex automation</h3><br />
<br />
<span>I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.</span><br />
<br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends'>codeberg.org/snonux/rexfiles/frontends</a><br />
<br />
<span>Rex isn't part of the OpenBSD base system, but I didn't need to install any external software on OpenBSD either as Rex is invoked from my Laptop!</span><br />
<br />
<h2 style='display: inline' id='MoreHA'>More HA</h2><br />
<br />
<span>Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD - also part of the OpenBSD base system) and the authoritative DNS servers (<span class='inlinecode'>nsd</span>) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host! </span><br />
<br />
<a class='textlink' href='https://www.OpenSMTPD.org/'>https://www.OpenSMTPD.org/</a><br />
<br />
<span>As a password manager, I use <span class='inlinecode'>geheim</span>, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the <span class='inlinecode'>sync</span> command there. </span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/geheim'>https://codeberg.org/snonux/geheim</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other *BSD and KISS related posts are:</span><br />
<br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br />
<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD (You are currently reading this)</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>A fine Fyne Android app for quickly logging ideas programmed in Go</title>
<link href="gemini://foo.zone/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi" />
<id>gemini://foo.zone/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi</id>
<updated>2024-03-03T00:07:21+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I am an ideas person. I find myself frequently somewhere on the streets with an idea in my head but no paper journal noting it down. </summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='AfineFyneAndroidappforquicklyloggingideasprogrammedinGo'>A fine Fyne Android app for quickly logging ideas programmed in Go</h1><br />
<br />
<span class='quote'>Published at 2024-03-03T00:07:21+02:00</span><br />
<br />
<span>I am an ideas person. I find myself frequently somewhere on the streets with an idea in my head but no paper journal noting it down. </span><br />
<br />
<span>I have tried many note apps for my Android (I use GrapheneOS) phone. Most of them either don't do what I want, are proprietary software, require Google Play services (I have the main profile on my phone de-googled) or are too bloated. I was never into mobile app development, as I'm not too fond of the complexity of the developer toolchains. I don't want to use Android Studio (as a NeoVim user), and I don't want to use Java or Kotlin. I want to use a language I know (and like) for mobile app development. Go would be one of those languages.</span><br />
<br />
<a href='2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang/logo-small.png'><img alt='Quick logger Logo' title='Quick logger Logo' src='2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang/logo-small.png' /></a><br />
<br />
<span>Enter Quick logger – a compact GUI Android (well, cross-platform due to Fyne) app I've crafted using Go and the nifty Fyne framework. With Fyne, the app can be compiled easily into an Android APK. As of this writing, this app's whole Go source code is only 75 lines short!! This little tool is designed for spontaneous moments, allowing me to quickly log my thoughts as plain text files on my Android phone. There are no fancy file formats. Just plain text!</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/quicklogger'>https://codeberg.org/snonux/quicklogger</a><br />
<a class='textlink' href='https://fyne.io'>https://fyne.io</a><br />
<a class='textlink' href='https://go.dev'>https://go.dev</a><br />
<br />
<span>There's no need to navigate complex menus or deal with sync issues. I jot down my Idea, and Quick logger saves it to a plain text file in a designated local folder on my phone. There is one text file per note (timestamp in the file name). Once logged, the file can't be edited anymore (it keeps it simple). If I want to correct or change a note, I simply write a new one. My notes are always small (usually one short sentence each), so there isn't the need for an edit functionality. I can edit them later on my actual computer if I want to.</span><br />
<br />
<span>With Syncthing, the note files are then synchronised to my home computer to my <span class='inlinecode'>~/Notes</span> directory. From there, a small glue Raku script adds them to my Taskwarrior DB so that I can process them later (e.g. take action on that one Idea I had). That then will delete the original note files from my computer and also (through Syncthing) from my phone.</span><br />
<br />
<a class='textlink' href='https://syncthing.net'>https://syncthing.net</a><br />
<a class='textlink' href='https://raku.org'>https://raku.org</a><br />
<a class='textlink' href='https://taskwarrior.org'>https://taskwarrior.org</a><br />
<br />
<span>Quick logger's user interface is as minimal as it gets. When I launch Quick logger, I'm greeted with a simple window where I can type plain text. Hit the "Log text" button, and voilà – the input is timestamped and saved as a file in my chosen directory. If I need to change the directory, the "Preferences" button brings up a window where I can set the notes folder and get back to logging.</span><br />
<br />
<span>For the code-savvy folks out there, Quick logger is a neat example of what you can achieve with Go and Fyne. It's a testament to building functional, cross-platform apps without getting bogged down in the nitty-gritty of platform-specific details. Thanks to Fyne, I am pleased with how easy it is to make mobile Android apps in Go.</span><br />
<br />
<a href='2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang/screenshot-android.png'><img alt='Quick logger running on Android' title='Quick logger running on Android' src='2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang/screenshot-android.png' /></a><br />
<br />
<span>My Android apps will never be polished, but they will get the job done, and this is precisely how I want them to be. Minimalistic but functional. I could spend more time polishing Quick logger, but my Quick logger app then may be the same as any other notes app out there (complicated or bloated).</span><br />
<br />
<h2 style='display: inline' id='Alleasypeasy'>All easy-peasy?</h2><br />
<br />
<span>I did have some issues with the app logo for Android, though. Android always showed the default app icon and not my custom icon whenever I used a custom <span class='inlinecode'>AndroidManifest.xml</span> for custom app storage permissions. Without a custom <span class='inlinecode'>AndroidAmnifest.xml</span> the app icon would be displayed under Android, but then the app would not have the <span class='inlinecode'>MANAGE_EXTERNAL_STORAGE</span> permission, which is required for Quick logger to write to a custom directory. I found a workaround, which I commented on here at Github:</span><br />
<br />
<a class='textlink' href='https://github.com/fyne-io/fyne/issues/3077#issuecomment-1912697360'>https://github.com/fyne-io/fyne/issues/3077#issuecomment-1912697360</a><br />
<br />
<span class='quote'>What worked however (app icon showing up) was to clone the fyne project, change the occurances of android.permission.INTERNET to android.permission.MANAGE_EXTERNAL_STORAGE (as these are all the changes I want in my custom android manifest) in the source tree, re-compile fyne. Now all works. I know, this is more of an hammer approach!</span><br />
<br />
<span>Hopefully, I won't need to use this workaround anymore. But for now, it is a fair tradeoff for what I am getting.</span><br />
<br />
<span>I hope this will inspire you to write your own small mobile apps in Go using the awesome Fyne framework! PS: The Quick logger logo was generated by ChatGPT.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other Go related posts are:</span><br />
<br />
<a class='textlink' href='./2023-04-09-algorithms-and-data-structures-in-golang-part-1.html'>2023-04-09 Algorithms and Data Structures in Go - Part 1</a><br />
<a class='textlink' href='./2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.html'>2024-03-03 A fine Fyne Android app for quickly logging ideas programmed in Go (You are currently reading this)</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>From `babylon5.buetow.org` to `*.buetow.cloud`</title>
<link href="gemini://foo.zone/gemfeed/2024-02-04-from-babylon5.buetow.org-to-.cloud.gmi" />
<id>gemini://foo.zone/gemfeed/2024-02-04-from-babylon5.buetow.org-to-.cloud.gmi</id>
<updated>2024-02-04T00:50:50+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Recently, my employer sent me to a week-long AWS course. After the course, there wasn't any hands-on project I could dive into immediately, so I moved parts of my personal infrastructure to AWS to level up a bit through practical hands-on.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>From <span class='inlinecode'>babylon5.buetow.org</span> to <span class='inlinecode'>*.buetow.cloud</span></h1><br />
<br />
<span class='quote'>Published at 2024-02-04T00:50:50+02:00</span><br />
<br />
<span>Recently, my employer sent me to a week-long AWS course. After the course, there wasn't any hands-on project I could dive into immediately, so I moved parts of my personal infrastructure to AWS to level up a bit through practical hands-on.</span><br />
<br />
<span>So, I migrated all of my Docker-based self-hosted services to AWS. Usually, I am not a big fan of big cloud providers and instead use smaller hosters or indie providers and self-made solutions. However, I also must go with the times and try out technologies currently hot on the job market. I don't want to become the old man who yells at cloud :D</span><br />
<br />
<a href='./from-.org-to-.cloud/old-man-yells-at-cloud.jpg'><img alt='Old man yells at cloud' title='Old man yells at cloud' src='./from-.org-to-.cloud/old-man-yells-at-cloud.jpg' /></a><br />
<br />
<h2 style='display: inline'>The old <span class='inlinecode'>*.buetow.org</span> way</h2><br />
<br />
<span>Before the migration, all those services were reachable through <span class='inlinecode'>buetow.org</span>-subdomains (Buetow is my last name) and ran on Docker containers on a single Rocky Linux 9 VM at Hetzner. And there was a Nginx reverse proxy with TLS offloading (with Let's Encrypt certificates). The Rocky Linux 9's hostname was <span class='inlinecode'>babylon5.buetow.org</span> (based on the Science Fiction series). </span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Babylon_5'>https://en.wikipedia.org/wiki/Babylon_5</a><br />
<br />
<span>The downsides of this setup were:</span><br />
<br />
<ul>
<li>Not highly available. If the server goes down, no service is reachable until it's repaired. To be fair, the Hetzner cloud VM is redundant by itself and would have re-spawned on a different worker node, I suppose. </li>
<li>Manual installation.</li>
</ul><br />
<span>About the manual installation part: I could have used a configuration management system like Rexify, Puppet, etc. But I decided against it back in time, as setting up Docker containers isn't so complicated through simple start scripts. And it's only a single Linux box where a manual installation is less painful. However, regular backups (which Hetzner can do automatically for you) were a must.</span><br />
<br />
<span>The benefits of this setup were:</span><br />
<br />
<ul>
<li>KISS (Keep it Simple Stupid)</li>
<li>Cheap</li>
</ul><br />
<h2 style='display: inline'>I kept my <span class='inlinecode'>buetow.org</span> OpenBSD boxes alive</h2><br />
<br />
<span>As pointed out, I only migrated the Docker-based self-hosted services (which run on the Babylon 5 Rocky Linux box) to AWS. Many self-hostable apps come with ready-to-use container images, making deploying them easy.</span><br />
<br />
<span>My other two OpenBSD VMs (<span class='inlinecode'>blowfish.buetow.org</span>, hosted at Hetzner, and <span class='inlinecode'>fishfinger.buetow.org</span>, hosted at OpenBSD Amsterdam) still run (and they will keep running) the following services:</span><br />
<br />
<ul>
<li>HTTP server for my websites (e.g. <span class='inlinecode'>https://foo.zone</span>, ...)</li>
<li>ACME for Let's Encrypt TLS certificate auto-renewal.</li>
<li>Gemini server for my capsules (e.g. <span class='inlinecode'>gemini://foo.zone</span>)</li>
<li>Authoritative DNS servers for my domains (but <span class='inlinecode'>buetow.cloud</span>, which is on Route 53 now)</li>
<li>Mail transfer agent (MTA)</li>
<li>My Gogios monitoring system.</li>
<li>My IRC bouncer.</li>
</ul><br />
<span>It is all automated with Rex, aka Rexify. This OpenBSD setup is my "fun" or "for pleasure" setup. Whereas the Rocky Linux 9 one I always considered the "pratical means to the end"-setup to have 3rd party Docker containers up and running with as little work as possible.</span><br />
<br />
<a class='textlink' href='https://www.rexify.org'>(R)?ex, the friendly automation framework</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>KISS server monitoring with Gogios</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's encrypt with OpenBSD and Rex</a><br />
<br />
<h2 style='display: inline'>The new <span class='inlinecode'>*.buetow.cloud</span> way</h2><br />
<br />
<span>With AWS, I decided to get myself a new domain name, as I could fully separate my AWS setup from my conventional setup and give Route 53 as an authoritative DNS a spin.</span><br />
<br />
<span>I decided to automate everything with Terraform, as I wanted to learn to use it as it appears standard now in the job market.</span><br />
<br />
<span>All services are installed automatically to AWS ECS Fargate. ECS is AWS's Elastic Container Service, and Fargate automatically manages the underlying hardware infrastructure (e.g., how many CPUs, RAM, etc.) for me. So I don't have to bother about having enough EC2 instances to serve my demands, for example.</span><br />
<br />
<span>The authoritative DNS for the <span class='inlinecode'>buetow.cloud</span> domain is AWS Route 53. TLS certificates are free here at AWS and offloaded through the AWS Application Load Balancer. The LB acts as a proxy to the ECS container instances of the services. A few services I run in ECS Fargate also require the AWS Network Load Balancer.</span><br />
<br />
<span>All services require some persistent storage. For that, I use an encrypted EFS file system, automatically replicated across all AZs (availability zones) of my region of choice, <span class='inlinecode'>eu-central-1</span>.</span><br />
<br />
<span>In case of an AZ outage, I could re-deploy all the failed containers in another AZ, and all the data would still be there.</span><br />
<br />
<span>The EFS automatically gets backed up by AWS for me following their standard Backup schedule. The daily backups are kept for 30 days. </span><br />
<br />
<span>Domain registration, TLS certificate configuration and configuration of the EFS backup were quickly done through the AWS web interface. These were only one-off tasks, so they weren't fully automated through Terraform. </span><br />
<br />
<span>You can find all Terraform manifests here:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/terraform'>https://codeberg.org/snonux/terraform</a><br />
<br />
<span>Whereas:</span><br />
<br />
<ul>
<li><span class='inlinecode'>org-buetow-base</span> sets up the bare VPC (IPv4 and IPv6 subnets in 3 AZs, EFS, ECR (the AWS container registry for some self-built containers) and Route 53 zone. It's the requirement for most other Terraform manifests in this repository.</li>
<li><span class='inlinecode'>org-buetow-bastion</span> sets up a minimal Amazon Linux EC2 instance where I can manually SSH into and look at the EFS file system (if required).</li>
<li><span class='inlinecode'>org-buetow-elb</span> sets up the Elastic Load Balancer, a prerequisite for any service running in ECS Fargate.</li>
<li><span class='inlinecode'>org-buetow-ecs</span> finally sets up and deploys all the Docker apps mentioned above. Any apps can be turned on or off via the <span class='inlinecode'>variables.tf</span> file.</li>
</ul><br />
<h2 style='display: inline'>The container apps</h2><br />
<br />
<span>And here, finally, is the list of all the container apps my Terraform manifests deploy. The FQDNs here may not be reachable. I spin them up only on demand (for cost reasons). All services are fully dual-stacked (IPv4 & IPv6). </span><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>flux.buetow.cloud</span></h3><br />
<br />
<span>Miniflux is a minimalist and opinionated feed reader. With the move to AWS, I also retired my bloated instance of NextCloud. So, with Miniflux, I retired from NextCloud News.</span><br />
<br />
<span>Miniflux requires two ECS containers. One is the Miniflux app, and the other is the PostgreSQL DB.</span><br />
<br />
<a class='textlink' href='https://miniflux.app/'>https://miniflux.app/</a><br />
<br />
<br />
<h3 style='display: inline'><span class='inlinecode'>audiobookshelf.buetow.cloud</span></h3><br />
<br />
<span>Audiobookshelf was the first Docker app I installed. It is a Self-hosted audiobook and podcast server. It comes with a neat web interface, and there is also an Android app available, which works also in offline mode. This is great, as I only have the ECS instance sometimes running for cost savings.</span><br />
<br />
<span>With Audiobookshelf, I replaced my former Audible subscription and my separate Podcast app. For Podcast synchronisation I used to use the Gpodder NextCloud sync app. But that one I retired now with Audiobookshelf as well :-)</span><br />
<br />
<a class='textlink' href='https://www.audiobookshelf.org'>https://www.audiobookshelf.org</a><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>syncthing.buetow.cloud</span></h3><br />
<br />
<span>Syncthing is a continuous file synchronisation program. In real-time, it synchronises files between two or more computers, safely protected from prying eyes. Your data is your own, and you deserve to choose where it is stored, whether it is shared with some third party, and how it's transmitted over the internet.</span><br />
<br />
<span>With Syncthing, I retired my old NextCloud Files and file sync client on all my devices. I also quit my NextCloud Notes setup. All my Notes are now plain Markdown files in a <span class='inlinecode'>Notes</span> directory. On Android, I can edit them with any text or Markdown editor (e.g. Obsidian), and they will be synchronised via Syncthing to my other computers, both forward and back.</span><br />
<br />
<span>I use Syncthing to synchronise some of my Phone's data (e.g. Notes, Pictures and other documents). Initially, I synced all of my pictures, videos, etc., with AWS. But that was pretty expensive. So for now, I use it only whilst travelling. Otherwise, I will use my Syncthing instance here on my LAN (I have a cheap cloud backup in AWS S3 Glacier Deep Archive, but that's for another blog post).</span><br />
<br />
<a class='textlink' href='https://syncthing.net/'>https://syncthing.net/</a><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>radicale.buetow.cloud</span></h3><br />
<br />
<span>Radicale is an excellent minimalist WebDAV calendar and contact synchronisation server. It was good enough to replace my NextCloud Calendar and NextCloud Contacts setup. Unfortunately, there wasn't a ready-to-use Docker image. So, I created my own.</span><br />
<br />
<span>On Android, it works great together with the DAVx5 client for synchronisation.</span><br />
<br />
<a class='textlink' href='https://radicale.org/'>https://radicale.org/</a><br />
<a class='textlink' href='https://codeberg.org/snonux/docker-radicale-server'>https://codeberg.org/snonux/docker-radicale-server</a><br />
<a class='textlink' href='https://www.davx5.com/'>https://www.davx5.com/</a><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>bag.buetow.cloud</span></h3><br />
<br />
<span>Wallabag is a self-hostable "save now - read later" service, and it also comes with an Android app which also has an offline mode. Think of Getpocket, but open-source!</span><br />
<br />
<a class='textlink' href='https://wallabag.org/'>https://wallabag.org/</a><br />
<a class='textlink' href='https://github.com/wallabag/wallabag'>https://github.com/wallabag/wallabag</a><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>anki.buetow.cloud</span></h3><br />
<br />
<span>Anki is a great (the greatest) flash-card learning program. I am currently learning Bulgarian as my 3rd language. There is also an Android app that has an offline mode, and advanced users can also self-host the server <span class='inlinecode'>anki-sync-server</span>. For some reason (not going into the details here), I had to build my own Docker image for the server.</span><br />
<br />
<a class='textlink' href='https://apps.ankiweb.net/'>https://apps.ankiweb.net/</a><br />
<a class='textlink' href='https://codeberg.org/snonux/docker-anki-sync-server'>https://codeberg.org/snonux/docker-anki-sync-server</a><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>vault.buetow.cloud</span></h3><br />
<br />
<span>Vaultwarden is an alternative implementation of the Bitwarden server API written in Rust and compatible with upstream Bitwarden clients, perfect for self-hosted deployment where running the official resource-heavy service might not be ideal. So, this is a great password manager server which can be used with any Bitwarden Android app.</span><br />
<br />
<span>I currently don't use it, but I may in the future. I made it available in my ECS Fargate setup anyway for now.</span><br />
<br />
<a class='textlink' href='https://github.com/dani-garcia/vaultwarden'>https://github.com/dani-garcia/vaultwarden</a><br />
<br />
<span>I currently use <span class='inlinecode'>geheim</span>, a Ruby command line tool I wrote, as my current password manager. You can read a little bit about it here under "More":</span><br />
<br />
<a class='textlink' href='./2022-06-15-sweating-the-small-stuff.html'>Sweating the small stuff </a><br />
<br />
<h3 style='display: inline'><span class='inlinecode'>bastion.buetow.cloud</span></h3><br />
<br />
<span>This is a tiny ARM-based Amazon Linux EC2 instance, which I sometimes spin up for investigation or manual work on my EFS file system in AWS.</span><br />
<br />
<h2 style='display: inline'>Conclusion</h2><br />
<br />
<span>I have learned a lot about AWS and Terraform during this migration. This was actually my first AWS hands-on project with practical use.</span><br />
<br />
<span>All of this was not particularly difficult (but at times a bit confusing). I see the use of Terraform managing more extensive infrastructures (it was even helpful for my small setup here). At least I know now what all the buzz is about :-). I don't think Terraform's HCL is a nice language. It get's it's job done, but it could be more elegant IMHO.</span><br />
<br />
<span>Deploying updates to AWS are much easier, and some of the manual maintenance burdens of my Rocky Linux 9 VM are no longer needed. So I will have more time for other projects! </span><br />
<br />
<span>Will I keep it in the cloud? I don't know yet. But maybe I won't renew the <span class='inlinecode'>buetow.cloud</span> domain and instead will use <span class='inlinecode'>*.cloud.buetow.org</span> or <span class='inlinecode'>*.aws.buetow.org</span> subdomains. </span><br />
<br />
<span>Will the AWS setup be cheaper than my old Rocky Linux setup? It might be more affordable as I only turn ECS and the load balancers on or off on-demand. Time will tell! The first forecasts suggest that it will be around the same costs.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>One reason why I love OpenBSD</title>
<link href="gemini://foo.zone/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi" />
<id>gemini://foo.zone/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi</id>
<updated>2024-01-13T22:55:33+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>HKISSFISHKISSFISHKISSFISHKISSFISH KISS</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='OnereasonwhyIloveOpenBSD'>One reason why I love OpenBSD</h1><br />
<br />
<span class='quote'>Published at 2024-01-13T22:55:33+02:00</span><br />
<br />
<pre>
FISHKISSFISHKIS
SFISHKISSFISHKISSFISH F
ISHK ISSFISHKISSFISHKISS FI
SHKISS FISHKISSFISHKISSFISS FIS
HKISSFISHKISSFISHKISSFISHKISSFISH KISS
FISHKISSFISHKISSFISHKISSFISHKISS FISHK
SSFISHKISSFISHKISSFISHKISSFISHKISSF
ISHKISSFISHKISSFISHKISSFISHKISSF ISHKI
SSFISHKISSFISHKISSFISHKISSFISHKIS SFIS
HKISSFISHKISSFISHKISSFISHKISS FIS
HKISSFISHKISSFISHKISSFISHK IS
SFISHKISSFISHKISSFISH K
ISSFISHKISSFISHK
</pre>
<br />
<span>I just upgraded my OpenBSD's from <span class='inlinecode'>7.3</span> to <span class='inlinecode'>7.4</span> by following the unattended upgrade guide:</span><br />
<br />
<a class='textlink' href='https://www.openbsd.org/faq/upgrade74.html'>https://www.openbsd.org/faq/upgrade74.html</a><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">$ doas installboot sd0 </font><i><font color="#ababab"># Update the bootloader (not for every upgrade required)</font></i>
<font color="#ff0000">$ doas sysupgrade </font><i><font color="#ababab"># Update all binaries (including Kernel)</font></i>
</pre>
<br />
<span><span class='inlinecode'>sysupgrade</span> downloaded and upgraded to the next release and rebooted the system. After the reboot, I run:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">$ doas sysmerge </font><i><font color="#ababab"># Update system configuration files</font></i>
<font color="#ff0000">$ doas pkg_add -u </font><i><font color="#ababab"># Update all packages</font></i>
<font color="#ff0000">$ doas reboot </font><i><font color="#ababab"># Just in case, reboot one more time</font></i>
</pre>
<br />
<span>That's it! Took me around 5 minutes in total! No issues, only these few comands, only 5 minutes! It just works! No problems, no conflicts, no tons (actually none) config file merge conflicts.</span><br />
<br />
<span>I followed the same procedure the previous times and never encountered any difficulties with any OpenBSD upgrades.</span><br />
<br />
<span>I have seen upgrades of other Operating Systems either take a long time or break the system (which takes manual steps to repair). That's just one of many reasons why I love OpenBSD! There appear never to be any problems. It just gets its job done!</span><br />
<br />
<a class='textlink' href='https://www.openbsd.org'>The OpenBSD Project</a><br />
<br />
<span>BTW: are you looking for an opinionated OpenBSD VM hoster? OpenBSD Amsterdam may be for you. They rock (I am having a VM there, too)!</span><br />
<br />
<a class='textlink' href='https://openbsd.amsterdam'>https://openbsd.amsterdam</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other *BSD related posts are:</span><br />
<br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD (You are currently reading this)</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect</title>
<link href="gemini://foo.zone/gemfeed/2024-01-09-site-reliability-engineering-part-3.gmi" />
<id>gemini://foo.zone/gemfeed/2024-01-09-site-reliability-engineering-part-3.gmi</id>
<updated>2024-01-09T18:35:48+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is the third part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='SiteReliabilityEngineeringPart3OnCallCultureandtheHumanAspect'>Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect</h1><br />
<br />
<span class='quote'>Published at 2024-01-09T18:35:48+02:00</span><br />
<br />
<span>This is the third part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</span><br />
<br />
<a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture</a><br />
<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE</a><br />
<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect (You are currently reading this)</a><br />
<br />
<pre>
..--""""----..
.-" ..--""""--.j-.
.-" .-" .--.""--..
.-" .-" ..--"-. \/ ;
.-" .-"_.--..--"" ..--' "-. :
.' .' / `. \..--"" __ _ \ ;
:.__.-" \ / .' ( )"-. Y
; ;: ( ) ( ). \
.': /:: : \ \
.'.-"\._ _.-" ; ; ( ) .-. ( ) \
" `.""" .j" : : \ ; ; \
bug /"""""/ ; ( ) "" :.( ) \
/\ / : \ \`.: _ \
: `. / ; `( ) (\/ :" \ \
\ `. : "-.(_)_.' t-' ;
\ `. ; ..--":
`. `. : ..--"" :
`. "-. ; ..--"" ;
`. "-.:_..--"" ..--"
`. : ..--""
"-. : ..--""
"-.;_..--""
</pre>
<br />
<h2 style='display: inline' id='OnCallCultureandtheHumanAspectPrioritisingWellbeingintheRealmofReliability'>On-Call Culture and the Human Aspect: Prioritising Well-being in the Realm of Reliability</h2><br />
<br />
<span>Site Reliability Engineering is synonymous with ensuring system reliability, but the human factor is an often-underestimated part of this discipline. Ensuring an healthy on-call culture is as critical as any technical solution. The well-being of the engineers is an important factor.</span><br />
<br />
<span>Firstly, a healthy on-call rotation is about more than just managing and responding to incidents. It's about the entire ecosystem that supports this practice. This involves reducing pain points, offering mentorship, rapid iteration, and ensuring that engineers have the right tools and processes. One ceavat is, that engineers should be willing to learn. Especially in on-call rotation embedding SREs with other engineers (for example Software Engineers or QA Engineers), it's difficult to motivate everyone to engage. QA Engineers want to test the software, Software Engineers want to implement new features; they don't want to troubleshoot and debug production incidents. It can be depressing for the mentoring SRE.</span><br />
<br />
<span>Furthermore, the metrics that measure the success of an on-call experience are only sometimes straightforward. While one might assume that fewer pages translate to better on-call expertise (which is true to a degree, as who wants to receive a page out of office hours?), it's not always the volume of pages that matters most. Trust, ownership, accountability, and effective communication play the important roles.</span><br />
<br />
<span>An important part is giving feedback about the on-call experience to ensure continuous learning. If alerts are mostly noise, they should be tuned or even eliminated. If alerts are actionable, can recurring tasks be automated? If there are knowledge gaps, is the documentation not good enough? Continuous retrospection ensures that not only do systems evolve, but the experience for the on-call engineers becomes progressively better.</span><br />
<br />
<span>Onboarding for on-call duties is a crucial aspect of ensuring the reliability and efficiency of systems. This process involves equipping new team members with the knowledge, tools, and support to handle incidents confidently. It begins with an overview of the system architecture and common challenges, followed by training on monitoring tools, alerting mechanisms, and incident response protocols. Shadowing experienced on-call engineers can offer practical exposure. Too often, new engineers are thrown into the cold water without proper onboarding and training because the more experienced engineers are too busy fire-fighting production issues in the first place.</span><br />
<br />
<span>An always-on, always-alert culture can lead to burnout. Engineers should be encouraged to recognise their limits, take breaks, and seek support when needed. This isn't just about individual health; a burnt-out engineer can have cascading effects on the entire team and the systems they manage. A successful on-call culture ensures that while systems are kept running, the engineers are kept happy, healthy, and supported. The more experienced engineers should take time to mentor the junior engineers, but the junior engineers should also be fully engaged, try to investigate and learn new things by themselves.</span><br />
<br />
<span>For the junior engineer, it's too easy to fall back and ask the experts in the team every time an issue arises. This seems reasonable, but serving recipes for solving production issues on a silver tablet won't scale forever, as there are infinite scenarios of how production systems can break. So every engineer should learn to debug, troubleshoot and resolve production incidents independently. The experts will still be there for guidance and step in when the junior gets stuck after trying, but the experts should also learn to step down so that lesser experienced engineers can step up and learn. But mistakes can always happen here; that's why having a blameless on-call culture is essential.</span><br />
<br />
<span>A blameless on-call culture is a must for a safe and collaborative environment where engineers can effectively respond to incidents without fear of retribution. This approach acknowledges that mistakes are a natural part of the learning and innovation process. When individuals are assured they won't be punished for errors, they're more likely to openly discuss mistakes, allowing the entire team to learn and grow from each incident. Furthermore, a blameless culture promotes psychological safety, enhances job satisfaction, reduces burnout, and ensures that talent remains committed and engaged.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Bash Golf Part 3</title>
<link href="gemini://foo.zone/gemfeed/2023-12-10-bash-golf-part-3.gmi" />
<id>gemini://foo.zone/gemfeed/2023-12-10-bash-golf-part-3.gmi</id>
<updated>2023-12-10T11:35:54+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is the third blog post about my Bash Golf series. This series is random Bash tips, tricks, and weirdnesses I have encountered over time. </summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='BashGolfPart3'>Bash Golf Part 3</h1><br />
<br />
<span class='quote'>Published at 2023-12-10T11:35:54+02:00</span><br />
<br />
<pre>
'\ '\ '\ . . |>18>>
\ \ \ . ' . |
O>> O>> O>> . 'o |
\ .\. .. .\. .. . |
/\ . /\ . /\ . . |
/ / . / / .'. / / .' . |
jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Art by Joan Stark, mod. by Paul Buetow
</pre>
<br />
<span>This is the third blog post about my Bash Golf series. This series is random Bash tips, tricks, and weirdnesses I have encountered over time. </span><br />
<br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3 (You are currently reading this)</a><br />
<br />
<h2 style='display: inline' id='FUNCNAME'><span class='inlinecode'>FUNCNAME</span></h2><br />
<br />
<span><span class='inlinecode'>FUNCNAME</span> is an array you are looking for a way to dynamically determine the name of the current function (which could be considered the callee in the context of its own execution), you can use the special variable <span class='inlinecode'>FUNCNAME</span>. This is an array variable that contains the names of all shell functions currently in the execution call stack. The element <span class='inlinecode'>FUNCNAME[0]</span> holds the name of the currently executing function, <span class='inlinecode'>FUNCNAME[1]</span> the name of the function that called that, and so on.</span><br />
<br />
<span>This is particularly useful for logging when you want to include the callee function in the log output. E.g. look at this log helper:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">log ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -r </font><font color="#ff0000">level</font><font color="#F3E651">=</font><font color="#bb00ff">"$1"</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">shift</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -r </font><font color="#ff0000">message</font><font color="#F3E651">=</font><font color="#bb00ff">"$1"</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">shift</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -i </font><font color="#ff0000">pid</font><font color="#F3E651">=</font><font color="#bb00ff">"$"</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -r </font><font color="#ff0000">callee</font><font color="#F3E651">=</font><font color="#ff0000">${FUNCNAME[1]}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -r </font><font color="#ff0000">stamp</font><font color="#F3E651">=</font><font color="#ff0000">$(</font><font color="#ff0000">date </font><font color="#F3E651">+%</font><font color="#ff0000">Y</font><font color="#F3E651">%</font><font color="#ff0000">m</font><font color="#F3E651">%</font><font color="#ff0000">d-</font><font color="#F3E651">%</font><font color="#ff0000">H</font><font color="#F3E651">%</font><font color="#ff0000">M</font><font color="#F3E651">%</font><font color="#ff0000">S</font><font color="#F3E651">)</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"$level|$stamp|$pid|$callee|$message"</font><font color="#ff0000"> </font><font color="#F3E651">>&</font><font color="#bb00ff">2</font>
<font color="#ff0000">}</font>
<font color="#7bc710">at_home_friday_evening ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> log INFO </font><font color="#bb00ff">'One Peperoni Pizza, please'</font>
<font color="#ff0000">}</font>
<font color="#ff0000">at_home_friday_evening</font>
</pre>
<br />
<span>The output is as follows:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">❯ </font><font color="#F3E651">.</font><font color="#ff0000">/logexample</font><font color="#F3E651">.</font><font color="#ff0000">sh</font>
<font color="#ff0000">INFO</font><font color="#F3E651">|</font><font color="#bb00ff">20231210</font><font color="#ff0000">-</font><font color="#bb00ff">082732</font><font color="#F3E651">|</font><font color="#bb00ff">123002</font><font color="#F3E651">|</font><font color="#ff0000">at_home_friday_evening</font><font color="#F3E651">|</font><font color="#ff0000">One Peperoni Pizza</font><font color="#F3E651">,</font><font color="#ff0000"> please</font>
</pre>
<br />
<h2 style='display: inline' id=''><span class='inlinecode'>:(){ :|:& };:</span></h2><br />
<br />
<span>This one may be widely known already, but I am including it here as I found a cute image illustrating it. But to break <span class='inlinecode'>:(){ :|:& };:</span> down:</span><br />
<br />
<ul>
<li><span class='inlinecode'>:(){ }</span> is really a declaration of the function <span class='inlinecode'>:</span></li>
<li>The <span class='inlinecode'>;</span> is ending the current statement</li>
<li>The <span class='inlinecode'>:</span> at the end is calling the function <span class='inlinecode'>:</span></li>
<li><span class='inlinecode'>:|:&</span> is the function body</li>
</ul><br />
<span>Let's break down the function body <span class='inlinecode'>:|:&</span>: </span><br />
<br />
<ul>
<li>The first <span class='inlinecode'>:</span> is calling the function recursively</li>
<li>The <span class='inlinecode'>|:</span> is piping the output to the function <span class='inlinecode'>:</span> again (parallel recursion)</li>
<li>The <span class='inlinecode'>&</span> lets it run in the background.</li>
</ul><br />
<span>So, it's a fork bomb. If you run it, your computer will run out of resources eventually. (Modern Linux distributions could have reasonable limits configured for your login session, so it won't bring down your whole system anymore unless you run it as <span class='inlinecode'>root</span>!)</span><br />
<br />
<span>And here is the cute illustration:</span><br />
<br />
<a href='./2023-12-10-bash-golf-part-3/bash-fork-bomb.jpg'><img alt='Bash fork bomb' title='Bash fork bomb' src='./2023-12-10-bash-golf-part-3/bash-fork-bomb.jpg' /></a><br />
<br />
<h2 style='display: inline' id='Innerfunctions'>Inner functions</h2><br />
<br />
<span>Bash defines variables as it is interpreting the code. The same applies to function declarations. Let's consider this code:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">outer()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><font color="#7bc710">inner()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">'Intel inside!'</font>
<font color="#ff0000"> }</font>
<font color="#ff0000"> inner</font>
<font color="#ff0000">}</font>
<font color="#ff0000">inner</font>
<font color="#ff0000">outer</font>
<font color="#ff0000">inner</font>
</pre>
<br />
<span>And let's execute it:</span><br />
<br />
<pre>
❯ ./inner.sh
/tmp/inner.sh: line 10: inner: command not found
Intel inside!
Intel inside!
</pre>
<br />
<span>What happened? The first time <span class='inlinecode'>inner</span> was called, it wasn't defined yet. That only happens after the <span class='inlinecode'>outer</span> run. Note that <span class='inlinecode'>inner</span> will still be globally defined. But functions can be declared multiple times (the last version wins):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">outer1()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><font color="#7bc710">inner()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">'Intel inside!'</font>
<font color="#ff0000"> }</font>
<font color="#ff0000"> inner</font>
<font color="#ff0000">}</font>
<font color="#7bc710">outer2()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><font color="#7bc710">inner()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">'Wintel inside!'</font>
<font color="#ff0000"> }</font>
<font color="#ff0000"> inner</font>
<font color="#ff0000">}</font>
<font color="#ff0000">outer1</font>
<font color="#ff0000">inner</font>
<font color="#ff0000">outer2</font>
<font color="#ff0000">inner</font>
</pre>
<br />
<span>And let's run it:</span><br />
<br />
<pre>
❯ ./inner2.sh
Intel inside!
Intel inside!
Wintel inside!
Wintel inside!
</pre>
<br />
<h2 style='display: inline' id='Exportingfunctions'>Exporting functions</h2><br />
<br />
<span>Have you ever wondered how to execute a shell function in parallel through <span class='inlinecode'>xargs</span>? The problem is that this won't work:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">some_expensive_operations()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"Doing expensive operations with '$1' from pid $"</font>
<font color="#ff0000">}</font>
<b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> {</font><font color="#bb00ff">0</font><font color="#F3E651">..</font><font color="#bb00ff">9</font><font color="#ff0000">}</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b><font color="#ff0000"> echo </font><font color="#ff0000">$i</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">done</font></b><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000"> xargs -P</font><font color="#bb00ff">10</font><font color="#ff0000"> -I{} bash -c </font><font color="#bb00ff">'some_expensive_operations "{}"'</font>
</pre>
<br />
<span>We try here to run ten parallel processes; each of them should run the <span class='inlinecode'>some_expensive_operations</span> function with a different argument. The arguments are provided to <span class='inlinecode'>xargs</span> through <span class='inlinecode'>STDIN</span> one per line. When executed, we get this:</span><br />
<br />
<pre>
❯ ./xargs.sh
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
bash: line 1: some_expensive_operations: command not found
</pre>
<br />
<span>There's an easy solution for this. Just export the function! It will then be magically available in any sub-shell!</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">some_expensive_operations()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"Doing expensive operations with '$1' from pid $"</font>
<font color="#ff0000">}</font>
<b><font color="#ffffff">export</font></b><font color="#ff0000"> -f some_expensive_operations</font>
<b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> {</font><font color="#bb00ff">0</font><font color="#F3E651">..</font><font color="#bb00ff">9</font><font color="#ff0000">}</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b><font color="#ff0000"> echo </font><font color="#ff0000">$i</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">done</font></b><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000"> xargs -P</font><font color="#bb00ff">10</font><font color="#ff0000"> -I{} bash -c </font><font color="#bb00ff">'some_expensive_operations "{}"'</font>
</pre>
<br />
<span>When we run this now, we get:</span><br />
<br />
<pre>
❯ ./xargs.sh
Doing expensive operations with '0' from pid 132831
Doing expensive operations with '1' from pid 132832
Doing expensive operations with '2' from pid 132833
Doing expensive operations with '3' from pid 132834
Doing expensive operations with '4' from pid 132835
Doing expensive operations with '5' from pid 132836
Doing expensive operations with '6' from pid 132837
Doing expensive operations with '7' from pid 132838
Doing expensive operations with '8' from pid 132839
Doing expensive operations with '9' from pid 132840
</pre>
<br />
<span>If <span class='inlinecode'>some_expensive_function</span> would call another function, the other function must also be exported. Otherwise, there will be a runtime error again. E.g., this won't work:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">some_other_function()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"$1"</font>
<font color="#ff0000">}</font>
<font color="#7bc710">some_expensive_operations()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> some_other_function </font><font color="#bb00ff">"Doing expensive operations with '$1' from pid $"</font>
<font color="#ff0000">}</font>
<b><font color="#ffffff">export</font></b><font color="#ff0000"> -f some_expensive_operations</font>
<b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> {</font><font color="#bb00ff">0</font><font color="#F3E651">..</font><font color="#bb00ff">9</font><font color="#ff0000">}</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b><font color="#ff0000"> echo </font><font color="#ff0000">$i</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">done</font></b><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000"> xargs -P</font><font color="#bb00ff">10</font><font color="#ff0000"> -I{} bash -c </font><font color="#bb00ff">'some_expensive_operations "{}"'</font>
</pre>
<br />
<span>... because <span class='inlinecode'>some_other_function</span> isn't exported! You will also need to add an <span class='inlinecode'>export -f some_other_function</span>!</span><br />
<br />
<h2 style='display: inline' id='Dynamicvariableswithlocal'>Dynamic variables with <span class='inlinecode'>local</span></h2><br />
<br />
<span>You may know that <span class='inlinecode'>local</span> is how to declare local variables in a function. Most don't know that those variables actually have dynamic scope. Let's consider the following example:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#7bc710">foo()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">foo</font><font color="#F3E651">=</font><font color="#ff0000">bar </font><i><font color="#ababab"># Declare local/dynamic variable</font></i>
<font color="#ff0000"> bar</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"$foo"</font>
<font color="#ff0000">}</font>
<font color="#7bc710">bar()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"$foo"</font>
<font color="#ff0000"> </font><font color="#ff0000">foo</font><font color="#F3E651">=</font><font color="#ff0000">baz</font>
<font color="#ff0000">}</font>
<font color="#ff0000">foo</font><font color="#F3E651">=</font><font color="#ff0000">foo </font><i><font color="#ababab"># Declare global variable</font></i>
<font color="#ff0000">foo </font><i><font color="#ababab"># Call function foo</font></i>
<font color="#ff0000">echo </font><font color="#bb00ff">"$foo"</font>
</pre>
<br />
<span>Let's pause a minute. What do you think the output would be?</span><br />
<br />
<span>Let's run it:</span><br />
<br />
<pre>
❯ ./dynamic.sh
bar
baz
foo
</pre>
<br />
<span>What happened? The variable <span class='inlinecode'>foo</span> (declared with <span class='inlinecode'>local</span>) is available in the function it was declared in and in all other functions down the call stack! We can even modify the value of <span class='inlinecode'>foo</span>, and the change will be visible up the call stack. It's not a global variable; on the last line, <span class='inlinecode'>echo "$foo"</span> echoes the global variable content.</span><br />
<br />
<br />
<h2 style='display: inline' id='ifconditionals'><span class='inlinecode'>if</span> conditionals</h2><br />
<br />
<span>Consider all variants here more or less equivalent:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<b><font color="#ffffff">declare</font></b><font color="#ff0000"> -r </font><font color="#ff0000">foo</font><font color="#F3E651">=</font><font color="#ff0000">foo</font>
<b><font color="#ffffff">declare</font></b><font color="#ff0000"> -r </font><font color="#ff0000">bar</font><font color="#F3E651">=</font><font color="#ff0000">bar</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> bar </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo ok1</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">==</font><font color="#ff0000"> bar </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo ok2a</font>
<b><font color="#ffffff">fi</font></b>
<font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">==</font><font color="#ff0000"> bar </font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">&&</font><font color="#ff0000"> echo ok2b</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[[</font><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">==</font><font color="#ff0000"> bar </font><font color="#F3E651">]];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo ok3a</font>
<b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><font color="#F3E651">[[</font><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">==</font><font color="#ff0000"> bar </font><font color="#F3E651">]]</font><font color="#ff0000"> </font><font color="#F3E651">&&</font><font color="#ff0000"> echo ok3b</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><b><font color="#ffffff">test</font></b><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><b><font color="#ffffff">test</font></b><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> bar</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo ok4a</font>
<b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">test</font></b><font color="#ff0000"> </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> foo </font><font color="#F3E651">&&</font><font color="#ff0000"> </font><b><font color="#ffffff">test</font></b><font color="#ff0000"> </font><font color="#bb00ff">"$bar"</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> bar </font><font color="#F3E651">&&</font><font color="#ff0000"> echo ok4b</font>
</pre>
<br />
<span>The output we get is:</span><br />
<br />
<pre>
❯ ./if.sh
ok1
ok2a
ok2b
ok3a
ok3b
ok4a
ok4b
</pre>
<br />
<h2 style='display: inline' id='Multilinecomments'>Multi-line comments</h2><br />
<br />
<span>You all know how to comment. Put a <span class='inlinecode'>#</span> in front of it. You could use multiple single-line comments or abuse heredocs and redirect it to the <span class='inlinecode'>:</span> no-op command to emulate multi-line comments. </span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<i><font color="#ababab"># Single line comment</font></i>
<i><font color="#ababab"># These are two single line</font></i>
<i><font color="#ababab"># comments one after another</font></i>
<font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#F3E651"><<</font><font color="#ff0000">COMMENT</font>
<font color="#ff0000">This is another way a</font>
<font color="#ff0000">multi line comment</font>
<font color="#ff0000">could be written</font><font color="#F3E651">!</font>
<font color="#ff0000">COMMENT</font>
</pre>
<br />
<span>I will not demonstrate the execution of this script, as it won't print anything! It's obviously not the most pretty way of commenting on your code, but it could sometimes be handy!</span><br />
<br />
<h2 style='display: inline' id='Dontchangeitwhileitsexecuted'>Don't change it while it's executed</h2><br />
<br />
<span>Consider this script:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/usr/bin/env bash</font></i>
<font color="#ff0000">echo foo</font>
<font color="#ff0000">echo echo baz </font><font color="#F3E651">>></font><font color="#ff0000"> </font><font color="#ff0000">$0</font>
<font color="#ff0000">echo bar</font>
</pre>
<br />
<span>When it is run, it will do:</span><br />
<br />
<pre>
❯ ./if.sh
foo
bar
baz
❯ cat if.sh
#!/usr/bin/env bash
echo foo
echo echo baz >> $0
echo bar
echo baz
</pre>
<br />
<span>So what happened? The <span class='inlinecode'>echo baz</span> line was appended to the script while it was still executed! And the interpreter also picked it up! It tells us that Bash evaluates each line as it encounters it. This can lead to nasty side effects when editing the script while it is still being executed! You should always keep this in mind!</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>2021-05-16 Personal Bash coding style guide</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3 (You are currently reading this)</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Site Reliability Engineering - Part 2: Operational Balance in SRE</title>
<link href="gemini://foo.zone/gemfeed/2023-11-19-site-reliability-engineering-part-2.gmi" />
<id>gemini://foo.zone/gemfeed/2023-11-19-site-reliability-engineering-part-2.gmi</id>
<updated>2023-11-19T00:18:18+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is the second part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='SiteReliabilityEngineeringPart2OperationalBalanceinSRE'>Site Reliability Engineering - Part 2: Operational Balance in SRE</h1><br />
<br />
<span class='quote'>Published at 2023-11-19T00:18:18+03:00</span><br />
<br />
<span>This is the second part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</span><br />
<br />
<a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture</a><br />
<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE (You are currently reading this)</a><br />
<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect</a><br />
<br />
<pre>
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⣾⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⣾⠿⠿⠿⠶⠾⠿⠿⣿⣿⣿⣿⣿⣿⠿⠿⠶⠶⠿⠿⠿⣷⠀⠀⠀⠀
⠀⠀⠀⣸⢿⣆⠀⠀⠀⠀⠀⠀⠀⠙⢿⡿⠉⠀⠀⠀⠀⠀⠀⠀⣸⣿⡆⠀⠀⠀
⠀⠀⢠⡟⠀⢻⣆⠀⠀⠀⠀⠀⠀⠀⣾⣧⠀⠀⠀⠀⠀⠀⠀⣰⡟⠀⢻⡄⠀⠀
⠀⢀⣾⠃⠀⠀⢿⡄⠀⠀⠀⠀⠀⢠⣿⣿⡀⠀⠀⠀⠀⠀⢠⡿⠀⠀⠘⣷⡀⠀
⠀⣼⣏⣀⣀⣀⣈⣿⡀⠀⠀⠀⠀⣸⣿⣿⡇⠀⠀⠀⠀⢀⣿⣃⣀⣀⣀⣸⣧⠀
⠀⢻⣿⣿⣿⣿⣿⣿⠃⠀⠀⠀⠀⣿⣿⣿⣿⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⣿⡿⠀
⠀⠀⠉⠛⠛⠛⠋⠁⠀⠀⠀⠀⢸⣿⣿⣿⣿⡆⠀⠀⠀⠀⠈⠙⠛⠛⠛⠉⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣾⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣸⣿⣿⣿⣿⣿⣿⣆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠴⠶⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠶⠦⠀⠀
</pre>
<br />
<h2 style='display: inline' id='OperationalBalanceinSREFindingtheEquilibriuminReliabilityandVelocity'>Operational Balance in SRE: Finding the Equilibrium in Reliability and Velocity</h2><br />
<br />
<span>Site Reliability Engineering has established itself as more than just a set of best practices or methodologies. Instead, it stands as a beacon of operational excellence, which guides engineering teams through the turbulent waters of modern software development and system management.</span><br />
<br />
<span>In the universe of software production, two fundamental forces are often at odds: The drive for rapid feature release (velocity) and the need for system reliability. Traditionally, the faster teams moved, the more risk was introduced into systems. SRE offers a approach to mitigate these conflicting drives through concepts like error budgets and SLIs/SLOs. These mechanisms offer a tangible metric, allowing teams to quantify how much they can push changes while ensuring they don't compromise system health. Thus, the error budget becomes a balancing act, where teams weigh the trade-offs between innovation and reliability.</span><br />
<br />
<span>An important part of this balance is the dichotomy between operations and coding. According to SRE principles, an engineer should ideally spend an equal amount of time on operations work and coding - 50% on each. This isn't just a random metric; it's a reflection of the value SRE places on both maintaining operational excellence and progressing forward with innovations. This balance ensures that while SREs are solving today's problems, they are also preparing for tomorrow's challenges. </span><br />
<br />
<span>However, not all operational tasks are equal. SRE differentiates between "ops work" and "toil". While ops work is integral to system maintenance and can provide value, toil represents repetitive, mundane tasks which offer little value in the long run. Recognising and minimising toil is crucial. A culture that allows engineers to drown in toil stifles innovation and growth. Hence, an organisation's approach to toil indicates its operational health and commitment to balance.</span><br />
<br />
<span>A cornerstone of achieving operational balance lies in the tools and processes SREs use. Effective monitoring, observability tools, and ensuring that tools can handle high cardinality data are foundational. These aren't just technical requisites but reflective of an organisational culture prioritising proactive problem-solving. By having systems that effectively flag potential issues before they escalate, SREs can maintain the balance between system stability and forward momentum.</span><br />
<br />
<span>Moreover, operational balance isn't just a technological or process challenge; it's a human one. The health of on-call engineers is as crucial as the health of the services they manage. On-call postmortems, continuous feedback loops, and recognising gaps (be it tooling, operational expertise, or resources) ensure that the human elements of operations are noticed. </span><br />
<br />
<span>In conclusion, operational balance in SRE isn't static thing but an ongoing journey. It requires organisations to constantly evaluate their practices, tools, and, most importantly, their culture. By achieving this balance, organisations can ensure that they have time for innovation while maintaining the robustness and reliability of their systems, resulting in sustainable long-term success.</span><br />
<br />
<span>That all sounds very romantic. The truth is, it's brutal to archive the perfect balance. No system will ever be perfect. But at least we should aim for it!</span><br />
<br />
<span>Continue with the third part of this series:</span><br />
<br />
<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>'Mind Management' book notes</title>
<link href="gemini://foo.zone/gemfeed/2023-11-11-mind-management-book-notes.gmi" />
<id>gemini://foo.zone/gemfeed/2023-11-11-mind-management-book-notes.gmi</id>
<updated>2023-11-11T22:21:47+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>These are my personal takeaways after reading 'Mind Management' by David Kadavy. Note that the book contains much more knowledge wisdom and that these notes only contain points I personally found worth writing down. This is mainly for my own use, but you might find it helpful too.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='MindManagementbooknotes'>"Mind Management" book notes</h1><br />
<br />
<span class='quote'>Published at 2023-11-11T22:21:47+02:00</span><br />
<br />
<span>These are my personal takeaways after reading "Mind Management" by David Kadavy. Note that the book contains much more knowledge wisdom and that these notes only contain points I personally found worth writing down. This is mainly for my own use, but you might find it helpful too.</span><br />
<br />
<pre>
,.......... ..........,
,..,' '.' ',..,
,' ,' : ', ',
,' ,' : ', ',
,' ,' : ', ',
,' ,'............., : ,.............', ',
,' '............ '.' ............' ',
'''''''''''''''''';''';''''''''''''''''''
'''
</pre>
<br />
<span>Productivity isn't about time management - it's about mind management. When you put a lot of effort into something, there are:</span><br />
<br />
<ul>
<li>The point of diminishing returns</li>
<li>The point of negative return</li>
</ul><br />
<h2 style='display: inline' id='Emptyslotsinthecalendar'>Empty slots in the calendar</h2><br />
<br />
<span>If we do more things in less time and use all possible slots, speed read, etc., we are more productive. But in reality, that's not the entire truth. You also exchange one thing against everything else.... You cut out too much from your actual life.</span><br />
<br />
<h2 style='display: inline' id='Whenyousafetime'>When you safe time...</h2><br />
<br />
<span>...keep it.</span><br />
<br />
<ul>
<li>stare out of the window; that's good for you.</li>
<li>Creative thinking needs space. It will pay dividends tomorrow.</li>
<li>You will be rewarded with the "Eureka effect" - a sudden new insight.</li>
</ul><br />
<h2 style='display: inline' id='Followyourmood'>Follow your mood</h2><br />
<br />
<span>Ask yourself: what is my mood now? We never have the energy to do anything, so the better strategy is to follow your current mode and energy. E.g.:</span><br />
<br />
<ul>
<li>Didn't sleep enough today? Then, do simple, non-demanding tasks at work</li>
<li>Had a great sleep, and there is even time before work starts? Pull in a workout...</li>
</ul><br />
<h2 style='display: inline' id='Boostingcreativity'>Boosting creativity</h2><br />
<br />
<span>The morning without coffee is a gift for creativity, but you often get distracted. Minimize distractions, too. I have no window to stare out but a plain blank wall.</span><br />
<br />
<ul>
<li>The busier you are, the less creative you will be.</li>
<li>Event time (divergent thinking) vs clock time (convergent thinking)</li>
<li>Don't race with time but walk alongside it as rough time lines.</li>
<li>Don't judge every day after the harvest, but the seed you lay</li>
</ul><br />
<h2 style='display: inline' id='Therightmoodforthetaskathand'>The right mood for the task at hand</h2><br />
<br />
<span>We need to try many different combinations. Limiting ourselves and trying too hard makes us frustrated and burn out. Creativity requires many iterations.</span><br />
<br />
<span>I can only work according to my available brain power. </span><br />
<br />
<span>I can also change my mood according to what needs improvement. Just imagine the last time you were in that mood and then try to get into it. It can take several tries to hit a working mood. Try to replicate that mental state. This can also be by location or by another habit, e.g. by a beer.</span><br />
<br />
<span>Once you are in a mental state, don't try to change it. It will take a while for your brain to switch to a completely different state.</span><br />
<br />
<span>Week of want. For a week, only do what you want and not what you must do. Your ideas will get much more expansive.</span><br />
<br />
<span>It gives you pleasure and is in a good mood. This increases creativity if you do what you want to do.</span><br />
<br />
<h2 style='display: inline' id='Creativityhacks'>Creativity hacks</h2><br />
<br />
<ul>
<li>Coffee can cause anxiety.</li>
<li>Take phentermine with coffee to take off the edge and have a relaxed focus</li>
<li>Green tea, which tastes sweet plus supplement boost.</li>
<li>Also wine. But be careful with alcohol. Don't drink a whole bottle.</li>
<li>Have a machine without distractions and internet access for writing.</li>
<li>Go to open spaces for creativity.</li>
<li>Go to closed spaces for polishing.</li>
</ul><br />
<h2 style='display: inline' id='Planningandstrategizing'>Planning and strategizing</h2><br />
<br />
<span>Minds work better in sprints and not in marathons. Have a weekly plan, not a daily one.</span><br />
<br />
<ul>
<li>Alternating incubation to avoid blocks.</li>
<li>Build on systems that use chaos for growth, e.g. unplanned disasters.</li>
<li>Things don't go after the plan is the plan. Be anti-fragile.</li>
</ul><br />
<span>Organize by mental state. In the time management context, the mental state doesn't exist. You schedule as many things as possible by project. In the mind management context, mental state is everything. You could prepare by mental state and not by assignment.</span><br />
<br />
<span>You could schedule exploratory tasks when you are under grief. Sound systems should create slack for creativity. Plan only for a few minutes.</span><br />
<br />
<h2 style='display: inline' id='Fakeituntilyoumakeit'>Fake it until you make it. </h2><br />
<br />
<ul>
<li>E.g. act calm if you want to be calm.</li>
<li>Talk slowly and deepen your voice a bit to appear more confident. You will also become more confident.</li>
<li>Also, use power positions for better confidence.</li>
</ul><br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other book notes of mine are:</span><br />
<br />
<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 "The Pragmatic Programmer" book notes</a><br />
<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 "Never split the difference" book notes</a><br />
<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 "The Obstacle is the Way" book notes</a><br />
<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes</a><br />
<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 "Mind Management" book notes (You are currently reading this)</a><br />
<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 "Slow Productivity" book notes</a><br />
<br />
<a class='textlink' href='../resources.html'>More books and other resources I found useful.</a><br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>KISS static web photo albums with `photoalbum.sh`</title>
<link href="gemini://foo.zone/gemfeed/2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.gmi" />
<id>gemini://foo.zone/gemfeed/2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.gmi</id>
<updated>2023-10-29T22:25:04+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Once in a while, I share photos on the inter-web with either family and friends or on my The Irregular Ninja photo site. One hobby of mine is photography (even though I don't have enough time for it - so I am primarily a point-and-shoot photographer).</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='KISSstaticwebphotoalbumswithphotoalbumsh'>KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></h1><br />
<br />
<span class='quote'>Published at 2023-10-29T22:25:04+02:00</span><br />
<br />
<pre>
___ .---------.._
______!fsc!_....-' .g8888888p. '-------....._
.' // .g8: :8p..---....___ \'.
| foo.zone // () d88: :88b|==========! !|
| // 888: :888|==========| !|
|___ \\_______'T88888888888P''----------'//|
| \ """"""""""""""""""""""""""""""""""/ |
| !...._____ .="""=. .[] ____...! |
| / ! .g$p. ! .[] : |
| ! : $$$ : .[] : |
| !irregular.ninja ! 'T$P' ! .[] '.|
| \__ "=._.=" .() __ |
|.--' '----._______________________.----' '--.|
'._____________________________________________.'
</pre>
<br />
<h2 style='display: inline' id='Motivation'>Motivation</h2><br />
<br />
<span>Once in a while, I share photos on the inter-web with either family and friends or on my The Irregular Ninja photo site. One hobby of mine is photography (even though I don't have enough time for it - so I am primarily a point-and-shoot photographer).</span><br />
<br />
<span>I'm not particularly eager to use any photo social sharing platforms such as Flickr, 500px (I used them regularly in the past), etc., anymore. I value self-hosting, DIY and privacy (nobody should data mine my photos), and no third party should have any rights to my pictures. </span><br />
<br />
<span>I value KISS (keep it simple and stupid) and simplicity. All that's required for a web photo album is some simple HTML and spice it up with CSS. No need for JavaScript, no need for a complex dynamic website. </span><br />
<br />
<h2 style='display: inline' id='Introducingphotoalbumsh'>Introducing <span class='inlinecode'>photoalbum.sh</span></h2><br />
<br />
<span><span class='inlinecode'>photoalbum.sh</span> is a minimal Bash (Bourne Again Shell) script for Unix-like operating systems (such as Linux) to generate static web photo albums. The resulting static photo album is pure HTML+CSS (without any JavaScript!). It is specially designed to be as simple as possible.</span><br />
<br />
<h2 style='display: inline' id='Installation'>Installation</h2><br />
<br />
<span>Installation is straightforward. All required is a recent version of GNU Bash, GNU Make, Git and ImageMagick. On Fedora, the dependencies are installed with:</span><br />
<br />
<pre>
% sudo dnf install -y ImageMagick make git
</pre>
<br />
<span>Now, clone, make and install the script:</span><br />
<br />
<pre>
% git clone https://codeberg.org/snonux/photoalbum
Cloning into 'photoalbum'...
remote: Enumerating objects: 1624, done.
remote: Total 1624 (delta 0), reused 0 (delta 0), pack-reused 1624
Receiving objects: 100% (1624/1624), 193.36 KiB | 1.49 MiB/s, done.
Resolving deltas: 100% (1227/1227), done.
% cd photoalbum
/home/paul/photoalbum
% make
cut -d' ' -f2 changelog | head -n 1 | sed 's/(//;s/)//' > .version
test ! -d ./bin && mkdir ./bin || exit 0
sed "s/PHOTOALBUMVERSION/$(cat .version)/" src/photoalbum.sh > ./bin/photoalbum
chmod 0755 ./bin/photoalbum
% sudo make install
test ! -d /usr/bin && mkdir -p /usr/bin || exit 0
cp ./bin/* /usr/bin
test ! -d /usr/share/photoalbum/templates && mkdir -p /usr/share/photoalbum/templates || exit 0
cp -R ./share/templates /usr/share/photoalbum/
test ! -d /etc/default && mkdir -p /etc/default || exit 0
cp ./src/photoalbum.default.conf /etc/default/photoalbum
</pre>
<br />
<span>You should now have the <span class='inlinecode'>photoalbum</span> command in your <span class='inlinecode'>$PATH</span>. But wait to use it! First, it needs to be set up!</span><br />
<br />
<pre>
% photoalbum version
This is Photoalbum Version 0.5.1
</pre>
<br />
<h2 style='display: inline' id='Settingitup'>Setting it up</h2><br />
<br />
<span>Now, it's time to set up the Irregular Ninja static web photo album (or any other web photo album you may be setting up!)! Create a directory (here: <span class='inlinecode'>irregular.ninja</span> for the Irregular Ninja Photo site - or any oter sub-directory reflecting your album's name), and inside of that directory, create an <span class='inlinecode'>incoming</span> directory. The <span class='inlinecode'>incoming</span> directory. Copy all photos to be part of the album there.</span><br />
<br />
<pre>
% mkdir irregular.ninja
% cd irregular.ninja
% # cp -Rpv ~/Photos/your-photos ./incoming
</pre>
<br />
<span>In this example, I am skipping the <span class='inlinecode'>cp ...</span> part as I intend to use an alternative <span class='inlinecode'>incoming</span> directory, as you will see later in the configuration file.</span><br />
<br />
<span>The general usage of <span class='inlinecode'>potoalbum</span> is as follows:</span><br />
<br />
<pre>
photoalbum clean|generate|version [rcfile] photoalbum
photoalbum makemake
</pre>
<br />
<span>Whereas:</span><br />
<br />
<ul>
<li><span class='inlinecode'>clean</span>: Cleans up the workspace</li>
<li><span class='inlinecode'>generate</span>: Generates the static photo album</li>
<li><span class='inlinecode'>version</span>: Prints out the version</li>
<li><span class='inlinecode'>makemake</span>: Creates a <span class='inlinecode'>Makefile</span> and <span class='inlinecode'>photoalbumrc</span> in the current working directory.</li>
</ul><br />
<span>So what we will do next is to run the following inside of the <span class='inlinecode'>irregular.ninja/</span> directory; it will generate a <span class='inlinecode'>Makefile</span> and a configuration file <span class='inlinecode'>photoalbumrc</span> containing a few configurable options:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> photoalbum makemake</font>
<font color="#ff0000">You may now customize </font><font color="#F3E651">.</font><font color="#ff0000">/photoalbumrc and run make</font>
<font color="#F3E651">%</font><font color="#ff0000"> cat Makefile</font>
<font color="#ff0000">all</font><font color="#F3E651">:</font>
<font color="#ff0000"> photoalbum generate photoalbumrc</font>
<font color="#ff0000">clean</font><font color="#F3E651">:</font>
<font color="#ff0000"> photoalbum clean photoalbumrc</font>
<font color="#F3E651">%</font><font color="#ff0000"> cat photoalbumrc</font>
<i><font color="#ababab"># The title of the photoalbum</font></i>
<font color="#ff0000">TITLE</font><font color="#F3E651">=</font><font color="#bb00ff">'A simple Photoalbum'</font>
<i><font color="#ababab"># Thumbnail height geometry</font></i>
<font color="#ff0000">THUMBHEIGHT</font><font color="#F3E651">=</font><font color="#bb00ff">300</font>
<i><font color="#ababab"># Normal geometry height (when viewing photo). Uncomment, to keep original size.</font></i>
<font color="#ff0000">HEIGHT</font><font color="#F3E651">=</font><font color="#bb00ff">1200</font>
<i><font color="#ababab"># Max previews per page.</font></i>
<font color="#ff0000">MAXPREVIEWS</font><font color="#F3E651">=</font><font color="#bb00ff">40</font>
<i><font color="#ababab"># Randomly shuffle all previews.</font></i>
<i><font color="#ababab"># SHUFFLE=yes</font></i>
<i><font color="#ababab"># Diverse directories, need to be full paths, not relative!</font></i>
<font color="#ff0000">INCOMING_DIR</font><font color="#F3E651">=</font><font color="#ff0000">$(pwd)</font><font color="#ff0000">/incoming</font>
<font color="#ff0000">DIST_DIR</font><font color="#F3E651">=</font><font color="#ff0000">$(pwd)</font><font color="#ff0000">/dist</font>
<font color="#ff0000">TEMPLATE_DIR</font><font color="#F3E651">=</font><font color="#ff0000">/usr/share/photoalbum/templates/default</font>
<i><font color="#ababab">#TEMPLATE_DIR=/usr/share/photoalbum/templates/minimal</font></i>
<i><font color="#ababab"># Includes a .tar of the incoming dir in the dist, can be yes or no</font></i>
<font color="#ff0000">TARBALL_INCLUDE</font><font color="#F3E651">=</font><font color="#ff0000">yes</font>
<font color="#ff0000">TARBALL_SUFFIX</font><font color="#F3E651">=.</font><font color="#ff0000">tar</font>
<font color="#ff0000">TAR_OPTS</font><font color="#F3E651">=</font><font color="#bb00ff">'-c'</font>
<i><font color="#ababab"># Some debugging options</font></i>
<i><font color="#ababab">#set -e</font></i>
<i><font color="#ababab">#set -x</font></i>
</pre>
<br />
<span>In the case for <span class='inlinecode'>irregular.ninja</span>, I changed the defaults to the following:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">--- photoalbumrc 2023-10-29 21:42:00.894202045 +0200</font>
<font color="#ff0000">+++ photoalbumrc.new 2023-06-04 10:40:08.030994440 +0300</font>
<font color="#ff0000">@@ -1,23 +1,24 @@</font>
<font color="#ff0000"> # The title of the photoalbum</font>
<font color="#ff0000">-TITLE='A simple Photoalbum'</font>
<font color="#ff0000">+TITLE='Irregular.Ninja'</font>
<font color="#ff0000"> # Thumbnail height geometry</font>
<font color="#ff0000">-THUMBHEIGHT=300</font>
<font color="#ff0000">+THUMBHEIGHT=400</font>
<font color="#ff0000"> # Normal geometry height (when viewing photo). Uncomment, to keep original size.</font>
<font color="#ff0000">-HEIGHT=1200</font>
<font color="#ff0000">+HEIGHT=1800</font>
<font color="#ff0000"> # Max previews per page.</font>
<font color="#ff0000"> MAXPREVIEWS=40</font>
<font color="#ff0000">-# Randomly shuffle all previews.</font>
<font color="#ff0000">-# SHUFFLE=yes</font>
<font color="#ff0000">+# Randomly shuffle</font>
<font color="#ff0000">+SHUFFLE=yes</font>
<font color="#ff0000"> # Diverse directories, need to be full paths, not relative!</font>
<font color="#ff0000">-INCOMING_DIR=$(pwd)/incoming</font>
<font color="#ff0000">+INCOMING_DIR=~/Nextcloud/Photos/irregular.ninja</font>
<font color="#ff0000"> DIST_DIR=$(pwd)/dist</font>
<font color="#ff0000"> TEMPLATE_DIR=/usr/share/photoalbum/templates/default</font>
<font color="#ff0000"> #TEMPLATE_DIR=/usr/share/photoalbum/templates/minimal</font>
<font color="#ff0000"> # Includes a .tar of the incoming dir in the dist, can be yes or no</font>
<font color="#ff0000">-TARBALL_INCLUDE=yes</font>
<font color="#ff0000">+TARBALL_INCLUDE=no</font>
<font color="#ff0000"> TARBALL_SUFFIX=.tar</font>
<font color="#ff0000"> TAR_OPTS='-c'</font>
</pre>
<br />
<span>So I changed the album title, adjusted some image and thumbnail dimensions, and I want all images to be randomly shuffled every time the album is generated! I also have all my photos in my Nextcloud Photo directory and don't want to copy them to the local <span class='inlinecode'>incoming</span> directory. Also, a tarball containing the whole album as a download isn't provided.</span><br />
<br />
<h2 style='display: inline' id='Generatingthestaticphotoalbum'>Generating the static photo album</h2><br />
<br />
<span>Let's generate it. Depending on the image sizes and count, the following step may take a while. </span><br />
<br />
<pre>
% make
photoalbum generate photoalbumrc
Processing 1055079_cool-water-wallpapers-hd-hd-desktop-wal.jpg to /home/paul/irregular.ninja/dist/photos/1055079_cool-water-wallpapers-hd-hd-desktop-wal.jpg
Processing 11271242324.jpg to /home/paul/irregular.ninja/dist/photos/11271242324.jpg
Processing 11271306683.jpg to /home/paul/irregular.ninja/dist/photos/11271306683.jpg
Processing 13950707932.jpg to /home/paul/irregular.ninja/dist/photos/13950707932.jpg
Processing 14077406487.jpg to /home/paul/irregular.ninja/dist/photos/14077406487.jpg
Processing 14859380100.jpg to /home/paul/irregular.ninja/dist/photos/14859380100.jpg
Processing 14869239578.jpg to /home/paul/irregular.ninja/dist/photos/14869239578.jpg
Processing 14879132910.jpg to /home/paul/irregular.ninja/dist/photos/14879132910.jpg
.
.
.
Generating /home/paul/irregular.ninja/dist/html/7-4.html
Creating thumb /home/paul/irregular.ninja/dist/thumbs/20211130_091051.jpg
Creating blur /home/paul/irregular.ninja/dist/blurs/20211130_091051.jpg
Generating /home/paul/irregular.ninja/dist/html/page-7.html
Generating /home/paul/irregular.ninja/dist/html/7-5.html
Generating /home/paul/irregular.ninja/dist/html/7-5.html
Generating /home/paul/irregular.ninja/dist/html/7-5.html
Creating thumb /home/paul/irregular.ninja/dist/thumbs/DSCF0188.JPG
Creating blur /home/paul/irregular.ninja/dist/blurs/DSCF0188.JPG
Generating /home/paul/irregular.ninja/dist/html/page-7.html
Generating /home/paul/irregular.ninja/dist/html/7-6.html
Generating /home/paul/irregular.ninja/dist/html/7-6.html
Generating /home/paul/irregular.ninja/dist/html/7-6.html
Creating thumb /home/paul/irregular.ninja/dist/thumbs/P3500897-01.jpg
Creating blur /home/paul/irregular.ninja/dist/blurs/P3500897-01.jpg
.
.
.
Generating /home/paul/irregular.ninja/dist/html/8-0.html
Generating /home/paul/irregular.ninja/dist/html/8-41.html
Generating /home/paul/irregular.ninja/dist/html/9-0.html
Generating /home/paul/irregular.ninja/dist/html/9-41.html
Generating /home/paul/irregular.ninja/dist/html/index.html
Generating /home/paul/irregular.ninja/dist/.//index.html
</pre>
<br />
<span>The result will be in the distribution directory <span class='inlinecode'>./dist</span>. This directory is publishable to the inter-web:</span><br />
<br />
<pre>
% ls ./dist
blurs html index.html photos thumbs
</pre>
<br />
<span>I usually do that via rsync to my web server (I use OpenBSD with the standard <span class='inlinecode'>httpd</span> web server, btw.), which is as simple as:</span><br />
<br />
<pre>
% rsync --delete -av ./dist/. admin@blowfish.buetow.org:/var/www/htdocs/irregular.ninja/
</pre>
<br />
<span>Have a look at the end result here:</span><br />
<br />
<a class='textlink' href='https://irregular.ninja'>https://irregular.ninja</a><br />
<br />
<span class='quote'>PS: There's also a server-side synchronisation script mirroring the same content to another server for high availability reasons (out of scope for this blog post).</span><br />
<br />
<h2 style='display: inline' id='Cleaningitup'>Cleaning it up</h2><br />
<br />
<span>A simple <span class='inlinecode'>make clean</span> will clean up the <span class='inlinecode'>./dist</span> directory and all other (if any) temp files created.</span><br />
<br />
<h2 style='display: inline' id='HTMLtemplates'>HTML templates</h2><br />
<br />
<span>Poke around in this source directory. You will find a bunch of Bash-HTML template files. You could tweak them to your liking. </span><br />
<br />
<h2 style='display: inline' id='Conclusion'>Conclusion</h2><br />
<br />
<span>A decent looking (in my opinion, at least) in less than 500 (273 as of this writing, to be precise) lines of Bash code and with minimal dependencies; what more do you want? How many LOCs would this be in Raku with the same functionality (can it be sub-100?). </span><br />
<br />
<span>Also, I like the CSS effects which I recently added. In particular, for the Irregular Ninja site, I randomly shuffled the CSS effects you see. The background blur images are the same but rotated 180 degrees and blurred out.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/photoalbum'><span class='inlinecode'>photoalbum.sh</span> source code on Codeberg.</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other Bash and KISS-related posts are:</span><br />
<br />
<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>2021-05-16 Personal Bash coding style guide</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2021-09-12-keep-it-simple-and-stupid.html'>2021-09-12 Keep it simple and stupid</a><br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br />
<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span> (You are currently reading this)</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>DTail usage examples</title>
<link href="gemini://foo.zone/gemfeed/2023-09-25-dtail-usage-examples.gmi" />
<id>gemini://foo.zone/gemfeed/2023-09-25-dtail-usage-examples.gmi</id>
<updated>2023-09-25T14:57:42+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Hey there. As I am pretty busy this month personally (I am now on Paternity Leave) and as I still want to post once monthly, the blog post of this month will only be some DTail usage examples. They're from the DTail documentation, but not all readers of my blog may be aware of those!</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='DTailusageexamples'>DTail usage examples</h1><br />
<br />
<span class='quote'>Published at 2023-09-25T14:57:42+03:00</span><br />
<br />
<span>Hey there. As I am pretty busy this month personally (I am now on Paternity Leave) and as I still want to post once monthly, the blog post of this month will only be some DTail usage examples. They're from the DTail documentation, but not all readers of my blog may be aware of those!</span><br />
<br />
<span>DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go.</span><br />
<br />
<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br />
<br />
<pre>
,_---~~~~~----._
_,,_,*^____ _____``*g*\"*,
____ _____ _ _ / __/ /' ^. / \ ^@q f
| _ \_ _|_ _(_) | @f | ((@| |@)) l 0 _/
| | | || |/ _` | | | \`/ \~____ / __ \_____/ \
| |_| || | (_| | | | | _l__l_ I
|____/ |_|\__,_|_|_| } [______] I
] | | | |
] ~ ~ |
| Let's tail those logs! |
| |
</pre>
<br />
<span>DTail consists out of a server and several client binaries. In this post, I am showcasing their use!</span><br />
<br />
<ul>
<li>Use <span class='inlinecode'>dtail</span> to follow logs</li>
<li>Use <span class='inlinecode'>dtail</span> to aggregate logs while they are followed</li>
<li>Use <span class='inlinecode'>dcat</span> to display logs and other text files already written</li>
<li>Use <span class='inlinecode'>dgrep</span> to grep (search) logs and other text files already written</li>
<li>Use <span class='inlinecode'>dmap</span> to aggregate logs and other text files already written</li>
<li><span class='inlinecode'>dserver</span> is the DTail server, where all the clients can connect to</li>
</ul><br />
<h2 style='display: inline' id='Followinglogs'>Following logs</h2><br />
<br />
<span>The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string <span class='inlinecode'>INFO</span>. Any other Go compatible regular expression can also be used instead of <span class='inlinecode'>INFO</span>.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt --grep INFO --files </font><font color="#bb00ff">"/var/log/dserver/*.log"</font>
</pre>
<br />
<span>Hint: you can also provide a comma separated server list, e.g.: <span class='inlinecode'>servers server1.example.org,server2.example.org:PORT,...</span></span><br />
<br />
<a href='./dtail-usage-examples/dtail.gif'><img alt='Tail example' title='Tail example' src='./dtail-usage-examples/dtail.gif' /></a><br />
<br />
<span class='quote'>Hint: You can also use the shorthand version (omitting the <span class='inlinecode'>--files</span>)</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt --grep INFO </font><font color="#bb00ff">"/var/log/dserver/*.log"</font>
</pre>
<br />
<h2 style='display: inline' id='Aggregatinglogs'>Aggregating logs</h2><br />
<br />
<span>To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output.</span><br />
<br />
<span class='quote'>Hint: To run a map-reduce query across log lines written in the past, please use the <span class='inlinecode'>dmap</span> command instead.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">\</font>
<font color="#ff0000"> --files </font><font color="#bb00ff">'/var/log/dserver/*.log'</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> --query </font><font color="#bb00ff">'from STATS select sum($goroutines),sum($cgocalls),</font>
<font color="#bb00ff"> last($time),max(lifetimeConnections)'</font>
</pre>
<br />
<span>Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information.</span><br />
<br />
<a href='./dtail-usage-examples/dtail-map.gif'><img alt='Tail map-reduce example' title='Tail map-reduce example' src='./dtail-usage-examples/dtail-map.gif' /></a><br />
<br />
<span class='quote'>Hint: You can also use the shorthand version:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">\</font>
<font color="#ff0000"> --files </font><font color="#bb00ff">'/var/log/dserver/*.log'</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> </font><font color="#bb00ff">'from STATS select sum($goroutines),sum($cgocalls),</font>
<font color="#bb00ff"> last($time),max(lifetimeConnections)'</font>
</pre>
<br />
<span>Here is another example:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">\</font>
<font color="#ff0000"> --files </font><font color="#bb00ff">'/var/log/dserver/*.log'</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> --query </font><font color="#bb00ff">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font>
<font color="#bb00ff"> lifetimeConnections group by $hostname order by max($cgocalls)'</font>
</pre>
<br />
<a href='./dtail-usage-examples/dtail-map2.gif'><img alt='Tail map-reduce example 2' title='Tail map-reduce example 2' src='./dtail-usage-examples/dtail-map2.gif' /></a><br />
<br />
<span>You can also continuously append the results to a CSV file by adding <span class='inlinecode'>outfile append filename.csv</span> to the query:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">\</font>
<font color="#ff0000"> --files </font><font color="#bb00ff">'/var/log/dserver/*.log'</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> --query </font><font color="#bb00ff">'from STATS select ... outfile append result.csv'</font>
</pre>
<br />
<h2 style='display: inline' id='Howtousedcat'>How to use <span class='inlinecode'>dcat</span></h2><br />
<br />
<span>The following example demonstrates how to cat files (display the full content of the files) on several servers at once.</span><br />
<br />
<span>As you can see in this example, a DTail client also creates a local log file of all received data in <span class='inlinecode'>~/log</span>. You can also use the <span class='inlinecode'>noColor</span> and <span class='inlinecode'>-plain</span> flags (this all also work with other DTail commands than <span class='inlinecode'>dcat</span>).</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dcat --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt --files /etc/hostname</font>
</pre>
<br />
<a href='./dtail-usage-examples/dcat.gif'><img alt='Cat example' title='Cat example' src='./dtail-usage-examples/dcat.gif' /></a><br />
<br />
<span class='quote'>Hint: You can also use the shorthand version:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dcat --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt /etc/hostname</font>
</pre>
<br />
<h2 style='display: inline' id='Howtousedgrep'>How to use <span class='inlinecode'>dgrep</span></h2><br />
<br />
<span>The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in <span class='inlinecode'>/etc/passwd</span>. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the <span class='inlinecode'>-before</span>, <span class='inlinecode'>-after</span> and <span class='inlinecode'>-max</span> flags (see animation).</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dgrep --servers server1</font><font color="#F3E651">.</font><font color="#ff0000">example</font><font color="#F3E651">.</font><font color="#ff0000">org</font><font color="#F3E651">:</font><font color="#bb00ff">2223</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> --files /etc/passwd </font><font color="#F3E651">\</font>
<font color="#ff0000"> --regex nologin</font>
</pre>
<br />
<span>Generally, <span class='inlinecode'>dgrep</span> is also a very useful way to search historic application logs for certain content.</span><br />
<br />
<a href='./dtail-usage-examples/dgrep.gif'><img alt='Grep example' title='Grep example' src='./dtail-usage-examples/dgrep.gif' /></a><br />
<br />
<span class='quote'>Hint: <span class='inlinecode'>-regex</span> is an alias for <span class='inlinecode'>-grep</span>.</span><br />
<br />
<h2 style='display: inline' id='Howtousedmap'>How to use <span class='inlinecode'>dmap</span></h2><br />
<br />
<span>To run a map-reduce aggregation over logs written in the past, the <span class='inlinecode'>dmap</span> command can be used. The following example aggregates all map-reduce fields <span class='inlinecode'>dmap</span> will print interim results every few seconds. You can also write the result to an CSV file by adding <span class='inlinecode'>outfile result.csv</span> to the query.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dmap --servers serverlist</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">\</font>
<font color="#ff0000"> --files </font><font color="#bb00ff">'/var/log/dserver/*.log'</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> --query </font><font color="#bb00ff">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font>
<font color="#bb00ff"> lifetimeConnections group by $hostname order by max($cgocalls)'</font>
</pre>
<br />
<span>Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in <span class='inlinecode'>internal/mapr/logformat</span> or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. </span><br />
<br />
<a href='./dtail-usage-examples/dmap.gif'><img alt='DMap example' title='DMap example' src='./dtail-usage-examples/dmap.gif' /></a><br />
<br />
<h2 style='display: inline' id='HowtousetheDTailserverlessmode'>How to use the DTail serverless mode</h2><br />
<br />
<span>Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files.</span><br />
<br />
<span>The serverless mode does not require any <span class='inlinecode'>dserver</span> up and running and therefore there is no networking/SSH involved. </span><br />
<br />
<span>All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode.</span><br />
<br />
<h3 style='display: inline' id='Serverlessmapreducequery'>Serverless map-reduce query</h3><br />
<br />
<span>The following <span class='inlinecode'>dmap</span> example is the same as the previously shown one, but the difference is that it operates on a local log file directly:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dmap --files /var/log/dserver/dserver</font><font color="#F3E651">.</font><font color="#ff0000">log</font>
<font color="#ff0000"> --query </font><font color="#bb00ff">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font>
<font color="#bb00ff"> lifetimeConnections group by $hostname order by max($cgocalls)'</font>
</pre>
<br />
<span>As a shorthand version the following command can be used:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dmap </font><font color="#bb00ff">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font>
<font color="#bb00ff"> lifetimeConnections group by $hostname order by max($cgocalls)'</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> /var/log/dsever/dserver</font><font color="#F3E651">.</font><font color="#ff0000">log</font>
</pre>
<br />
<span>You can also use a file input pipe as follows:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> cat /var/log/dserver/dserver</font><font color="#F3E651">.</font><font color="#ff0000">log </font><font color="#F3E651">|</font><font color="#ff0000"> </font><font color="#F3E651">\</font>
<font color="#ff0000"> dmap </font><font color="#bb00ff">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font>
<font color="#bb00ff"> lifetimeConnections group by $hostname order by max($cgocalls)'</font>
</pre>
<br />
<h3 style='display: inline' id='AggregatingCSVfiles'>Aggregating CSV files</h3><br />
<br />
<span>In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> cat example</font><font color="#F3E651">.</font><font color="#ff0000">csv</font>
<font color="#ff0000">name</font><font color="#F3E651">,</font><font color="#ff0000">lastname</font><font color="#F3E651">,</font><font color="#ff0000">age</font><font color="#F3E651">,</font><font color="#ff0000">profession</font>
<font color="#ff0000">Michael</font><font color="#F3E651">,</font><font color="#ff0000">Jordan</font><font color="#F3E651">,</font><font color="#bb00ff">40</font><font color="#F3E651">,</font><font color="#ff0000">Basketball player</font>
<font color="#ff0000">Michael</font><font color="#F3E651">,</font><font color="#ff0000">Jackson</font><font color="#F3E651">,</font><font color="#bb00ff">100</font><font color="#F3E651">,</font><font color="#ff0000">Singer</font>
<font color="#ff0000">Albert</font><font color="#F3E651">,</font><font color="#ff0000">Einstein</font><font color="#F3E651">,</font><font color="#bb00ff">200</font><font color="#F3E651">,</font><font color="#ff0000">Physician</font>
<font color="#F3E651">%</font><font color="#ff0000"> dmap --query </font><font color="#bb00ff">'select lastname,name where age > 40 logformat csv outfile result.csv'</font><font color="#ff0000"> example</font><font color="#F3E651">.</font><font color="#ff0000">csv</font>
<font color="#F3E651">%</font><font color="#ff0000"> cat result</font><font color="#F3E651">.</font><font color="#ff0000">csv</font>
<font color="#ff0000">lastname</font><font color="#F3E651">,</font><font color="#ff0000">name</font>
<font color="#ff0000">Jackson</font><font color="#F3E651">,</font><font color="#ff0000">Michael</font>
<font color="#ff0000">Einstein</font><font color="#F3E651">,</font><font color="#ff0000">Albert</font>
</pre>
<br />
<span>DMap can also be used to query and aggregate CSV files from remote servers.</span><br />
<br />
<h3 style='display: inline' id='Otherserverlesscommands'>Other serverless commands</h3><br />
<br />
<span>The serverless mode works transparently with all other DTail commands. Here are some examples:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail /var/log/dserver/dserver</font><font color="#F3E651">.</font><font color="#ff0000">log</font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dtail --logLevel trace /var/log/dserver/dserver</font><font color="#F3E651">.</font><font color="#ff0000">log</font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dcat /etc/passwd</font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dcat --plain /etc/passwd </font><font color="#F3E651">></font><font color="#ff0000"> /etc/test</font>
<i><font color="#ababab"># Should show no differences.</font></i>
<font color="#ff0000">diff /etc/test /etc/passwd </font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dgrep --regex ERROR --files /var/log/dserver/dsever</font><font color="#F3E651">.</font><font color="#ff0000">log</font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> dgrep --before </font><font color="#bb00ff">10</font><font color="#ff0000"> --after </font><font color="#bb00ff">10</font><font color="#ff0000"> --max </font><font color="#bb00ff">10</font><font color="#ff0000"> --grep ERROR /var/log/dserver/dsever</font><font color="#F3E651">.</font><font color="#ff0000">log</font>
</pre>
<br />
<span>Use <span class='inlinecode'>--help</span> for more available options. Or go to the DTail page for more information! Hope you find DTail useful!</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-04-22-dtail-the-distributed-log-tail-program.html'>2021-04-22 DTail - The distributed log tail program</a><br />
<a class='textlink' href='./2022-03-06-the-release-of-dtail-4.0.0.html'>2022-03-06 The release of DTail 4.0.0</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2023-09-25-dtail-usage-examples.html'>2023-09-25 DTail usage examples (You are currently reading this)</a><br />
<br />
<span>I hope you find the tools presented in this post useful!</span><br />
<br />
<span>Paul</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Site Reliability Engineering - Part 1: SRE and Organizational Culture</title>
<link href="gemini://foo.zone/gemfeed/2023-08-18-site-reliability-engineering-part-1.gmi" />
<id>gemini://foo.zone/gemfeed/2023-08-18-site-reliability-engineering-part-1.gmi</id>
<updated>2023-08-18T22:43:47+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>The universe of Site Reliability Engineering (SRE) is like an intricate tapestry woven with diverse technology, culture, and personal grit threads. Site Reliability Engineering is one of the most demanding jobs. With all the facets, it's impossible to get bored. There is always a new challenge to master, and there is always a new technology to tinker with. It's not just technical; it's also about communication, collaboration and teamwork. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='SiteReliabilityEngineeringPart1SREandOrganizationalCulture'>Site Reliability Engineering - Part 1: SRE and Organizational Culture</h1><br />
<br />
<span class='quote'>Published at 2023-08-18T22:43:47+03:00</span><br />
<br />
<span>The universe of Site Reliability Engineering (SRE) is like an intricate tapestry woven with diverse technology, culture, and personal grit threads. Site Reliability Engineering is one of the most demanding jobs. With all the facets, it's impossible to get bored. There is always a new challenge to master, and there is always a new technology to tinker with. It's not just technical; it's also about communication, collaboration and teamwork. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</span><br />
<br />
<a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture (You are currently reading this)</a><br />
<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE</a><br />
<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect</a><br />
<br />
<pre>
▓▓▓▓░░
DC on fire:
▓▓ ▓▓ ▓▓
░░ ░░ ▓▓▓▓ ██ ░░ ▓▓▓▓ ▓▓
▓▓░░░░ ░░ ▓▓▓▓ ▓▓░░ ▓▓▓▓
░░░░ ▓▓▓▓▓▓ ▓▓ ▓▓ ▓▓ ▓▓▓▓▓▓ ▓▓
▓▓░░ ▓▓▒▒▒▒▓▓▓▓ ▓▓ ▓▓▓▓ ▓▓▓▓▓▓ ▓▓▒▒▒▒▓▓▓▓ ▓▓▓▓
██▓▓ ▓▓▒▒░░▒▒▓▓ ▓▓██ ▓▓▓▓▓▓ ▓▓▒▒▓▓ ▓▓▒▒░░▒▒▓▓ ██▓▓▓▓
▓▓▓▓██ ▓▓▒▒░░░░▒▒▓▓ ▓▓▓▓ ▓▓▒▒▒▒▓▓ ▓▓▒▒░░▒▒▓▓██▓▓ ▓▓▒▒░░░░▒▒▓▓ ▓▓▒▒▒▒▓▓
▓▓▒▒▒▒▓▓▓▓▒▒░░▒▒▓▓▓▓▓▓▒▒▒▒▓▓ ▓▓▓▓░░▒▒▓▓ ▓▓▒▒░░▒▒▓▓▒▒▒▒▓▓ ▓▓▒▒░░▒▒▓▓▓▓▓▓▓▓░░▒▒▓▓
▒▒░░▒▒▓▓▓▓▒▒░░▒▒▓▓▓▓▒▒░░▒▒▓▓ ▓▓▒▒░░▒▒▓▓ ▓▓░░░░▒▒▒▒░░░░▒▒██████▒▒░░▒▒██▓▓▓▓▒▒░░▒▒▓▓██
░░░░▒▒▓▓▒▒░░▒▒▓▓▓▓▓▓▒▒░░▒▒▓▓██▒▒░░░░▒▒▓▓ ▓▓▒▒░░▒▒▓▓▒▒▒▒░░▒▒▓▓▓▓▒▒░░▒▒▓▓▓▓▓▓▒▒░░░░▒▒▓▓▓▓
░░░░▒▒▓▓▒▒░░░░▓▓██▒▒░░░░▒▒▓▓██▒▒░░░░▒▒██▓▓▓▓▒▒░░▒▒▓▓▓▓▒▒░░░░▒▒▓▓▒▒░░░░██▓▓▓▓▒▒░░░░▒▒████
▒▒░░▒▒▓▓▓▓░░░░▒▒▓▓▒▒▒▒░░░░▒▒▓▓▓▓▒▒░░░░▒▒▓▓▓▓▒▒░░░░▒▒▓▓▒▒░░▒▒▓▓▓▓▓▓░░░░▒▒▓▓▓▓▓▓▒▒░░░░▒▒▓▓
▒▒░░▒▒▓▓▒▒▒▒░░▒▒██▒▒▒▒░░▒▒▒▒██▒▒▒▒░░░░░░▒▒▓▓▒▒░░░░▒▒▒▒░░░░▒▒████▒▒▒▒░░▒▒██▓▓▒▒▒▒░░░░░░▒▒
░░░░░░▒▒░░░░░░░░▒▒▒▒▒▒░░░░▒▒▒▒▒▒░░░░░░░░▒▒▒▒░░░░░░▒▒▒▒░░░░░░▒▒▒▒░░░░░░░░▒▒▒▒▒▒░░░░░░░░▒▒
░░░░░░░░░░▒▒░░░░░░░░░░░░░░░░░░░░░░░░▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒░░░░░░░░░░░░░░░░░░
</pre>
<br />
<h2 style='display: inline' id='SREandOrganizationalCultureNavigatingtheNexus'>SRE and Organizational Culture: Navigating the Nexus</h2><br />
<br />
<span>At the heart of SRE lies the proactive mindset of "prevention over cure." Traditional IT models focused predominantly on reactive solutions, but SRE mandates a shift towards foresight. By adopting Service Level Indicators (SLIs) and Service Level Objectives (SLOs), teams are equipped with clear metrics and goals that guide them toward ensuring reliability and user satisfaction. They reflect an organisational culture prioritising user experience and constant system alignment with user needs. </span><br />
<br />
<span>Another defining SRE idea concept the "error budget." This ingenious framework accepts that no system is flawless. Failures are inevitable. However, instead of being punitive, the culture here is to accept, learn, and iterate. By providing teams with a "budget" for errors, organisations create an environment where innovation is encouraged, and failures are viewed as learning opportunities.</span><br />
<br />
<span>But SRE isn't just about technology and metrics; it's also human. It challenges the "hero culture" that plagues many IT teams. While individual heroics might occasionally save the day, a sustainable model requires collective expertise. An SRE culture recognises that heroes achieve their best within teams, negating the need for a hero-centric environment. This philosophy promotes a balanced on-call experience, emphasising the importance of trust, ownership, effective communication, and collaboration as cornerstones of team success. I personally have fallen into the hero trap, and know it's unsustainable to be the only go-to person for every arising problem.</span><br />
<br />
<span>Additionally, the SRE model requires good documentation. However, it's essential ensuring that this documentation undergoes the same quality checks as code, reinforcing effective onboarding, training and communication.</span><br />
<br />
<span>Organisations might face a significant challenge when adopting SRE. Some might feel SRE principles counter their goals. They might prioritise feature rollouts over reliability or view SRE practices as cumbersome. Hence, creating an SRE culture often demands patient explanations and showcasing benefits, such as increased release velocity and improved user experience.</span><br />
<br />
<span>Monitoring and observability form another SRE aspect, emphasising the need for high-quality tools to query and analyse data. This ties back to the cultural emphasis on continuous learning and adaptability. SREs, by nature, need to be curious, ready to delve into anomalies, and keen on adopting new tools and practices. </span><br />
<br />
<span>The success of SRE within any organisation depends on the broader acceptance of its principles. It demands a move away from siloed operations, where SRE acts as a bandage on flawed systems, to a model where reliability is everyone's responsibility.</span><br />
<br />
<span>In essence, the integration of SRE principles transcends technical practices. It paves the way for a shift in organisational culture that values proactive prevention, continuous learning, collaboration, and transparent communication. The successful melding of SRE and corporate culture promises not just reliable systems but also a robust, resilient, and progressive work environment.</span><br />
<br />
<span>Organisations with the implementation of SLIs, SLOs and error budgets are already advanced in their SRE journey. It takes a lot of communication, convincing, and patience until that point is reached.</span><br />
<br />
<span>Continue with the second part of this series:</span><br />
<br />
<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Gemtexter 2.1.0 - Let's Gemtext again³</title>
<link href="gemini://foo.zone/gemfeed/2023-07-21-gemtexter-2.1.0-lets-gemtext-again-3.gmi" />
<id>gemini://foo.zone/gemfeed/2023-07-21-gemtexter-2.1.0-lets-gemtext-again-3.gmi</id>
<updated>2023-07-21T10:19:31+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I proudly announce that I've released Gemtexter version `2.1.0`. What is Gemtexter? It's my minimalist static site generator for Gemini Gemtext, HTML and Markdown, written in GNU Bash.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='Gemtexter210LetsGemtextagain'>Gemtexter 2.1.0 - Let's Gemtext again³</h1><br />
<br />
<span class='quote'>Published at 2023-07-21T10:19:31+03:00</span><br />
<br />
<pre>
-=[ typewriters ]=- 1/98
.-------.
.-------. _|~~ ~~ |_
_|~~ ~~ |_ .-------. =(_|_______|_)
=(_|_______|_)= _|~~ ~~ |_ |:::::::::|
|:::::::::| =(_|_______|_) |:::::::[]|
|:::::::[]| |:::::::::| |o=======.|
|o=======.| |:::::::[]| `"""""""""`
jgs `"""""""""` |o=======.|
mod. by Paul Buetow `"""""""""`
</pre>
<br />
<span>I proudly announce that I've released Gemtexter version <span class='inlinecode'>2.1.0</span>. What is Gemtexter? It's my minimalist static site generator for Gemini Gemtext, HTML and Markdown, written in GNU Bash.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/gemtexter'>https://codeberg.org/snonux/gemtexter</a><br />
<br />
<h2 style='display: inline' id='WhyBash'>Why Bash?</h2><br />
<br />
<span>This project is too complex for a Bash script. Writing it in Bash was to try out how maintainable a "larger" Bash script could be. It's still pretty maintainable and helps me try new Bash tricks here and then!</span><br />
<br />
<span>Let's list what's new!</span><br />
<br />
<h2 style='display: inline' id='SwitchtoGPL3license'>Switch to GPL3 license</h2><br />
<br />
<span>Many (almost all) of the tools and commands (GNU Bash, GMU Sed, GNU Date, GNU Grep, GNU Source Highlight) used by <span class='inlinecode'>Gemtexter</span> are licensed under the GPL anyway. So why not use the same? This was an easy switch, as I was the only code contributor so far!</span><br />
<br />
<h2 style='display: inline' id='Sourcecodehighlightingsupport'>Source code highlighting support</h2><br />
<br />
<span>The HTML output now supports source code highlighting, which is pretty neat if your site is about programming. The requirement is to have the <span class='inlinecode'>source-highlight</span> command, which is GNU Source Highlight, to be installed. Once done, you can annotate a bare block with the language to be highlighted. E.g.:</span><br />
<br />
<pre>
```bash
if [ -n "$foo" ]; then
echo "$foo"
fi
```
</pre>
<br />
<span>The result will look like this (you can see the code highlighting only in the Web version, not in the Geminispace version of this site):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> -n </font><font color="#bb00ff">"$foo"</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"$foo"</font>
<b><font color="#ffffff">fi</font></b>
</pre>
<br />
<span>Please run <span class='inlinecode'>source-highlight --lang-list</span> for a list of all supported languages.</span><br />
<br />
<h2 style='display: inline' id='HTMLexactvariant'>HTML exact variant</h2><br />
<br />
<span>Gemtexter is there to convert your Gemini Capsule into other formats, such as HTML and Markdown. An HTML exact variant can now be enabled in the <span class='inlinecode'>gemtexter.conf</span> by adding the line <span class='inlinecode'>declare -rx HTML_VARIANT=exact</span>. The HTML/CSS output changed to reflect a more exact Gemtext appearance and to respect the same spacing as you would see in the Geminispace. </span><br />
<br />
<h2 style='display: inline' id='UseofHackwebfontbydefault'>Use of Hack webfont by default</h2><br />
<br />
<span>The Hack web font is a typeface designed explicitly for source code. It's a derivative of the Bitstream Vera and DejaVu Mono lineage, but it features many improvements and refinements that make it better suited to reading and writing code.</span><br />
<br />
<span>The font has distinctive glyphs for every character, which helps to reduce confusion between similar-looking characters. For example, the characters "0" (zero), "O" (capital o), and "o" (lowercase o), or "1" (one), "l" (lowercase L), and "I" (capital i) all have distinct looks in Hack, making it easier to read and understand code at a glance.</span><br />
<br />
<span>Hack is open-source and freely available for use and modification under the MIT License.</span><br />
<br />
<h2 style='display: inline' id='HTMLMastodonverificationsupport'>HTML Mastodon verification support</h2><br />
<br />
<span>The following link explains how URL verification works in Mastodon:</span><br />
<br />
<a class='textlink' href='https://joinmastodon.org/verification'>https://joinmastodon.org/verification</a><br />
<br />
<span>So we have to hyperlink to the Mastodon profile to be verified and also to include a <span class='inlinecode'>rel='me'</span> into the tag. In order to do that add this to the <span class='inlinecode'>gemtexter.conf</span> (replace the URI to your Mastodon profile accordingly):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">declare</font></b><font color="#ff0000"> -xr </font><font color="#ff0000">MASTODON_URI</font><font color="#F3E651">=</font><font color="#bb00ff">'https://fosstodon.org/@snonux'</font>
</pre>
<br />
<span>and add the following into your <span class='inlinecode'>index.gmi</span>:</span><br />
<br />
<pre>
=> https://fosstodon.org/@snonux Me at Mastodon
</pre>
<br />
<span>The resulting line in the HTML output will be something as follows:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff"><a</font></b><font color="#ff0000"> </font><b><font color="#F35E1E">href</font></b><font color="#F3E651">=</font><font color="#bb00ff">'https://fosstodon.org/@snonux'</font><font color="#ff0000"> </font><b><font color="#F35E1E">rel</font></b><font color="#F3E651">=</font><font color="#bb00ff">'me'</font><b><font color="#ffffff">></font></b><font color="#ff0000">Me at Mastodon</font><b><font color="#ffffff"></a></font></b>
</pre>
<br />
<h2 style='display: inline' id='More'>More</h2><br />
<br />
<span>Additionally, there were a couple of bug fixes, refactorings and overall improvements in the documentation made. </span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-04-24-welcome-to-the-geminispace.html'>2021-04-24 Welcome to the Geminispace</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2022-08-27-gemtexter-1.1.0-lets-gemtext-again.html'>2022-08-27 Gemtexter 1.1.0 - Let's Gemtext again</a><br />
<a class='textlink' href='./2023-03-25-gemtexter-2.0.0-lets-gemtext-again-2.html'>2023-03-25 Gemtexter 2.0.0 - Let's Gemtext again²</a><br />
<a class='textlink' href='./2023-07-21-gemtexter-2.1.0-lets-gemtext-again-3.html'>2023-07-21 Gemtexter 2.1.0 - Let's Gemtext again³ (You are currently reading this)</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>'Software Developmers Career Guide and Soft Skills' book notes</title>
<link href="gemini://foo.zone/gemfeed/2023-07-17-career-guide-and-soft-skills-book-notes.gmi" />
<id>gemini://foo.zone/gemfeed/2023-07-17-career-guide-and-soft-skills-book-notes.gmi</id>
<updated>2023-07-17T04:56:20+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>These notes are of two books by 'John Sommez' I found helpful. I also added some of my own keypoints to it. These notes are mainly for my own use, but you might find them helpful, too.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='SoftwareDevelopmersCareerGuideandSoftSkillsbooknotes'>"Software Developmers Career Guide and Soft Skills" book notes</h1><br />
<br />
<span class='quote'>Published at 2023-07-17T04:56:20+03:00</span><br />
<br />
<span>These notes are of two books by "John Sommez" I found helpful. I also added some of my own keypoints to it. These notes are mainly for my own use, but you might find them helpful, too.</span><br />
<br />
<pre>
,.......... ..........,
,..,' '.' ',..,
,' ,' : ', ',
,' ,' : ', ',
,' ,' : ', ',
,' ,'............., : ,.............', ',
,' '............ '.' ............' ',
'''''''''''''''''';''';''''''''''''''''''
'''
</pre>
<br />
<h1 style='display: inline' id='Improve'>Improve</h1><br />
<br />
<h2 style='display: inline' id='Alwayslearnnewthings'>Always learn new things</h2><br />
<br />
<span>When you learn something new, e.g. a programming language, first gather an overview, learn from multiple sources, play around and learn by doing and not consuming and form your own questions. Don't read too much upfront. A large amount of time is spent in learning technical skills which were never use. You want to have a practical set of skills you are actually using. You need to know 20 percent to get out 80 percent of the results.</span><br />
<br />
<ul>
<li>Learn a technology with a goal, e.g. implement a tool. Practice practise practice.</li>
<li>"I know X can do Y, I don't know exactly how, but I can look it up."</li>
<li>Read what experts are writing, for example follow blogs. Stay up to date and spent half an hour per day trading blogs and books.</li>
<li>Pick an open source application, read the code and try to understand it to get a feel of the syntax of the programming language.</li>
<li>Understand, that the standard library makes you a much better programmer.</li>
<li>Self learning is the top skill a programmer can have and is also useful in other aspects in your life.</li>
<li>Keep learning skills every day. Code every day. Don't be overconfident for job security. Read blogs, read books.</li>
<li>If you want to learn, then do it by exploring. Also teach what you learned (for example write a blog post or hold a presentation).</li>
</ul><br />
<span>Fake it until you make it. But be honest about your abilities or lack of. There is however only time between now and until you make it. Refer to your abilities to learn.</span><br />
<br />
<span>Boot camps: The advantage of a boot camp is to pragmatically learn things fast. We almost always overestimate what we can do in a day. Especially during boot camps. Connect to others during the boot camps</span><br />
<br />
<h2 style='display: inline' id='Setgoals'>Set goals</h2><br />
<br />
<span>Your own goals are important but the manager also looks at how the team performs and how someone can help the team perform better. Check whether you are on track with your goals every 2 weeks in order to avoid surprises for the annual review. Make concrete goals for next review. Track and document your progress. Invest in your education. Make your goals known. If you want something, then ask for it. Nobody but you knows what you want.</span><br />
<br />
<h2 style='display: inline' id='Ratings'>Ratings</h2><br />
<br />
<span>That's a trap: If you have to rate yourself, that's a trap. That never works in an unbiased way. Rate yourself always the best way but rate your weakest part as high as possible minus one point. Rate yourself as good as you can otherwise. Nobody is putting for fun a gun on his own head. </span><br />
<br />
<ul>
<li>Don't do peer rating, it can fire back on you. What if the colleague becomes your new boss?</li>
<li>Cooperate rankings are unfortunately HR guidelines and politics and only mirror a little your actual performance.</li>
</ul><br />
<h2 style='display: inline' id='Promotions'>Promotions</h2><br />
<br />
<span>The most valuable employees are the ones who make themselves obsolete and automate all away. Keep a safety net of 3 to 6 months of finances. Safe at least 10 percent of your earnings. Also, if you make money it does not mean that you have to spent more money. Is a new car better than a used car which both can bring you from A to B? Liability vs assets.</span><br />
<br />
<ul>
<li>Raise or promotion, what's better? Promotion is better as money will follow anyway then.</li>
<li>Take projects no-one wants and make them shine. A promotion will follow.</li>
<li>A promotion is not going to come to you because you deserve it. You have to hunt and ask for it.</li>
<li>Track all kudos (e.g. ask for emails from your colleagues).</li>
<li>Big corporations HRs don't expect a figjit. That's why it's so important to keep track of your accomplishments and kudos'.</li>
<li>If you want a raise be specific how much and know to back your demands. Don't make a thread and no ultimatums.</li>
<li>Best way for a promotion is to switch jobs. You can even switch back with a better salary.</li>
</ul><br />
<h2 style='display: inline' id='Finishthings'>Finish things</h2><br />
<br />
<span>Hard work is necessary for accomplish results. However, work smarter not harder. Furthermore, working smart is not a substitute for working hard. Work both, hard and smart.</span><br />
<br />
<ul>
<li>Learn to finish things without motivation. Things will pay off when you stick to stuff and eventually motivation can also come back.</li>
<li>You will fail if you don't plan realistically. Set also a schedule and follow to it as of life depends on it.</li>
<li>Advances come only of you give more than asked. Consistency, commitment and knowing what you need to do is more key than hard work.</li>
<li>Any action is better than no action. If you get stuck you have gained nothing.</li>
<li>You need to know the unknowns. Identify as many unknown not known things as possible. </li>
</ul><br />
<span>Hard vs fun: Both engage the brain (video games vs work). Some work is hard and other is easy. Hard work is boring. The harsh truth is you have to put in hard and boring work in order to accomplish and be successful. Work won't be always boring though, as joy will follow with mastery.</span><br />
<br />
<span>Defeat is finally give up. Failure is the road to success, embrace it. Failure does not define you but how you respond to it. Events don't make your unhappy, but how you react to events do.</span><br />
<br />
<h2 style='display: inline' id='Expandtheempire'>Expand the empire</h2><br />
<br />
<span>The larger your empire is, the larger your circle of influence is. The larger the circle of influence is, the more opportunities you have.</span><br />
<br />
<ul>
<li>Do the dirty work if you want to expand the empire. That's there the opportunities are.</li>
<li>SCRUM often fails due to the lack to commitment. The backlog just becomes a wish to get completed.</li>
<li>Apply work on your quality standards. Don't cross the line of compromise. Always improve your skills. Never be happy being good enough.</li>
</ul><br />
<span>Become visible, keep track that you accomplishments. E.g. write a weekly summary. Do presentations, be seen. Learn new things and share your learnings. Be the problem solver and not the blamer.</span><br />
<br />
<h2 style='display: inline' id='Bepragmaticandalsomanageyourtime'>Be pragmatic and also manage your time</h2><br />
<br />
<span>Make use of time boxing via the Pomodoro technique: Set a target of rounds and track the rounds. That give you exact focused work time. That's really the trick. For example set a goal of 6 daily pomodores.</span><br />
<br />
<ul>
<li>Every time you do something question why does it make sense be pragmatic and don't follow because it is best practice.</li>
<li>You can also apply the time boxing technique (Cal Newport) for focused deep work.</li>
</ul><br />
<span>You should feel good of the work done even if you don't finished the task. You will feel good about pomodoro wise even you don't finish the task on hand yet. Helps you to enjoy time off more. Working longer may not sell anything.</span><br />
<br />
<h3 style='display: inline' id='Thequotasystem'>The quota system</h3><br />
<br />
<span>Defined quota of things done. E.g. N runs per week or M Blog posts per month or O pomodoros per week. This helps with consistency. Truly commit to these quotas. Failure is not an option. Start with small commitments. Don't commit to something you can't fulfill otherwise you set yourself up for failure.</span><br />
<br />
<ul>
<li>Why does the quota System work? Slow and consistent pace is the key. It also overcomes willpower weaknesses as goals are preset.</li>
<li>Internal motivation is more important over external motivation. Check out Daniels book drive.</li>
<li>Multitasking: Batching is effective. E.g. emails twice daily at pre-set times..</li>
</ul><br />
<h3 style='display: inline' id='Dontwastetime'>Don't waste time</h3><br />
<br />
<span>The biggest time waster is TV watching. The TV is programming you. It's insane that Americans watch so much TV as they work full time. Schedule one show at a time and watch it when you want to watch it. Most movies are crap anyways. The good movies will come to you as people will talk about them.</span><br />
<br />
<ul>
<li>Social media is time waster as well. Schedule your Social Media times. For example be on Facebook only for max one hour on Saturdays.</li>
<li>Meetings can waste time as well. Simply don't go to them. Try to cancel meeting if it can be dealt with via email.</li>
<li>Enjoying things is not a waste of time. E.g. you could still play a game once in a while. It is important not to cut away all you enjoy from your life.</li>
</ul><br />
<h3 style='display: inline' id='Habits'>Habits</h3><br />
<br />
<span>Try to have as many good habits as possible. Start with easy habits, and make them a little bit more challenging over time. Set ankers and rewards. Over time the routines will become habits naturally.</span><br />
<br />
<span>Habit stacking is effective, which is combining multiple habits at the same time. For example you can workout on a circular trainer while while watching a learning video on O'Reilly Safari Online while getting closer to your weekly step goal.</span><br />
<br />
<ul>
<li>We don't have control over our habits but our own routines.</li>
<li>Routines help to form the habits, though.</li>
</ul><br />
<h1 style='display: inline' id='Worklifebalance'>Work-life balance</h1><br />
<br />
<span>Avoid overwork hours. That's not as beneficial as you might think and comes only with very small rewards. Invest rather in yourself and not in your employer.</span><br />
<br />
<ul>
<li>Work-life balance is a myth. Make it so that you enjoy work and your personal life and not just personal life.</li>
<li>Maintain fewer but good relationships. As a reward, better and integrated your life will be.</li>
<li>Life in the present Moment. Make the best of every moment of your life.</li>
<li>Enjoy every aspect of your life. If you want to take away one thing from this book that is it.</li>
</ul><br />
<span>Use your most productive hours to work on you. Make that your priority. Take care of yourself a priority (E.g. do workouts or learn a new language). You can always workout 2 or 1 hour per day, but will you pay the price?</span><br />
<br />
<h2 style='display: inline' id='Mentalhealth'>Mental health</h2><br />
<br />
<ul>
<li>Friendships and positive thinking help to have and maintain better health, longer Life, better productivity and increased happiness.</li>
<li>Positive thinking can be trained and be a habit. Read the book "The Power of Positive Thinking".</li>
<li>Stoicism helps. Meditation helps. Playing for fun helps too.</li>
</ul><br />
<span>Become the person you want to become (your self image). Program your brain unconsciously. Don't become the person other people want you to be. Embrace yourself, you are you.</span><br />
<br />
<span>In most cases burnout is just an illusion. If you don't have motivation push through the wall. People usually don't pass the wall as they feel they are burned out. After pushing through the wall you will have the most fun, for example you will be able playing the guitar greatly.</span><br />
<br />
<h2 style='display: inline' id='Physicalhealth'>Physical health</h2><br />
<br />
<span>Utilise a standing desk and treadmill (you could walk and type at the same time). Increase the incline in order to burn more calories. Even on the standing desk you burn more calories than sitting. When you use pomodoro then you can use the small breaks for push-ups (maybe won't do as good when you are in a fasted state).</span><br />
<br />
<ul>
<li>You can only do one thing, lose fat or gain muscles. Not both at the same time.</li>
<li>Train your strength by heavy lifting, but only with a very few repetitions (e.g. 5 max for each exercise, everything over this is body building).</li>
<li>If you want to increase the muscle mass use medium weights but lift them more often. If you want to increase your endurance lift light weights but with even more reps.</li>
<li>Avoid highly processed foods</li>
</ul><br />
<span>Intermittent fasting is an effective method to maintain weight and health. But it does not mean that you can only eat junk food in the feeding windows. Also, diet and nutrition is the most important for health and fitness. They make it also easier to stay focused and positive.</span><br />
<br />
<h2 style='display: inline' id='Nodrama'>No drama</h2><br />
<br />
<span>Avoid drama at work. Where are humans there is drama. You can decide where to spent your energy in. But don't avoid conflict. Conflict is healthy in any kind of relationship. Be tactful and state your opinion. The goal is to find the best solution to the problem.</span><br />
<br />
<span>Don't worry about other people what they do and don't do. You only worry about you. Shut up and get your own things done. But you could help to inspire a not working colleague.</span><br />
<br />
<ul>
<li>During an argument, take the opponent's position and see how your opinion changes.</li>
<li>If you they to convince someone else it's an argument. Of you try to find the best solution it is a good resolution.</li>
<li>If someone is hurting the team let the manager know but phrase it nicely.</li>
<li>How to get rid of a never ending talking person? Set up focus hours officially where you don't want to be interrupted. Present as if it is your defect that you get interrupted easily.</li>
<li>TOXIC PEOPLE: AVOID THEM. RUN.</li>
<li>Boss likes if you get shit done without getting asked all the time about things and also without drama.</li>
</ul><br />
<span>You have to learn how to work in a team. Be honest but tactful. It's not too be the loudest but about selling your ideas. Don't argue otherwise you won't sell anything. Be persuasive by finding the common ground. Or lead the colleagues to your idea and don't sell it upfront. Communicate clearly.</span><br />
<br />
<h1 style='display: inline' id='Personalbrand'>Personal brand</h1><br />
<br />
<ul>
<li>Invest your value outside the company. Build your personal brand. Show how valuable you are, also to other companies. Become an asset.</li>
<li>Invest in your education. Make your goals known. If you want something ask for it (see also the sections about goals in this document).</li>
</ul><br />
<h2 style='display: inline' id='Marketyourself'>Market yourself</h2><br />
<br />
<ul>
<li>The best way to market yourself is to make you usable.</li>
<li>Create a brand. Decide your focus. Throw your name out as often as possible.</li>
</ul><br />
<span>Have a blog. Schedule your posts. Consistency beats every other factor. E.g. post once a month a new post. Find your voice, you don't have to sound academic. Keep writing, if you keep it long enough the rewards will be coming. Your own blog can take 5 years to take off. Most people give up too soon.</span><br />
<br />
<ul>
<li>Consistency of your blog is key. Also write quality content. Don't try to be a man of success but try to be a man of value.</li>
<li>Have an elevator pitch: "buetow.org - Having fun with computers!"</li>
<li>Have social media accounts, especially the ones which are more tech related.</li>
</ul><br />
<h2 style='display: inline' id='Networking'>Networking</h2><br />
<br />
<span>Ask people so they talk about themselves. They are not really interested in you. Use meetup.com to find groups you are interested and build up the network over time. Don't drink on social networking events even when others do. Talking to other people at events only has upsides. Just saying "hi" and introducing yourself is enough. What worse can happen? If the person rejects you so what, life goes on. Ask open questions and no "yes" and "no" questions. E.g.: "What is your story, why are you here?".</span><br />
<br />
<h2 style='display: inline' id='Publicspeaking'>Public speaking</h2><br />
<br />
<span>Before your talk go on stage 10 minutes in advance. Introduce yourself to the front row people. During the talk they will smile at you and encourage you during your talk.</span><br />
<br />
<ul>
<li>Try at least 5 times before giving up public speaking. You can also start small, e.g. present a topic at work you are learning.</li>
<li>Practise your talk and timing. You can also record your practicing.</li>
</ul><br />
<span>Just do it. Just go to conferences. Even if you are not speaking. Sell your boss what you would learn and "this and that" and you would present the learnings to the team afterwards.</span><br />
<br />
<h1 style='display: inline' id='Newjob'>New job</h1><br />
<br />
<h2 style='display: inline' id='Fortheinterview'>For the interview</h2><br />
<br />
<ul>
<li>Build up a network before the interview. E.g., follow and comment blogs. Or go to meet-ups and conferences. Join user groups.</li>
<li>Ask to touch base before the real interview and ask questions about the company. Do "pre-interviews".</li>
<li>Have a blog, a CV can only be 2 pages and an interview only can last only 2 hours. A blog helps you also to be a better communicator.</li>
</ul><br />
<span>If you are specialized then there is a better chance to get a fitting job. No one will hire a general lawyer if there are specialized lawyers available. Even if you are specialized, you will have a wide range of skills (T-shape knowledge).</span><br />
<br />
<h2 style='display: inline' id='Findtherighttypeofcompany'>Find the right type of company</h2><br />
<br />
<span>Not all companies are equal. They have individual cultures and guidelines.</span><br />
<br />
<ul>
<li>Startup: dynamic and larger impact. Many hats on.</li>
<li>Medium size companies: most stable ones. Not cutting edge technologies. No crazy working hours.</li>
<li>Large company: very established with a lot of structure however constant layoffs and restructurings. Less impact you can have. Complex politics.</li>
<li>Working for yourself: This is harder than you think, probably much harder.</li>
</ul><br />
<span>Work in a tech. company if you want to work on/with cutting edge technologies.</span><br />
<br />
<h2 style='display: inline' id='Applyforthenewjob'>Apply for the new job</h2><br />
<br />
<span>Get a professional resume writer. Get referrals of writers and get samples from there. Get sufficient with algorithm and data structures interview questions. Cracking the coding interview book and blog </span><br />
<br />
<ul>
<li>Apply for each job with a specialised CV each. Each CV fits the job better.</li>
<li>Best get a job via a personal referral or inbound marketing. The latter is somehow rare.</li>
<li>Inbound marketing is for example someone responds to your blog and offers you a job.</li>
<li>Interview the interviewer. Be persistent.</li>
<li>Create creative looking resumes, see simple programmer website. Action-result style for a resume.</li>
</ul><br />
<span>Invest in your dress code as appearance masters. It does make sense to invest in your style. You could even hire a professional stylist (not my personal way though).</span><br />
<br />
<h2 style='display: inline' id='Negotiation'>Negotiation</h2><br />
<br />
<ul>
<li>Whoever names the number first loses. You don't know what someone else is expecting unless told. Low ball number may be an issue but you have to know the market.</li>
<li>Salary is not about what you need but what you are worth. Try to find out what you are worth.</li>
<li>Big tech companies have a pay scale. You can ask for this.</li>
<li>Don't tell your current salary. Only do one counter offer and say "If you do X then I commit today". Be tactful and not rude. Nobody wants to be taken advantage of. Also, don't be arrogant.</li>
<li>If the company wants to know your range, respond: "I would rather learn more about the job and compensation. You have a range in mind, correct?" Be brave and just pause here.</li>
<li>Otherwise, if the company refuses then say "if you tell me what the range is and although I am not yet sure yet what are my exact salary requirements are I can see if the range is of what I am looking for. If they absolute refuse give a high ball range you would expect and make it conditional to the overall compensation package. E.g. 70k to 100k depending on the compensation package. THE LOW END SHOULD BE YOUR REAL LOW END. Play a little bit of hardball here and be brave. Practise it.</li>
<li>Put 10 percent on top of the salary range into a counter offer.</li>
<li>Everything is negotiable, not only the salary.</li>
<li>Job markup rate: Check it regarding the recruitment rate negotiation.</li>
<li>Don't make a rushed decision based on deadlines. Make a fairly high counter offer shortly before deadline.</li>
<li>You should also cope with rejections while selling yourself. There is no such thing as job security.</li>
</ul><br />
<ul>
<li>Never spilt the difference is the best book for learning negotiation techniques..</li>
</ul><br />
<h2 style='display: inline' id='Leavingtheoldjob'>Leaving the old job</h2><br />
<br />
<span>When leaving a job make a clean and non personal as possible. Never complain and never explain. Don't worry about abandonment of the team. Everybody is replacement and you make a business decision. Don't threaten to quit as you are replaceable.</span><br />
<br />
<h1 style='display: inline' id='Otherthings'>Other things</h1><br />
<br />
<ul>
<li>As a leader lead by example and don't lead from the Eiffel tower.</li>
<li>As a leader you are responsible for the team. If the team fails then it's your fault only.</li>
</ul><br />
<h2 style='display: inline' id='Testing'>Testing</h2><br />
<br />
<span>Unit testing Vs regression testing: Unit tests test the smallest possible unit and get rewritten if the unit gets changed. It's like programming against a specification n. Regression tests test whether the software still works after the change. Now you know more than most software engineers.</span><br />
<br />
<h2 style='display: inline' id='Bookstoread'>Books to read</h2><br />
<br />
<ul>
<li>Clean Code</li>
<li>Code Complete</li>
<li>Cracking the Interview - Lessons and Solutions.</li>
<li>Daniels Book "Drive" (about internal and external motivation)</li>
<li>God's degree (inventor of Dilbert)</li>
<li>Head first Design Patterns</li>
<li>How to win Friends and influence People</li>
<li>Never Split the Difference [X]</li>
<li>Structure and programming functional programs</li>
<li>The obstacle is the way [X]</li>
<li>The passionate programmer</li>
<li>The Power of Positive Thinking (Highly religious - I personally don't like it)</li>
<li>The Pragmatic Programmer [X]</li>
<li>The war of Art (to combat procrastination)</li>
<li>Willpower Instinct</li>
</ul><br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other book notes of mine are:</span><br />
<br />
<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 "The Pragmatic Programmer" book notes</a><br />
<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 "Never split the difference" book notes</a><br />
<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 "The Obstacle is the Way" book notes</a><br />
<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes (You are currently reading this)</a><br />
<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 "Mind Management" book notes</a><br />
<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 "Slow Productivity" book notes</a><br />
<br />
<a class='textlink' href='../resources.html'>More books and other resources I found useful.</a><br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>KISS server monitoring with Gogios</title>
<link href="gemini://foo.zone/gemfeed/2023-06-01-kiss-server-monitoring-with-gogios.gmi" />
<id>gemini://foo.zone/gemfeed/2023-06-01-kiss-server-monitoring-with-gogios.gmi</id>
<updated>2023-06-01T21:10:17+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Gogios is a minimalistic and easy-to-use monitoring tool I programmed in Google Go designed specifically for small-scale self-hosted servers and virtual machines. The primary purpose of Gogios is to monitor my personal server infrastructure for `foo.zone`, my MTAs, my authoritative DNS servers, my NextCloud, Wallabag and Anki sync server installations, etc.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='KISSservermonitoringwithGogios'>KISS server monitoring with Gogios</h1><br />
<br />
<span class='quote'>Published at 2023-06-01T21:10:17+03:00</span><br />
<br />
<a href='./2023-06-01-kiss-server-monitoring-with-gogios/gogios-small.png'><img alt='Gogios logo' title='Gogios logo' src='./2023-06-01-kiss-server-monitoring-with-gogios/gogios-small.png' /></a><br />
<br />
<h2 style='display: inline' id='Introduction'>Introduction</h2><br />
<br />
<span>Gogios is a minimalistic and easy-to-use monitoring tool I programmed in Google Go designed specifically for small-scale self-hosted servers and virtual machines. The primary purpose of Gogios is to monitor my personal server infrastructure for <span class='inlinecode'>foo.zone</span>, my MTAs, my authoritative DNS servers, my NextCloud, Wallabag and Anki sync server installations, etc.</span><br />
<br />
<span>With compatibility with the Nagios Check API, Gogios offers a simple yet effective solution to monitor a limited number of resources. In theory, Gogios scales to a couple of thousand checks, though. You can clone it from Codeberg here:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/gogios'>https://codeberg.org/snonux/gogios</a><br />
<br />
<pre>
_____________________________ ____________________________
/ \ / \
| _______________________ || ______________________ |
| / \ || / \ |
| | # Alerts with status c| || | # Unhandled alerts: | |
| | hanged: | || | | |
| | | || | CRITICAL: Check Pizza| |
| | OK->CRITICAL: Check Pi| || | : Late delivery | |
| | zza: Late delivery | || | | |
| | | || | WARNING: Check Thirst| |
| | | || | : OutofKombuchaExcept| |
| \_______________________/ || \______________________/ |
| /|\ GOGIOS MONITOR 1 _ || /|\ GOGIOS MONITOR 2 _ |
\_____________________________/ \____________________________/
!_________________________! !________________________!
------------------------------------------------
ASCII art was modified by Paul Buetow
The original can be found at
https://asciiart.website/index.php?art=objects/computers
</pre>
<br />
<h2 style='display: inline' id='Motivation'>Motivation</h2><br />
<br />
<span>With experience in monitoring solutions like Nagios, Icinga, Prometheus and OpsGenie, these tools often came with many features that I didn't necessarily need for personal use. Contact groups, host groups, check clustering, and the requirement of operating a DBMS and a WebUI added complexity and bloat to my monitoring setup.</span><br />
<br />
<span>My primary goal was to have a single email address for notifications and a simple mechanism to periodically execute standard Nagios check scripts and notify me of any state changes. I wanted the most minimalistic monitoring solution possible but wasn't satisfied with the available options.</span><br />
<br />
<span>This led me to create Gogios, a lightweight monitoring tool tailored to my specific needs. I chose the Go programming language for this project as it comes, in my opinion, with the best balance of ease to use and performance.</span><br />
<br />
<h2 style='display: inline' id='Features'>Features</h2><br />
<br />
<ul>
<li>Compatible with Nagios Check scripts: Gogios leverages the widely-used Nagios Check API, allowing to use existing Nagios plugins.</li>
<li>Lightweight and Minimalistic: Gogios is designed to be simple and fairly easy to set up.</li>
<li>Configurable Check Timeout and Concurrency: Gogios allows you to set a timeout for checks and configure the number of concurrent checks, offering flexibility in monitoring your resources.</li>
<li>Configurable check dependency: A check can depend on another check, which enables scenarios like not executing an HTTP check when the server isn't pingable.</li>
<li>Retries: Check retry and retry intervals are configurable per check.</li>
<li>Email Notifications: Gogios can send email notifications regarding the status of monitored services, ensuring you stay informed about potential issues.</li>
<li>CRON-based Execution: Gogios can be quickly scheduled to run periodically via CRON, allowing you to automate monitoring without needing a complex setup.</li>
</ul><br />
<h2 style='display: inline' id='Examplealert'>Example alert</h2><br />
<br />
<span>This is an example alert report received via E-Mail. Whereas, <span class='inlinecode'>[C:2 W:0 U:0 OK:51]</span> means that we've got two alerts in status critical, 0 warnings, 0 unknowns and 51 OKs.</span><br />
<br />
<pre>
Subject: GOGIOS Report [C:2 W:0 U:0 OK:51]
This is the recent Gogios report!
# Alerts with status changed:
OK->CRITICAL: Check ICMP4 vulcan.buetow.org: Check command timed out
OK->CRITICAL: Check ICMP6 vulcan.buetow.org: Check command timed out
# Unhandled alerts:
CRITICAL: Check ICMP4 vulcan.buetow.org: Check command timed out
CRITICAL: Check ICMP6 vulcan.buetow.org: Check command timed out
Have a nice day!
</pre>
<br />
<h2 style='display: inline' id='Installation'>Installation</h2><br />
<br />
<h3 style='display: inline' id='CompilingandinstallingGogios'>Compiling and installing Gogios</h3><br />
<br />
<span>This document is primarily written for OpenBSD, but applying the corresponding steps to any Unix-like (e.g. Linux-based) operating system should be easy. On systems other than OpenBSD, you may always have to replace <span class='inlinecode'>does</span> with the <span class='inlinecode'>sudo</span> command and replace the <span class='inlinecode'>/usr/local/bin</span> path with <span class='inlinecode'>/usr/bin</span>.</span><br />
<br />
<span>To compile and install Gogios on OpenBSD, follow these steps:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">git clone https</font><font color="#F3E651">:</font><font color="#ff0000">//codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/gogios</font><font color="#F3E651">.</font><font color="#ff0000">git</font>
<font color="#ff0000">cd gogios</font>
<font color="#ff0000">go build -o gogios cmd/gogios/main</font><font color="#F3E651">.</font><font color="#ff0000">go</font>
<font color="#ff0000">doas cp gogios /usr/local/bin/gogios</font>
<font color="#ff0000">doas chmod </font><font color="#bb00ff">755</font><font color="#ff0000"> /usr/local/bin/gogios</font>
</pre>
<br />
<span>You can use cross-compilation if you want to compile Gogios for OpenBSD on a Linux system without installing the Go compiler on OpenBSD. Follow these steps:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">export</font></b><font color="#ff0000"> </font><font color="#ff0000">GOOS</font><font color="#F3E651">=</font><font color="#ff0000">openbsd</font>
<b><font color="#ffffff">export</font></b><font color="#ff0000"> </font><font color="#ff0000">GOARCH</font><font color="#F3E651">=</font><font color="#ff0000">amd64</font>
<font color="#ff0000">go build -o gogios cmd/gogios/main</font><font color="#F3E651">.</font><font color="#ff0000">go</font>
</pre>
<br />
<span>On your OpenBSD system, copy the binary to <span class='inlinecode'>/usr/local/bin/gogios</span> and set the correct permissions as described in the previous section. All steps described here you could automate with your configuration management system of choice. I use Rexify, the friendly configuration management system, to automate the installation, but that is out of the scope of this document.</span><br />
<br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<br />
<h3 style='display: inline' id='Settingupusergroupanddirectories'>Setting up user, group and directories</h3><br />
<br />
<span>It is best to create a dedicated system user and group for Gogios to ensure proper isolation and security. Here are the steps to create the <span class='inlinecode'>_gogios</span> user and group under OpenBSD:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">doas adduser -group _gogios -batch _gogios</font>
<font color="#ff0000">doas usermod -d /var/run/gogios _gogios</font>
<font color="#ff0000">doas mkdir -p /var/run/gogios</font>
<font color="#ff0000">doas chown _gogios</font><font color="#F3E651">:</font><font color="#ff0000">_gogios /var/run/gogios</font>
<font color="#ff0000">doas chmod </font><font color="#bb00ff">750</font><font color="#ff0000"> /var/run/gogios</font>
</pre>
<br />
<span>Please note that creating a user and group might differ depending on your operating system. For other operating systems, consult their documentation for creating system users and groups.</span><br />
<br />
<h3 style='display: inline' id='Installingmonitoringplugins'>Installing monitoring plugins</h3><br />
<br />
<span>Gogios relies on external Nagios or Icinga monitoring plugin scripts. On OpenBSD, you can install the <span class='inlinecode'>monitoring-plugins</span> package with Gogios. The monitoring-plugins package is a collection of monitoring plugins, similar to Nagios plugins, that can be used to monitor various services and resources:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">doas pkg_add monitoring-plugins</font>
<font color="#ff0000">doas pkg_add nrpe </font><i><font color="#ababab"># If you want to execute checks remotely via NRPE.</font></i>
</pre>
<br />
<span>Once the installation is complete, you can find the monitoring plugins in the <span class='inlinecode'>/usr/local/libexec/nagios</span> directory, which then can be configured to be used in <span class='inlinecode'>gogios.json</span>.</span><br />
<br />
<h2 style='display: inline' id='Configuration'>Configuration</h2><br />
<br />
<h3 style='display: inline' id='MTA'>MTA</h3><br />
<br />
<span>Gogios requires a local Mail Transfer Agent (MTA) such as Postfix or OpenBSD SMTPD running on the same server where the CRON job (see about the CRON job further below) is executed. The local MTA handles email delivery, allowing Gogios to send email notifications to monitor status changes. Before using Gogios, ensure that you have a properly configured MTA installed and running on your server to facilitate the sending of emails. Once the MTA is set up and functioning correctly, Gogios can leverage it to send email notifications.</span><br />
<br />
<span>You can use the mail command to send an email via the command line on OpenBSD. Here's an example of how to send a test email to ensure that your email server is working correctly:</span><br />
<br />
<pre>
echo 'This is a test email from OpenBSD.' | mail -s 'Test Email' your-email@example.com
</pre>
<br />
<span>Check the recipient's inbox to confirm the delivery of the test email. If the email is delivered successfully, it indicates that your email server is configured correctly and functioning. Please check your MTA logs in case of issues.</span><br />
<br />
<h3 style='display: inline' id='ConfiguringGogios'>Configuring Gogios</h3><br />
<br />
<span>To configure Gogios, create a JSON configuration file (e.g., <span class='inlinecode'>/etc/gogios.json</span>). Here's an example configuration:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">EmailTo</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">paul@dev.buetow.org</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">EmailFrom</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">gogios@buetow.org</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">CheckTimeoutS</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">10</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">CheckConcurrency</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">2</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">StateDir</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">/var/run/gogios</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Checks</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Check ICMP4 www.foo.zone</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Plugin</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">/usr/local/libexec/nagios/check_ping</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Args</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-H</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">www.foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-4</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-w</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">50,10%</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-c</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">100,15%</font><font color="#ff0000">"</font><font color="#ff0000"> </font><font color="#F3E651">],</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Retries</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">3</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">RetryInterval</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">10</font>
<font color="#ff0000"> </font><font color="#F3E651">},</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Check ICMP6 www.foo.zone</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Plugin</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">/usr/local/libexec/nagios/check_ping</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Args</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-H</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">www.foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-6</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-w</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">50,10%</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-c</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">100,15%</font><font color="#ff0000">"</font><font color="#ff0000"> </font><font color="#F3E651">],</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Retries</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">3</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">RetryInterval</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">10</font>
<font color="#ff0000"> </font><font color="#F3E651">},</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">www.foo.zone HTTP IPv4</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Plugin</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">/usr/local/libexec/nagios/check_http</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Args</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000">"</font><font color="#ff0000">www.foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-4</font><font color="#ff0000">"</font><font color="#F3E651">],</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">DependsOn</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000">"</font><font color="#ff0000">Check ICMP4 www.foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">]</font>
<font color="#ff0000"> </font><font color="#F3E651">},</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">www.foo.zone HTTP IPv6</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Plugin</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">/usr/local/libexec/nagios/check_http</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Args</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000">"</font><font color="#ff0000">www.foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-6</font><font color="#ff0000">"</font><font color="#F3E651">],</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">DependsOn</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000">"</font><font color="#ff0000">Check ICMP6 www.foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">]</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Check NRPE Disk Usage foo.zone</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Plugin</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">/usr/local/libexec/nagios/check_nrpe</font><font color="#ff0000">"</font><font color="#F3E651">,</font>
<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">Args</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">[</font><font color="#ff0000">"</font><font color="#ff0000">-H</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">foo.zone</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-c</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">check_disk</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-p</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">5666</font><font color="#ff0000">"</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">-4</font><font color="#ff0000">"</font><font color="#F3E651">]</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#F3E651">}</font>
</pre>
<br />
<ul>
<li><span class='inlinecode'>EmailTo</span>: Specifies the recipient of the email notifications.</li>
<li><span class='inlinecode'>EmailFrom</span>: Indicates the sender's email address for email notifications.</li>
<li><span class='inlinecode'>CheckTimeoutS</span>: Sets the timeout for checks in seconds.</li>
<li><span class='inlinecode'>CheckConcurrency</span>: Determines the number of concurrent checks that can run simultaneously.</li>
<li><span class='inlinecode'>StateDir</span>: Specifies the directory where Gogios stores its persistent state in a <span class='inlinecode'>state.json</span> file. </li>
<li><span class='inlinecode'>Checks</span>: Defines a list of checks to be performed, each with a unique name, plugin path, and arguments.</li>
</ul><br />
<span>Adjust the configuration file according to your needs, specifying the checks you want Gogios to perform.</span><br />
<br />
<span>If you want to execute checks only when another check succeeded (status OK), use <span class='inlinecode'>DependsOn</span>. In the example above, the HTTP checks won't run when the hosts aren't pingable. They will show up as <span class='inlinecode'>UNKNOWN</span> in the report.</span><br />
<br />
<span><span class='inlinecode'>Retries</span> and <span class='inlinecode'>RetryInterval</span> are optional check configuration parameters. In case of failure, Gogios will retry <span class='inlinecode'>Retries</span> times each <span class='inlinecode'>RetryInterval</span> seconds.</span><br />
<br />
<span>For remote checks, use the <span class='inlinecode'>check_nrpe</span> plugin. You also need to have the NRPE server set up correctly on the target host (out of scope for this document).</span><br />
<br />
<span>The <span class='inlinecode'>state.json</span> file mentioned above keeps track of the monitoring state and check results between Gogios runs, enabling Gogios only to send email notifications when there are changes in the check status.</span><br />
<br />
<h2 style='display: inline' id='RunningGogios'>Running Gogios</h2><br />
<br />
<span>Now it is time to give it a first run. On OpenBSD, do:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">doas -u _gogios /usr/local/bin/gogios -cfg /etc/gogios</font><font color="#F3E651">.</font><font color="#ff0000">json</font>
</pre>
<br />
<span>To run Gogios via CRON on OpenBSD as the <span class='inlinecode'>gogios</span> user and check all services once per minute, follow these steps:</span><br />
<br />
<span>Type <span class='inlinecode'>doas crontab -e -u _gogios</span> and press Enter to open the crontab file for the <span class='inlinecode'>_gogios</span> user for editing and add the following lines to the crontab file:</span><br />
<br />
<pre>
- /5 8-22 * * * /usr/local/bin/gogios -cfg /etc/gogios.json
0 7 * * * /usr/local/bin/gogios -renotify -cfg /etc/gogios.json
</pre>
<br />
<span>Gogios is now configured to run every five minutes from 8 am to 10 pm via CRON as the <span class='inlinecode'>_gogios</span> user. It will execute the checks and send monitoring status whenever a check status changes via email according to your configuration. Also, Gogios will run once at 7 am every morning and re-notify all unhandled alerts as a reminder.</span><br />
<br />
<h3 style='display: inline' id='Highavailability'>High-availability</h3><br />
<br />
<span>To create a high-availability Gogios setup, you can install Gogios on two servers that will monitor each other using the NRPE (Nagios Remote Plugin Executor) plugin. By running Gogios in alternate CRON intervals on both servers, you can ensure that even if one server goes down, the other will continue monitoring your infrastructure and sending notifications.</span><br />
<br />
<ul>
<li>Install Gogios on both servers following the compilation and installation instructions provided earlier.</li>
<li>Install the NRPE server (out of scope for this document) and plugin on both servers. This plugin allows you to execute Nagios check scripts on remote hosts.</li>
<li>Configure Gogios on both servers to monitor each other using the NRPE plugin. Add a check to the Gogios configuration file (<span class='inlinecode'>/etc/gogios.json</span>) on both servers that uses the NRPE plugin to execute a check script on the other server. For example, if you have Server A and Server B, the configuration on Server A should include a check for Server B, and vice versa.</li>
<li>Set up alternate CRON intervals on both servers. Configure the CRON job on Server A to run Gogios at minutes 0, 10, 20, ..., and on Server B to run at minutes 5, 15, 25, ... This will ensure that if one server goes down, the other server will continue monitoring and sending notifications. </li>
<li>Gogios doesn't support clustering. So it means when both servers are up, unhandled alerts will be notified via E-Mail twice; from each server once. That's the trade-off for simplicity.</li>
</ul><br />
<span>There are plans to make it possible to execute certain checks only on certain nodes (e.g. on elected leader or master nodes). This is still in progress (check out my Gorum Git project).</span><br />
<br />
<h2 style='display: inline' id='Conclusion'>Conclusion:</h2><br />
<br />
<span>Gogios is a lightweight and straightforward monitoring tool that is perfect for small-scale environments. With its compatibility with the Nagios Check API, email notifications, and CRON-based scheduling, Gogios offers an easy-to-use solution for those looking to monitor a limited number of resources. I personally use it to execute around 500 checks on my personal server infrastructure. I am very happy with this solution.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other KISS-related posts are:</span><br />
<br />
<a class='textlink' href='./2021-09-12-keep-it-simple-and-stupid.html'>2021-09-12 Keep it simple and stupid</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios (You are currently reading this)</a><br />
<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>'The Obstacle is the Way' book notes</title>
<link href="gemini://foo.zone/gemfeed/2023-05-06-the-obstacle-is-the-way-book-notes.gmi" />
<id>gemini://foo.zone/gemfeed/2023-05-06-the-obstacle-is-the-way-book-notes.gmi</id>
<updated>2023-05-06T17:23:16+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>These are my personal takeaways after reading 'The Obstacle Is the Way' by Ryan Holiday. This is mainly for my own use, but you might find it helpful too.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='TheObstacleistheWaybooknotes'>"The Obstacle is the Way" book notes</h1><br />
<br />
<span class='quote'>Published at 2023-05-06T17:23:16+03:00</span><br />
<br />
<span>These are my personal takeaways after reading "The Obstacle Is the Way" by Ryan Holiday. This is mainly for my own use, but you might find it helpful too.</span><br />
<br />
<br />
<pre>
,.......... ..........,
,..,' '.' ',..,
,' ,' : ', ',
,' ,' : ', ',
,' ,' : ', ',
,' ,'............., : ,.............', ',
,' '............ '.' ............' ',
'''''''''''''''''';''';''''''''''''''''''
'''
</pre>
<br />
<span>"The obstacle is the way" is a powerful statement that encapsulates the wisdom of turning challenges into opportunities for growth and success. We will explore using obstacles as fuel, transforming weaknesses into strengths, and adopting a mindset that allows us to be creative and persistent in the face of adversity.</span><br />
<br />
<h2 style='display: inline' id='Reframeyourperspective'>Reframe your perspective</h2><br />
<br />
<span>The obstacle in your path can become your path to success. Instead of being paralyzed by challenges, see them as opportunities to learn and grow. Remember, the things that hurt us often instruct us. </span><br />
<br />
<span>We spend a lot of time trying to get things perfect and look at the rules, but what matters is that it works; it doesn't need to be after the book. Focus on results rather than on beautiful methods. In Jujitsu, it does matter that you bring your opponent down, but not how. There are many ways from point A to point B; it doesn't need to be a straight line. So many try to find the best solution but need to catch up on what is in Infront of them. Think progress and not perfection.</span><br />
<br />
<span>Don't always try to use the front door; a backdoor could open. It's nonsense. Don't fight the judo master with judo. Non-action can be action, exposing the weaknesses of others.</span><br />
<br />
<br />
<h2 style='display: inline' id='Embracerationality'>Embrace rationality</h2><br />
<br />
<span>It is a superpower to see things rationally when others are fearful. Focus on the reality of the situation without letting emotions, such as anger, cloud your judgment. This ability will enable you to make better decisions in adversity. Ability to see things what they really are. E.g. wine is old fermented grapes, or other people behaving like animals during a fight. Show the middle finger if someone persists on the stupid rules occasionally.</span><br />
<br />
<h2 style='display: inline' id='Controlyourresponse'>Control your response</h2><br />
<br />
<span>You can choose how you respond to obstacles. Focus on what you can control, and don't let yourself feel harmed by external circumstances. Remember, you decide how things affect you; nobody else does. Choose to feel good in response to any situation. Embrace the challenges and obstacles that come your way, as they are opportunities for growth and learning.</span><br />
<br />
<h2 style='display: inline' id='Practiceemotionalandphysicalresilience'>Practice emotional and physical resilience</h2><br />
<br />
<span>Martial artists know the importance of developing physical and emotional strength. Cultivate the art of not panicking; it will help you avoid making mistakes during high-pressure situations.</span><br />
<br />
<span>Focus on what you can control. Don't choose to feel harmed, and then you won't be harmed. I decide things that affect me; nobody else does. E.g., in prison, your mind stays your own. Don't ignore fear but explain it away, have a different view.</span><br />
<br />
<h2 style='display: inline' id='Persistenceandpatience'>Persistence and patience</h2><br />
<br />
<span>Practice persistence and patience in your pursuits. Focus on the process rather than the prize and take one step at a time. Remember, the journey is about finishing tasks, projects, or workouts to the best of your ability. Never be in a hurry and never be desperate. There is no reason to be rushed; there are all in the long haul. Follow the process and not the price. Take it one step at a time. The process is about finishing (workout, task, project, etc.).</span><br />
<br />
<h2 style='display: inline' id='Embracefailure'>Embrace failure</h2><br />
<br />
<span>Failure is a natural part of life and can make us stronger. Treat defeat as a stepping stone to success and education. What is defeat? The first step to education. Failure makes you stronger. If we do our best, we can be proud of it, regardless of the result. Do your job, but do it right. Only an asshole thinks he is too good at the things he does. Also, asking for forgiveness is easier than asking for permission. </span><br />
<br />
<h2 style='display: inline' id='Beadaptable'>Be adaptable</h2><br />
<br />
<span>There are many ways to achieve your goals; sometimes, unconventional methods are necessary. Feel free to break the rules or go off the beaten path if it will lead to better results. Transform weaknesses into strengths. We have a choice of how to respond to things. It's not about being positive but to be creative. Aim high, but stuff will happen; E.g., surprises will always happen.</span><br />
<br />
<h2 style='display: inline' id='Embracenonaction'>Embrace non-action</h2><br />
<br />
<span>We constantly push to the next thing. Sometimes the best course of action is standing still or even going backwards. Obstacles might resolve by themselves. Or going sideways. Sometimes, the best action is to stand still, go sideways, or even go backwards. Obstacles may resolve themselves or present new opportunities if you're patient and observant. People always want your input before you have all the facts. They want you to play after their rules. The question is, do you let them? The English call it the cool head. Being in control of Stress; requires practice. Appear, the absence of fear (Greek). When all others do it one way, it does not mean it is the correct or best practice.</span><br />
<br />
<h2 style='display: inline' id='Leveragecrisis'>Leverage crisis</h2><br />
<br />
<span>In times of crisis, seize the chance to do things never done before. Great people use negative situations to their advantage and become the most effective in challenging circumstances.</span><br />
<br />
<span>The art of not panicking; otherwise, you will make mistakes. When overs are shocked, you know which way to take due to your thinking of the problem at Hand. A crisis gives you a chance to do things which never done before. Ordinary people shy from negative situations; great people use these for their benefit and are the most effective. The obstacle is not just turned upside down but used as a catapult.</span><br />
<br />
<span>Be prepared for nothing to work. Problems are an opportunity to do your best, not to do miracles. Always manage your expectations. It will suck, but it will be ok. Be prepared to begin from the beginning. Be cheerful and eagerly work on the next obstacle. Each time you become better. Life is not a sprint but a marathon. After each obstacle lies another obstacle, there won't be anything without obstacles. Passing one means you are ready for the next.</span><br />
<br />
<h2 style='display: inline' id='Buildyourinnercitadel'>Build your inner citadel</h2><br />
<br />
<span>Develop your inner strength during good times so you can rely on it in bad times. Always prepare for adversity and face it with calmness and resilience. Be humble enough that things which happen will happen. Build your inner citadel. In good times strengthen it. In bad times rely on it.</span><br />
<br />
<span>We should always prepare for things to get tough. Your house burns down: no worries, we eliminated much rubbish. Imagine what can go wrong before things go wrong. We are prepared for adversity; it's other people who aren't. Phil Jackson's hip problem example. To receive unexpected benefits, you must first accept the unexpected obstacles. Meditate on death. It's a universal obstacle. Use it as a reminder to do your best.</span><br />
<br />
<h2 style='display: inline' id='Loveeverythingthathappens'>Love everything that happens</h2><br />
<br />
<span>Turn an obstacle the other way around for your benefit. Use it at fuel. It's simple but challenging. Most are paralyzed instead. The obstacle in the path becomes the path. Obstacles are neither good nor bad. The things which hurt, instruct.</span><br />
<br />
<span>Should I hate people who hate me? That's their problem and not mine. Be always calm and relaxed during the fight. The story of the battle is the story of the smile. Cheerfulness in all situations, especially the bad ones. Love for everything that happens; if it happens, it was meant to happen. We can choose how we react to things, so why not choose to feel good? I love everything that happens. You must never lower yourself to the person you don't like.</span><br />
<br />
<h2 style='display: inline' id='Conclusion'>Conclusion</h2><br />
<br />
<span>Life is a marathon, not a sprint. Each obstacle we overcome prepares us for the next one. Remember, the obstacle is not just a barrier to be turned upside down; it can also be used as a catapult to propel us forward. By embracing challenges and using them as opportunities for growth, we become stronger, more adaptable, and, ultimately, more successful.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other book notes of mine are:</span><br />
<br />
<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 "The Pragmatic Programmer" book notes</a><br />
<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 "Never split the difference" book notes</a><br />
<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 "The Obstacle is the Way" book notes (You are currently reading this)</a><br />
<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes</a><br />
<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 "Mind Management" book notes</a><br />
<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 "Slow Productivity" book notes</a><br />
<br />
<a class='textlink' href='../resources.html'>More books and other resources I found useful.</a><br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Unveiling `guprecords.raku`: Global Uptime Records with Raku</title>
<link href="gemini://foo.zone/gemfeed/2023-05-01-unveiling-guprecords:-uptime-records-with-raku.gmi" />
<id>gemini://foo.zone/gemfeed/2023-05-01-unveiling-guprecords:-uptime-records-with-raku.gmi</id>
<updated>2023-04-30T13:10:26+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>For fun, I am tracking the uptime of various personal machines (servers, laptops, workstations...). I have been doing this for over ten years now, so I have a lot of statistics collected.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='UnveilingguprecordsrakuGlobalUptimeRecordswithRaku'>Unveiling <span class='inlinecode'>guprecords.raku</span>: Global Uptime Records with Raku</h1><br />
<br />
<span class='quote'>Published at 2023-04-30T13:10:26+03:00</span><br />
<br />
<pre>
+-----+-----------------+-----------------------------+
| Pos | Host | Lifespan |
+-----+-----------------+-----------------------------+
| 1. | dionysus | 8 years, 6 months, 17 days |
| 2. | uranus | 7 years, 2 months, 16 days |
| 3. | alphacentauri | 6 years, 9 months, 13 days |
| 4. | *vulcan | 4 years, 5 months, 6 days |
| 5. | sun | 3 years, 10 months, 2 days |
| 6. | uugrn | 3 years, 5 months, 5 days |
| 7. | deltavega | 3 years, 1 months, 21 days |
| 8. | pluto | 2 years, 10 months, 30 days |
| 9. | tauceti | 2 years, 3 months, 22 days |
| 10. | callisto | 2 years, 3 months, 13 days |
+-----+-----------------+-----------------------------+
</pre>
<br />
<h1 style='display: inline' id='Introduction'>Introduction</h1><br />
<br />
<span>For fun, I am tracking the uptime of various personal machines (servers, laptops, workstations...). I have been doing this for over ten years now, so I have a lot of statistics collected.</span><br />
<br />
<span>As a result of this, I am introducing <span class='inlinecode'>guprecords.raku</span>, a handy Raku script that helps me combine uptime statistics from multiple servers into one comprehensive report. In this blog post, I'll explore what Guprecords is and some examples of its application. I will also add some notes on Raku.</span><br />
<br />
<span>Guprecords, or global uptime records, is a Raku script designed to generate a consolidated uptime report from multiple hosts:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/guprecords'>https://codeberg.org/snonux/guprecords</a><br />
<a class='textlink' href='https://raku.org'>The Raku Programming Language</a><br />
<br />
<span>A previous version of Guprecords was actually written in Perl, the older and more established language from which Raku was developed. One of the primary motivations for rewriting Guprecords in Raku was to learn the language and explore its features. Raku is a more modern and powerful language compared to Perl, and working on a real-world project like Guprecords provided a practical and engaging way to learn the language.</span><br />
<br />
<span>Over the last years, I have been reading the following books and resources about Raku:</span><br />
<br />
<ul>
<li>Raku Guide (at raku.guide)</li>
<li>Think Perl 6</li>
<li>Raku Fundamentals</li>
<li>Raku Recipes</li>
</ul><br />
<span>And I have been following the Raku newsletter, and sometimes I have been lurking around in the IRC channels, too. Watching Raku coding challenges on YouTube was pretty fun, too. However, nothing beats actually using Raku to learn the language. After reading all of these resources, I may have a good idea about the features and paradigms, but I am by far not an expert.</span><br />
<br />
<h2 style='display: inline' id='HowGuprecordsworks'>How Guprecords works</h2><br />
<br />
<span>Guprecords works in three stages:</span><br />
<br />
<ul>
<li>1. Generating uptime statistics using <span class='inlinecode'>uptimed</span>: First, I need to install and run <span class='inlinecode'>uptimed</span> on each host to generate uptime statistics. This tool is available for most common Linux and *BSD distributions and macOS via Homebrew.</li>
<li>2. Collecting uptime records to a central location: The next step involves collecting the raw uptime statistics files generated by <span class='inlinecode'>uptimed</span> on each host. It's a good idea to store all record files in a central git repository. The records file contains information about the total uptime since boot, boot time, and the operating system and kernel version. Guprecords itself does not do the collection part, but have a look at the <span class='inlinecode'>README.md</span> in the git repository for some guidance.</li>
<li>3. Generating global uptime stats: Finally, run the <span class='inlinecode'>guprecords.raku</span> script with the appropriate flags to create a global uptime report. For example, I can use the following command:</li>
</ul><br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">$ raku guprecords</font><font color="#F3E651">.</font><font color="#ff0000">raku --stats</font><font color="#F3E651">=</font><font color="#ff0000">dir</font><font color="#F3E651">=</font><font color="#ff0000">$HOME</font><font color="#ff0000">/git/uprecords/stats --all</font>
</pre>
<br />
<span>This command will generate a comprehensive uptime report from the collected statistics, making it easy to review and enjoy the data.</span><br />
<br />
<span>Guprecords supports the following features:</span><br />
<br />
<ul>
<li>Supports multiple categories: Host, Kernel, KernelMajor, and KernelName</li>
<li>Supports multiple metrics: Boots, Uptime, Score, Downtime, and Lifespan</li>
<li>Output formats available: Plaintext, Markdown, and Gemtext</li>
<li>Provides top entries based on the specified limit</li>
</ul><br />
<h2 style='display: inline' id='Example'>Example</h2><br />
<br />
<span>You have already seen an example at the very top of this post, where the hosts were grouped by their total lifespans (uptime+downtime). Here's an example of what the global uptime report (grouped by total host uptimes) might look like:</span><br />
<br />
<pre>
Top 20 Uptime's by Host
+-----+-----------------+-----------------------------+
| Pos | Host | Uptime |
+-----+-----------------+-----------------------------+
| 1. | *vulcan | 4 years, 5 months, 6 days |
| 2. | uranus | 3 years, 11 months, 21 days |
| 3. | sun | 3 years, 9 months, 26 days |
| 4. | uugrn | 3 years, 5 months, 5 days |
| 5. | deltavega | 3 years, 1 months, 21 days |
| 6. | pluto | 2 years, 10 months, 29 days |
| 7. | tauceti | 2 years, 3 months, 19 days |
| 8. | tauceti-f | 1 years, 9 months, 18 days |
| 9. | *ultramega15289 | 1 years, 8 months, 17 days |
| 10. | *earth | 1 years, 5 months, 22 days |
| 11. | *blowfish | 1 years, 4 months, 20 days |
| 12. | ultramega8477 | 1 years, 3 months, 25 days |
| 13. | host0 | 1 years, 3 months, 9 days |
| 14. | tauceti-e | 1 years, 2 months, 20 days |
| 15. | makemake | 1 years, 1 months, 6 days |
| 16. | callisto | 0 years, 10 months, 31 days |
| 17. | alphacentauri | 0 years, 10 months, 28 days |
| 18. | london | 0 years, 9 months, 16 days |
| 19. | twofish | 0 years, 8 months, 31 days |
| 20. | *fishfinger | 0 years, 8 months, 17 days |
+-----+-----------------+-----------------------------+
</pre>
<br />
<span>This table ranks the top 20 hosts based on their total uptime, with the host having the highest uptime at the top. The hosts marked with <span class='inlinecode'>*</span> are still active, means stats were collected within the last couple of months. </span><br />
<br />
<span>My up to date stats can be seen here:</span><br />
<br />
<a class='textlink' href='../uptime-stats.html'>My machine uptime stats</a><br />
<br />
<span>Just recently, I decommissioned <span class='inlinecode'>vulcan</span> (the number one stop from above), which used to be my CentOS 7 (initially CentOS 6) VM hosting my personal NextCloud and Wallabag (which I modernised just recently with a brand new shiny Rocky Linux 9 VM). This was the last <span class='inlinecode'>uptimed</span> output before shutting it down (it always makes me feel sentimental decommissioning one of my machines <span class='inlinecode'>:'-(</span>):</span><br />
<br />
<pre>
# Uptime | System Boot up
----------------------------+---------------------------------------------------
1 545 days, 17:58:15 | Linux 3.10.0-1160.15.2.e Sun Jul 25 19:32:25 2021
2 279 days, 10:12:14 | Linux 3.10.0-957.21.3.el Sun Jun 30 12:43:41 2019
3 161 days, 06:08:43 | Linux 3.10.0-1160.15.2.e Sun Feb 14 11:05:38 2021
4 107 days, 01:26:35 | Linux 3.10.0-957.1.3.el7 Thu Dec 20 09:29:13 2018
5 96 days, 21:13:49 | Linux 3.10.0-1127.13.1.e Sat Jul 25 17:56:22 2020
-> 6 89 days, 23:05:32 | Linux 3.10.0-1160.81.1.e Sun Jan 22 12:39:36 2023
7 63 days, 18:30:45 | Linux 3.10.0-957.10.1.el Sat Apr 27 18:12:43 2019
8 63 days, 06:53:33 | Linux 3.10.0-1127.8.2.el Sat May 23 10:41:08 2020
9 48 days, 11:44:49 | Linux 3.10.0-1062.18.1.e Sat Apr 4 22:56:07 2020
10 42 days, 08:00:13 | Linux 3.10.0-1127.19.1.e Sat Nov 7 11:47:33 2020
11 36 days, 22:57:19 | Linux 3.10.0-1160.6.1.el Sat Dec 19 19:47:57 2020
12 21 days, 06:16:28 | Linux 3.10.0-957.10.1.el Sat Apr 6 11:56:01 2019
13 12 days, 20:11:53 | Linux 3.10.0-1160.11.1.e Mon Jan 25 18:45:27 2021
14 7 days, 21:29:18 | Linux 3.10.0-1127.13.1.e Fri Oct 30 14:18:04 2020
15 6 days, 20:07:18 | Linux 3.10.0-1160.15.2.e Sun Feb 7 14:57:35 2021
16 1 day , 21:46:41 | Linux 3.10.0-957.1.3.el7 Tue Dec 18 11:42:19 2018
17 0 days, 01:25:57 | Linux 3.10.0-957.1.3.el7 Tue Dec 18 10:16:08 2018
18 0 days, 00:42:34 | Linux 3.10.0-1160.15.2.e Sun Jul 25 18:49:38 2021
19 0 days, 00:08:32 | Linux 3.10.0-1160.81.1.e Sun Jan 22 12:30:52 2023
----------------------------+---------------------------------------------------
1up in 6 days, 22:08:18 | at Sat Apr 29 10:53:25 2023
no1 in 455 days, 18:52:44 | at Sun Jul 21 07:37:51 2024
up 1586 days, 00:20:28 | since Tue Dec 18 10:16:08 2018
down 0 days, 01:08:32 | since Tue Dec 18 10:16:08 2018
%up 99.997 | since Tue Dec 18 10:16:08 2018
</pre>
<br />
<h2 style='display: inline' id='Conclusion'>Conclusion</h2><br />
<br />
<span>Guprecords is a small, yet powerful tool for analyzing uptime statistics. While developing Guprecords, I have come to truly appreciate and love Raku's expressiveness. The language is designed to be both powerful and flexible, allowing developers to express their intentions and logic more clearly and concisely.</span><br />
<br />
<span>Raku's expressive syntax, support for multiple programming paradigms, and unique features, such as grammars and lazy evaluation, make it a joy to work with. </span><br />
<br />
<span>Working on Guprecords in Raku has been an enjoyable experience, and I've found that Raku's expressiveness has significantly contributed to the overall quality and effectiveness of the script. The language's ability to elegantly express complex logic and data manipulation tasks makes it an excellent choice for developing tools like these, where expressiveness and productiveness are of the utmost importance.</span><br />
<br />
<span>So far, I have only scratched the surface of what Raku can do. I hope to find more time to become a regular Rakoon (a Raku Programmer). I have many Ideas for other small tools like Guprecords, but the challenge is finding the time. I'd love to explore Raku Grammars and also I would love to explore writing concurrent code in Raku (I also love Go (Golang), btw!). Ideas for future Raku personal projects include:</span><br />
<br />
<ul>
<li>A log file analyzer, for generating anonymized <span class='inlinecode'>foo.zone</span> visitor stats for both, the Web and Gemini.</li>
<li>A social media sharing scheduler a la <span class='inlinecode'>buffer.com</span>. I am using Buffer at the moment to share posts on Mastadon, Twitter, Telegram and LinkedIn, but it is proprietary and also it's not really reliable.</li>
<li>Rewrite the static photo album generator of <span class='inlinecode'>irregular.ninja</span> in Raku (from Bash).</li>
</ul><br />
<span>E-Mail your comments to hi@foo.zone :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2008-06-26-perl-poetry.html'>2008-06-26 Perl Poetry</a><br />
<a class='textlink' href='./2011-05-07-perl-daemon-service-framework.html'>2011-05-07 Perl Daemon (Service Framework)</a><br />
<a class='textlink' href='./2022-05-27-perl-is-still-a-great-choice.html'>2022-05-27 Perl is still a great choice</a><br />
<a class='textlink' href='./2022-06-15-sweating-the-small-stuff.html'>2022-06-15 Sweating the small stuff - Tiny projects of mine</a><br />
<a class='textlink' href='./2023-05-01-unveiling-guprecords:-uptime-records-with-raku.html'>2023-05-01 Unveiling <span class='inlinecode'>guprecords.raku</span>: Global Uptime Records with Raku (You are currently reading this)</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Algorithms and Data Structures in Go - Part 1</title>
<link href="gemini://foo.zone/gemfeed/2023-04-09-algorithms-and-data-structures-in-golang-part-1.gmi" />
<id>gemini://foo.zone/gemfeed/2023-04-09-algorithms-and-data-structures-in-golang-part-1.gmi</id>
<updated>2023-04-09T22:31:42+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is the first blog post about my Algorithms and Data Structures in Go series. I am not a Software Developer in my day job. In my current role, programming and scripting skills are desirable but not mandatory. I have been learning about Data Structures and Algorithms many years ago at University. I thought it would be fun to revisit/refresh my knowledge here and implement many of the algorithms in Go.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='AlgorithmsandDataStructuresinGoPart1'>Algorithms and Data Structures in Go - Part 1</h1><br />
<br />
<span class='quote'>Published at 2023-04-09T22:31:42+03:00</span><br />
<br />
<pre>
,_---~~~~~----._
_,,_,*^____ _____``*g*\"*,
/ __/ /' ^. / \ ^@q f
[ @f | @)) | | @)) l 0 _/
\`/ \~____ / __ \_____/ \
| _l__l_ I
} [______] I
] | | | |
] ~ ~ |
| |
| |
</pre>
<br />
<span>This is the first blog post about my Algorithms and Data Structures in Go series. I am not a Software Developer in my day job. In my current role, programming and scripting skills are desirable but not mandatory. I have been learning about Data Structures and Algorithms many years ago at University. I thought it would be fun to revisit/refresh my knowledge here and implement many of the algorithms in Go.</span><br />
<br />
<a class='textlink' href='./2023-04-09-algorithms-and-data-structures-in-golang-part-1.html'>2023-04-09 Algorithms and Data Structures in Go - Part 1 (You are currently reading this)</a><br />
<br />
<span>This post is about setting up some basic data structures and methods for this blog series. I promise, everything will be easy to follow in this post. It will become more interesting later in this series.</span><br />
<br />
<h2 style='display: inline' id='Typeconstraints'>Type constraints</h2><br />
<br />
<span>First, the package <span class='inlinecode'>ds</span> (data structures) defines the <span class='inlinecode'>types.go</span>. All examples will either operate on the <span class='inlinecode'>Integer</span> or <span class='inlinecode'>Number</span> type:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">package</font></b><font color="#ff0000"> ds</font>
<b><font color="#ffffff">import</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font>
<font color="#ff0000"> </font><font color="#bb00ff">"golang.org/x/exp/constraints"</font>
<font color="#F3E651">)</font>
<b><font color="#ffffff">type</font></b><font color="#ff0000"> Integer </font><b><font color="#ffffff">interface</font></b><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> constraints</font><font color="#F3E651">.</font><font color="#ff0000">Integer</font>
<font color="#F3E651">}</font>
<b><font color="#ffffff">type</font></b><font color="#ff0000"> Number </font><b><font color="#ffffff">interface</font></b><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> constraints</font><font color="#F3E651">.</font><font color="#ff0000">Integer </font><font color="#F3E651">|</font><font color="#ff0000"> constraints</font><font color="#F3E651">.</font><font color="#ff0000">Float</font>
<font color="#F3E651">}</font>
</pre>
<br />
<h2 style='display: inline' id='ArrayList'>ArrayList</h2><br />
<br />
<span>Next comes the <span class='inlinecode'>arraylist.go</span>, which defines the underlying data structure all the algorithms of this series will use. <span class='inlinecode'>ArrayList</span> is just a type alias of a Go array (or slice) with custom methods on it:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">package</font></b><font color="#ff0000"> ds</font>
<b><font color="#ffffff">import</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font>
<font color="#ff0000"> </font><font color="#bb00ff">"fmt"</font>
<font color="#ff0000"> </font><font color="#bb00ff">"math/rand"</font>
<font color="#ff0000"> </font><font color="#bb00ff">"strings"</font>
<font color="#F3E651">)</font>
<b><font color="#ffffff">type</font></b><font color="#ff0000"> ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V Number</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">[]</font><font color="#ff0000">V</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> NewArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V Number</font><font color="#F3E651">](</font><font color="#ff0000">l int</font><font color="#F3E651">)</font><font color="#ff0000"> ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#7bc710">make</font><font color="#F3E651">(</font><font color="#ff0000">ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">],</font><font color="#ff0000"> l</font><font color="#F3E651">)</font>
<font color="#F3E651">}</font>
</pre>
<br />
<span>As you can see, the code uses Go generics, which I refactored recently. Besides the default constructor (which only returns an empty <span class='inlinecode'>ArrayList</span> with a given capacity), there are also a bunch of special constructors. <span class='inlinecode'>NewRandomArrayList</span> is returning an <span class='inlinecode'>ArrayList</span> with random numbers, <span class='inlinecode'>NewAscendingArrayList</span> and <span class='inlinecode'>NewDescendingArrayList</span> are returning <span class='inlinecode'>ArrayList</span>s in either ascending or descending order. They all will be used later on for testing and benchmarking the algorithms.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">func</font></b><font color="#ff0000"> NewRandomArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V Number</font><font color="#F3E651">](</font><font color="#ff0000">l</font><font color="#F3E651">,</font><font color="#ff0000"> max int</font><font color="#F3E651">)</font><font color="#ff0000"> ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#7bc710">make</font><font color="#F3E651">(</font><font color="#ff0000">ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">],</font><font color="#ff0000"> l</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651"><</font><font color="#ff0000"> l</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">++</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> max </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">V</font><font color="#F3E651">(</font><font color="#ff0000">rand</font><font color="#F3E651">.</font><font color="#7bc710">Intn</font><font color="#F3E651">(</font><font color="#ff0000">max</font><font color="#F3E651">))</font>
<font color="#ff0000"> </font><b><font color="#ffffff">continue</font></b>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">V</font><font color="#F3E651">(</font><font color="#ff0000">rand</font><font color="#F3E651">.</font><font color="#7bc710">Int</font><font color="#F3E651">())</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> a</font>
<font color="#F3E651">}</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> NewAscendingArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V Number</font><font color="#F3E651">](</font><font color="#ff0000">l int</font><font color="#F3E651">)</font><font color="#ff0000"> ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#7bc710">make</font><font color="#F3E651">(</font><font color="#ff0000">ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">],</font><font color="#ff0000"> l</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651"><</font><font color="#ff0000"> l</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">++</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">V</font><font color="#F3E651">(</font><font color="#ff0000">i</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> a</font>
<font color="#F3E651">}</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> NewDescendingArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V Number</font><font color="#F3E651">](</font><font color="#ff0000">l int</font><font color="#F3E651">)</font><font color="#ff0000"> ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#7bc710">make</font><font color="#F3E651">(</font><font color="#ff0000">ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">],</font><font color="#ff0000"> l</font><font color="#F3E651">)</font>
<font color="#ff0000"> j </font><font color="#F3E651">:=</font><font color="#ff0000"> l </font><font color="#F3E651">-</font><font color="#ff0000"> </font><font color="#bb00ff">1</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651"><</font><font color="#ff0000"> l</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">++</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">V</font><font color="#F3E651">(</font><font color="#ff0000">j</font><font color="#F3E651">)</font>
<font color="#ff0000"> j</font><font color="#F3E651">--</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> a</font>
<font color="#F3E651">}</font>
</pre>
<br />
<h2 style='display: inline' id='Helpermethods'>Helper methods</h2><br />
<br />
<span>The <span class='inlinecode'>FirstN</span> method only returns the first N elements of the <span class='inlinecode'>ArrayList</span>. This is useful for printing out only parts of the data structure:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">func</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">a ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">])</font><font color="#ff0000"> </font><font color="#7bc710">FirstN</font><font color="#F3E651">(</font><font color="#ff0000">n int</font><font color="#F3E651">)</font><font color="#ff0000"> </font><b><font color="#F35E1E">string</font></b><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">var</font></b><font color="#ff0000"> sb strings</font><font color="#F3E651">.</font><font color="#ff0000">Builder</font>
<font color="#ff0000"> j </font><font color="#F3E651">:=</font><font color="#ff0000"> n</font>
<font color="#ff0000"> l </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#7bc710">len</font><font color="#F3E651">(</font><font color="#ff0000">a</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> j </font><font color="#F3E651">></font><font color="#ff0000"> l </font><font color="#F3E651">{</font>
<font color="#ff0000"> j </font><font color="#F3E651">=</font><font color="#ff0000"> l</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651"><</font><font color="#ff0000"> j</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">++</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> fmt</font><font color="#F3E651">.</font><font color="#7bc710">Fprintf</font><font color="#F3E651">(&</font><font color="#ff0000">sb</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">"%v "</font><font color="#F3E651">,</font><font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">])</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> j </font><font color="#F3E651"><</font><font color="#ff0000"> l </font><font color="#F3E651">{</font>
<font color="#ff0000"> fmt</font><font color="#F3E651">.</font><font color="#7bc710">Fprintf</font><font color="#F3E651">(&</font><font color="#ff0000">sb</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">"... "</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> sb</font><font color="#F3E651">.</font><font color="#7bc710">String</font><font color="#F3E651">()</font>
<font color="#F3E651">}</font>
</pre>
<br />
<span>The <span class='inlinecode'>Sorted</span> method checks whether the <span class='inlinecode'>ArrayList</span> is sorted. This will be used by the unit tests later on:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">func</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">a ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">])</font><font color="#ff0000"> </font><font color="#7bc710">Sorted</font><font color="#F3E651">()</font><font color="#ff0000"> </font><b><font color="#F35E1E">bool</font></b><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> i </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#7bc710">len</font><font color="#F3E651">(</font><font color="#ff0000">a</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">-</font><font color="#ff0000"> </font><font color="#bb00ff">1</font><font color="#F3E651">;</font><font color="#ff0000"> i </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#bb00ff">0</font><font color="#F3E651">;</font><font color="#ff0000"> i</font><font color="#F3E651">--</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651"><</font><font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">-</font><font color="#bb00ff">1</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> false</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> true</font>
<font color="#F3E651">}</font>
</pre>
<br />
<span>And the last utility method used is <span class='inlinecode'>Swap</span>, which allows swapping the values of two indices in the <span class='inlinecode'>ArrayList</span>:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">func</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font><font color="#ff0000">a ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">])</font><font color="#ff0000"> </font><font color="#7bc710">Swap</font><font color="#F3E651">(</font><font color="#ff0000">i</font><font color="#F3E651">,</font><font color="#ff0000"> j int</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> aux </font><font color="#F3E651">:=</font><font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font>
<font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">i</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">j</font><font color="#F3E651">]</font>
<font color="#ff0000"> a</font><font color="#F3E651">[</font><font color="#ff0000">j</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">=</font><font color="#ff0000"> aux</font>
<font color="#F3E651">}</font>
</pre>
<br />
<h2 style='display: inline' id='Sleepsort'>Sleep sort</h2><br />
<br />
<span>Let's implement our first algorithm, sleep sort. Sleep sort is a non-traditional and unconventional sorting algorithm based on the idea of waiting a certain amount of time corresponding to the value of each element in the input <span class='inlinecode'>ArrayList</span>. It's more of a fun, creative concept rather than an efficient or practical sorting technique. This is not a sorting algorithm you would use in any production code. As you can imagine, it is quite an inefficient sorting algorithm (it's only listed here as a warm-up exercise). This sorting method may also return false results depending on how the Goroutines are scheduled by the Go runtime. </span><br />
<br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">package</font></b><font color="#ff0000"> sort</font>
<b><font color="#ffffff">import</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font>
<font color="#ff0000"> </font><font color="#bb00ff">"codeberg.org/snonux/algorithms/ds"</font>
<font color="#ff0000"> </font><font color="#bb00ff">"sync"</font>
<font color="#ff0000"> </font><font color="#bb00ff">"time"</font>
<font color="#F3E651">)</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> Sleep</font><font color="#F3E651">[</font><font color="#ff0000">V ds</font><font color="#F3E651">.</font><font color="#ff0000">Integer</font><font color="#F3E651">](</font><font color="#ff0000">a ds</font><font color="#F3E651">.</font><font color="#ff0000">ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">])</font><font color="#ff0000"> ds</font><font color="#F3E651">.</font><font color="#ff0000">ArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">]</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> sorted </font><font color="#F3E651">:=</font><font color="#ff0000"> ds</font><font color="#F3E651">.</font><font color="#ff0000">NewArrayList</font><font color="#F3E651">[</font><font color="#ff0000">V</font><font color="#F3E651">](</font><font color="#7bc710">len</font><font color="#F3E651">(</font><font color="#ff0000">a</font><font color="#F3E651">))</font>
<font color="#ff0000"> numCh </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><font color="#7bc710">make</font><font color="#F3E651">(</font><b><font color="#ffffff">chan</font></b><font color="#ff0000"> V</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><b><font color="#ffffff">var</font></b><font color="#ff0000"> wg sync</font><font color="#F3E651">.</font><font color="#ff0000">WaitGroup</font>
<font color="#ff0000"> wg</font><font color="#F3E651">.</font><font color="#7bc710">Add</font><font color="#F3E651">(</font><font color="#7bc710">len</font><font color="#F3E651">(</font><font color="#ff0000">a</font><font color="#F3E651">))</font>
<font color="#ff0000"> </font><b><font color="#ffffff">go</font></b><font color="#ff0000"> </font><b><font color="#ffffff">func</font></b><font color="#F3E651">()</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> wg</font><font color="#F3E651">.</font><font color="#7bc710">Wait</font><font color="#F3E651">()</font>
<font color="#ff0000"> </font><font color="#7bc710">close</font><font color="#F3E651">(</font><font color="#ff0000">numCh</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font><font color="#F3E651">()</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> _</font><font color="#F3E651">,</font><font color="#ff0000"> num </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><b><font color="#ffffff">range</font></b><font color="#ff0000"> a </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">go</font></b><font color="#ff0000"> </font><b><font color="#ffffff">func</font></b><font color="#F3E651">(</font><font color="#ff0000">num V</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> </font><b><font color="#ffffff">defer</font></b><font color="#ff0000"> wg</font><font color="#F3E651">.</font><font color="#7bc710">Done</font><font color="#F3E651">()</font>
<font color="#ff0000"> time</font><font color="#F3E651">.</font><font color="#7bc710">Sleep</font><font color="#F3E651">(</font><font color="#ff0000">time</font><font color="#F3E651">.</font><font color="#7bc710">Duration</font><font color="#F3E651">(</font><font color="#ff0000">num</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">*</font><font color="#ff0000"> time</font><font color="#F3E651">.</font><font color="#ff0000">Second</font><font color="#F3E651">)</font>
<font color="#ff0000"> numCh </font><font color="#F3E651"><-</font><font color="#ff0000"> num</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font><font color="#F3E651">(</font><font color="#ff0000">num</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> num </font><font color="#F3E651">:=</font><font color="#ff0000"> </font><b><font color="#ffffff">range</font></b><font color="#ff0000"> numCh </font><font color="#F3E651">{</font>
<font color="#ff0000"> sorted </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">append</font><font color="#F3E651">(</font><font color="#ff0000">sorted</font><font color="#F3E651">,</font><font color="#ff0000"> num</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> sorted</font>
<font color="#F3E651">}</font>
</pre>
<br />
<span>This Go code implements the sleep sort algorithm using generics and goroutines. The main function <span class='inlinecode'>Sleep[V ds.Integer](a ds.ArrayList[V]) ds.ArrayList[V]</span> takes a generic <span class='inlinecode'>ArrayList</span> as input and returns a sorted <span class='inlinecode'>ArrayList</span>. The code creates a separate goroutine for each element in the input array, sleeps for a duration proportional to the element's value, and then sends the element to a channel. Another goroutine waits for all the sleeping goroutines to finish and then closes the channel. The sorted result <span class='inlinecode'>ArrayList</span> is constructed by appending the elements received from the channel in the order they arrive. The <span class='inlinecode'>sync.WaitGroup</span> is used to synchronize goroutines and ensure that all of them have completed before closing the channel.</span><br />
<br />
<h3 style='display: inline' id='Testing'>Testing</h3><br />
<br />
<span>For testing, we only allow values up to 10, as otherwise, it would take too long to finish:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">package</font></b><font color="#ff0000"> sort</font>
<b><font color="#ffffff">import</font></b><font color="#ff0000"> </font><font color="#F3E651">(</font>
<font color="#ff0000"> </font><font color="#bb00ff">"fmt"</font>
<font color="#ff0000"> </font><font color="#bb00ff">"testing"</font>
<font color="#ff0000"> </font><font color="#bb00ff">"codeberg.org/snonux/algorithms/ds"</font>
<font color="#F3E651">)</font>
<b><font color="#ffffff">func</font></b><font color="#ff0000"> </font><font color="#7bc710">TestSleepSort</font><font color="#F3E651">(</font><font color="#ff0000">t </font><font color="#F3E651">*</font><font color="#ff0000">testing</font><font color="#F3E651">.</font><font color="#ff0000">T</font><font color="#F3E651">)</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> a </font><font color="#F3E651">:=</font><font color="#ff0000"> ds</font><font color="#F3E651">.</font><font color="#ff0000">NewRandomArrayList</font><font color="#F3E651">[</font><font color="#ff0000">int</font><font color="#F3E651">](</font><font color="#bb00ff">10</font><font color="#F3E651">,</font><font color="#ff0000"> </font><font color="#bb00ff">10</font><font color="#F3E651">)</font>
<font color="#ff0000"> a </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">Sleep</font><font color="#F3E651">(</font><font color="#ff0000">a</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000">a</font><font color="#F3E651">.</font><font color="#7bc710">Sorted</font><font color="#F3E651">()</font><font color="#ff0000"> </font><font color="#F3E651">{</font>
<font color="#ff0000"> t</font><font color="#F3E651">.</font><font color="#7bc710">Errorf</font><font color="#F3E651">(</font><font color="#bb00ff">"Array not sorted: %v"</font><font color="#F3E651">,</font><font color="#ff0000"> a</font><font color="#F3E651">)</font>
<font color="#ff0000"> </font><font color="#F3E651">}</font>
<font color="#F3E651">}</font>
</pre>
<br />
<span>As you can see, it takes <span class='inlinecode'>9s</span> here for the algorithm to finish (which is the highest value in the <span class='inlinecode'>ArrayList</span>):</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">❯ go </font><b><font color="#ffffff">test</font></b><font color="#ff0000"> </font><font color="#F3E651">.</font><font color="#ff0000">/sort -v -run SleepSort</font>
<font color="#F3E651">===</font><font color="#ff0000"> RUN TestSleepSort</font>
<font color="#ff0000">--- PASS</font><font color="#F3E651">:</font><font color="#ff0000"> TestSleepSort </font><font color="#F3E651">(</font><font color="#bb00ff">9</font><font color="#F3E651">.</font><font color="#ff0000">00s</font><font color="#F3E651">)</font>
<font color="#ff0000">PASS</font>
<font color="#ff0000">ok codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/algorithms/sort </font><font color="#bb00ff">9</font><font color="#F3E651">.</font><font color="#ff0000">002s</font>
</pre>
<br />
<span>I won't write any benchmark for sleep sort; that will be done for the algorithms to come in this series :-).</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>'Never split the difference' book notes</title>
<link href="gemini://foo.zone/gemfeed/2023-04-01-never-split-the-difference-book-notes.gmi" />
<id>gemini://foo.zone/gemfeed/2023-04-01-never-split-the-difference-book-notes.gmi</id>
<updated>2023-04-01T20:00:17+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>These are my personal takeaways after reading 'Never split the difference' by Chris Voss. Note that the book contains much more knowledge wisdom and that these notes only contain points I personally found worth writing down. This is mainly for my own use, but you might find it helpful too.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='Neversplitthedifferencebooknotes'>"Never split the difference" book notes</h1><br />
<br />
<span class='quote'>Published at 2023-04-01T20:00:17+03:00</span><br />
<br />
<span>These are my personal takeaways after reading "Never split the difference" by Chris Voss. Note that the book contains much more knowledge wisdom and that these notes only contain points I personally found worth writing down. This is mainly for my own use, but you might find it helpful too.</span><br />
<br />
<pre>
,.......... ..........,
,..,' '.' ',..,
,' ,' : ', ',
,' ,' : ', ',
,' ,' : ', ',
,' ,'............., : ,.............', ',
,' '............ '.' ............' ',
'''''''''''''''''';''';''''''''''''''''''
'''
</pre>
<br />
<h2 style='display: inline' id='Tacticallisteningspreadingempathy'>Tactical listening, spreading empathy</h2><br />
<br />
<span>Be a mirror, copy each other to be comfy with each other to build up trust. Mirroring is mainly body language. A mirror is to repeat the words the other just said. Simple but effective.</span><br />
<br />
<ul>
<li>A mirror needs space and silence between the words. At least 4 seconds.</li>
<li>A mirror might be awkward to be used at first, especially with a question coupled to it.</li>
<li>We fear what's different and are drawn to what is similar.</li>
</ul><br />
<span>Mirror training is like Jedi training. Simple but effective. A mirror needs space. Be silent after "you want this?" </span><br />
<br />
<h2 style='display: inline' id='Mindsetofdiscovery'>Mindset of discovery</h2><br />
<br />
<span>Try to have multiple realities in your mind and use facts to distinguish between real and false.</span><br />
<br />
<ul>
<li>Focus on what the counterpart has to say and what he needs and wants. Understanding him makes him vulnerable.</li>
<li>Empathy understanding the other person from his perspective, but it does not mean agreeing with him.</li>
<li>Detect and label the emotions of others for your powers. </li>
<li>To be understood seems to solve all problems magically.</li>
</ul><br />
<span>Try: to put a label on someone's emotion and then be silent. Wait for the other to reveal himself. "You seem unhappy about this?"</span><br />
<br />
<h3 style='display: inline' id='Moretips'>More tips </h3><br />
<br />
<ul>
<li>Put on a poker face and don't show emotions.</li>
<li>Slow things down. Don't be a problem solver.</li>
<li>Smile while you are talking, even on the phone. Be easy and encouraging.</li>
<li>Being right is not the key to successful negotiation; being mindful is.</li>
<li>Be in the safe zone of empathy and acknowledge bad news.</li>
</ul><br />
<h2 style='display: inline' id='Nostartstheconversation'>"No" starts the conversation</h2><br />
<br />
<span>When the opponent starts with a "no", he feels in control and comfortable. That's why he has to start with "no".</span><br />
<br />
<ul>
<li>"Yes" and "maybe" might be worthless, but "no" starts the conversation.</li>
<li>If someone is saying "no" to you, he will be open to what you have to say next.</li>
<li>"No" is not stopping the negotiation but will open up opportunities you were not thinking about before.</li>
<li>Start with "no". Great negotiators seek "no" because that's when the great discussions begin.</li>
<li>A "no" can be scary if you are not used to it. If your biggest fear is "no", then you can't negotiate.</li>
</ul><br />
<span>Get a "That's right" when negotiating. Don't get a "you're right". You can summarise the opponent to get a "that's right".</span><br />
<br />
<h2 style='display: inline' id='Winwin'>Win-win</h2><br />
<br />
<span>Win-win is a naive approach when encountering the win-lose counterpart, but always cooperate. Don't compromise, and don't split the difference. We don't compromise because it's right; we do it because it is easy. You must embrace the hard stuff; that's where the great deals are.</span><br />
<br />
<h2 style='display: inline' id='OnDeadlines'>On Deadlines</h2><br />
<br />
<ul>
<li>All deadlines are imaginary.</li>
<li>Most of the time, deadlines unsettle us without a good reason.</li>
<li>They push a deal to a conclusion.</li>
<li>They rush the counterpart to cause pressure and anxiety.</li>
</ul><br />
<h2 style='display: inline' id='Analysetheopponent'>Analyse the opponent</h2><br />
<br />
<ul>
<li>Understand the motivation of people behind the table as well.</li>
<li>Ask how affected they will be.</li>
<li>Determine your and the opposite negotiation style. Accommodation, analyst, assertive.</li>
<li>Treat them how they need to be treated.</li>
</ul><br />
<span>The person on the other side is never the issue; the problem is the issue. Keep this in mind to avoid emotional issues with the person and focus on the problem, not the person. The bond is essential; never create an enemy.</span><br />
<br />
<h2 style='display: inline' id='Usedifferentwaysofsayingno'>Use different ways of saying "no."</h2><br />
<br />
<span class='quote'>I had paid my rent always in time. I had positive experiences with the building and would be sad for the landlord to lose a good tenant. I am looking for a win-win agreement between us. Pulling out the research, other neighbours offer much lower prices even if your building is a better location and services. How can I effort 200 more.... </span><br />
<br />
<span>...then put an extreme anker.</span><br />
<br />
<span>You always have to embrace thoughtful confrontation for good negotiation and life. Don't avoid honest, clear conflict. It will give you the best deals. Compromises are mostly bad deals for both sides. Most people don't negotiate a win-win but a win-lose. Know the best and worst outcomes and what is acceptable for you.</span><br />
<br />
<h2 style='display: inline' id='Calibratedquestion'>Calibrated question</h2><br />
<br />
<span>Calibrated questions. Give the opponent a sense of power. Ask open-how questions to get the opponent to solve your problem and move him in your direction. Calibrated questions are the best tools. Summarise everything, and then ask, "how I am supposed to do that?". Asking for help this way with a calibrated question is a powerful tool for joint problem solving</span><br />
<br />
<span>Being calm and respectful is essential. Without control of your emotions, it won't work. The counterpart will have no idea how constrained they are with your question. Avoid questions which get a yes or short answers. Use "why?".</span><br />
<br />
<span>Counterparts are more involved if these are their solutions. The counterpart must answer with "that's right", not "you are right". He has to own the problem. If not, then add more why questions.</span><br />
<br />
<ul>
<li>Tone and body language need to align with what people are saying.</li>
<li>Deal with it via a labelled question. </li>
<li>Liers tend to talk with "them" and "their" and not with "I".</li>
<li>Also, liars tend to talk in complex sentences.</li>
</ul><br />
<span>Prepare 3 to 5 calibrated questions for your counterpart. Be curious what is really motivating the other side. You can get out the "Black Swan".</span><br />
<br />
<h2 style='display: inline' id='Theblackswan'>The black swan </h2><br />
<br />
<span>What we don't know can break our deal. Uncovering it can bring us unexpected success. You get what you ask for in this world, but you must learn to ask correctly. Reveal the black swan by asking questions.</span><br />
<br />
<h2 style='display: inline' id='More'>More</h2><br />
<br />
<span>Establish a range at top places like corp. I get... (e.g. remote London on a project basis). Set a high salary range and not a number. Also, check on LinkedIn premium for the salaries.</span><br />
<br />
<ul>
<li>Give an unexpected gift, e.g. show them my pet project and publicity for engineering.</li>
<li>Use an odd number, which makes you seem to have thought a lot about the sum and calculated it.</li>
<li>Define success and metrics for your next raise.</li>
<li>What does it take to be successful here? Ask the question, and they will tell you and guide you.</li>
<li>Set an extreme anker. Make the counterpart the illusion of losing something.</li>
<li>Hope-based deals. Hope is not a strategy.</li>
<li>Tactical empathy, listening as a martial art. It is emotional intelligence on steroids.</li>
<li>Being right isn't the key to a successful negotiation, but having the correct mindset is.</li>
<li>Don't shop the groceries when you are hungry.</li>
</ul><br />
<span>Slow.... it.... down....</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other book notes of mine are:</span><br />
<br />
<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 "The Pragmatic Programmer" book notes</a><br />
<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 "Never split the difference" book notes (You are currently reading this)</a><br />
<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 "The Obstacle is the Way" book notes</a><br />
<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes</a><br />
<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 "Mind Management" book notes</a><br />
<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 "Slow Productivity" book notes</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Gemtexter 2.0.0 - Let's Gemtext again²</title>
<link href="gemini://foo.zone/gemfeed/2023-03-25-gemtexter-2.0.0-lets-gemtext-again-2.gmi" />
<id>gemini://foo.zone/gemfeed/2023-03-25-gemtexter-2.0.0-lets-gemtext-again-2.gmi</id>
<updated>2023-03-25T17:50:32+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I proudly announce that I've released Gemtexter version `2.0.0`. What is Gemtexter? It's my minimalist static site generator for Gemini Gemtext, HTML and Markdown written in GNU Bash.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='Gemtexter200LetsGemtextagain'>Gemtexter 2.0.0 - Let's Gemtext again²</h1><br />
<br />
<span class='quote'>Published at 2023-03-25T17:50:32+02:00</span><br />
<br />
<pre>
-=[ typewriters ]=- 1/98
.-------.
_|~~ ~~ |_ .-------.
=(_|_______|_)= _|~~ ~~ |_
|:::::::::| =(_|_______|_)
|:::::::[]| |:::::::::|
|o=======.| |:::::::[]|
jgs `"""""""""` |o=======.|
mod. by Paul Buetow `"""""""""`
</pre>
<br />
<span>I proudly announce that I've released Gemtexter version <span class='inlinecode'>2.0.0</span>. What is Gemtexter? It's my minimalist static site generator for Gemini Gemtext, HTML and Markdown written in GNU Bash.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/gemtexter'>https://codeberg.org/snonux/gemtexter</a><br />
<br />
<span>This is a new major release, so it contains a breaking change (see "Meta cache made obsolete").</span><br />
<br />
<span>Let's list what's new!</span><br />
<br />
<h2 style='display: inline' id='Minimaltemplateengine'>Minimal template engine</h2><br />
<br />
<span>Gemtexter now supports templating, enabling dynamically generated content to <span class='inlinecode'>.gmi</span> files before converting anything to any output format like HTML and Markdown.</span><br />
<br />
<span>A template file name must have the suffix <span class='inlinecode'>gmi.tpl</span>. A template must be put into the same directory as the Gemtext <span class='inlinecode'>.gmi</span> file to be generated. Gemtexter will generate a Gemtext file <span class='inlinecode'>index.gmi</span> from a given template <span class='inlinecode'>index.gmi.tpl</span>. A <span class='inlinecode'><<<</span> and <span class='inlinecode'>>>></span> encloses a multiline template. All lines starting with <span class='inlinecode'><< </span> will be evaluated as a single line of Bash code and the output will be written into the resulting Gemtext file.</span><br />
<br />
<span>For example, the template <span class='inlinecode'>index.gmi.tpl</span>:</span><br />
<br />
<pre>
# Hello world
<< echo "> This site was generated at $(date --iso-8601=seconds) by \`Gemtexter\`"
Welcome to this capsule!
<<<
for i in {1..10}; do
echo Multiline template line $i
done
>>>
</pre>
<br />
<span>... results into the following <span class='inlinecode'>index.gmi</span> after running <span class='inlinecode'>./gemtexter --generate</span> (or <span class='inlinecode'>./gemtexter --template</span>, which instructs to do only template processing and nothing else):</span><br />
<br />
<pre>
# Hello world
> This site was generated at 2023-03-15T19:07:59+02:00 by `Gemtexter`
Welcome to this capsule!
Multiline template line 1
Multiline template line 2
Multiline template line 3
Multiline template line 4
Multiline template line 5
Multiline template line 6
Multiline template line 7
Multiline template line 8
Multiline template line 9
Multiline template line 10
</pre>
<br />
<span>Another thing you can do is insert an index with links to similar blog posts. E.g.:</span><br />
<br />
<pre>
See more entries about DTail and Golang:
<< template::inline::index dtail golang
Blablabla...
</pre>
<br />
<span>... scans all other post entries with <span class='inlinecode'>dtail</span> and <span class='inlinecode'>golang</span> in the file name and generates a link list like this:</span><br />
<br />
<pre>
See more entries about DTail and Golang:
=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD
=> ./2022-04-22-programming-golang.gmi 2022-04-22 The Golang Programming language
=> ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0
=> ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program (You are currently reading this)
Blablabla...
</pre>
<br />
<h2 style='display: inline' id='Addedhooks'>Added hooks</h2><br />
<br />
<span>You can configure <span class='inlinecode'>PRE_GENERATE_HOOK</span> and <span class='inlinecode'>POST_PUBLISH_HOOK</span> to point to scripts to be executed before running <span class='inlinecode'>--generate</span>, or after running <span class='inlinecode'>--publish</span>. E.g. you could populate some of the content by an external script before letting Gemtexter do its thing or you could automatically deploy the site after running <span class='inlinecode'>--publish</span>.</span><br />
<br />
<span>The sample config file <span class='inlinecode'>gemtexter.conf</span> includes this as an example now; these scripts will only be executed when they actually exist:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><b><font color="#ffffff">declare</font></b><font color="#ff0000"> -xr </font><font color="#ff0000">PRE_GENERATE_HOOK</font><font color="#F3E651">=.</font><font color="#ff0000">/pre_generate_hook</font><font color="#F3E651">.</font><font color="#ff0000">sh</font>
<b><font color="#ffffff">declare</font></b><font color="#ff0000"> -xr </font><font color="#ff0000">POST_PUBLISH_HOOK</font><font color="#F3E651">=.</font><font color="#ff0000">/post_publish_hook</font><font color="#F3E651">.</font><font color="#ff0000">sh</font>
</pre>
<br />
<h2 style='display: inline' id='UseofsaferBashoptions'>Use of safer Bash options</h2><br />
<br />
<span>Gemtexter now does <span class='inlinecode'>set -euf -o pipefile</span>, which helps to eliminate bugs and to catch scripting errors sooner. Previous versions only <span class='inlinecode'>set -e</span>.</span><br />
<br />
<h2 style='display: inline' id='Metacachemadeobsolete'>Meta cache made obsolete</h2><br />
<br />
<span>Here is the breaking change to older versions of Gemtexter. The <span class='inlinecode'>$BASE_CONTENT_DIR/meta</span> directory was made obsolete. <span class='inlinecode'>meta</span> was used to store various information about all the blog post entries to make generating an Atom feed in Bash easier. Especially the publishing dates of each post were stored there. Instead, the publishing date is now encoded in the <span class='inlinecode'>.gmi</span> file. And if it is missing, Gemtexter will set it to the current date and time at first run.</span><br />
<br />
<span>An example blog post without any publishing date looks like this:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> cat gemfeed</font><font color="#F3E651">/</font><font color="#bb00ff">2023</font><font color="#ff0000">-</font><font color="#bb00ff">02</font><font color="#ff0000">-</font><font color="#bb00ff">26</font><font color="#ff0000">-title-here</font><font color="#F3E651">.</font><font color="#ff0000">gmi</font>
<i><font color="#ababab"># Title here</font></i>
<font color="#ff0000">The remaining content of the Gemtext file</font><font color="#F3E651">...</font>
</pre>
<br />
<span>Gemtexter will add a line starting with <span class='inlinecode'>> Published at ...</span> now. Any subsequent Atom feed generation will then use that date.</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">%</font><font color="#ff0000"> cat gemfeed</font><font color="#F3E651">/</font><font color="#bb00ff">2023</font><font color="#ff0000">-</font><font color="#bb00ff">02</font><font color="#ff0000">-</font><font color="#bb00ff">26</font><font color="#ff0000">-title-here</font><font color="#F3E651">.</font><font color="#ff0000">gmi</font>
<i><font color="#ababab"># Title here</font></i>
<font color="#F3E651">></font><font color="#ff0000"> Published at </font><font color="#bb00ff">2023</font><font color="#ff0000">-</font><font color="#bb00ff">02</font><font color="#ff0000">-26T21</font><font color="#F3E651">:</font><font color="#bb00ff">43</font><font color="#F3E651">:</font><font color="#bb00ff">51</font><font color="#F3E651">+</font><font color="#bb00ff">01</font><font color="#F3E651">:</font><font color="#bb00ff">00</font>
<font color="#ff0000">The remaining content of the Gemtext file</font><font color="#F3E651">...</font>
</pre>
<br />
<h2 style='display: inline' id='XMLLintsupport'>XMLLint support</h2><br />
<br />
<span>Optionally, when the <span class='inlinecode'>xmllint</span> binary is installed, Gemtexter will perform a simple XML lint check against the Atom feed generated. This is a double-check of whether the Atom feed is a valid XML.</span><br />
<br />
<h2 style='display: inline' id='More'>More</h2><br />
<br />
<span>Additionally, there were a couple of bug fixes, refactorings and overall improvements in the documentation made. </span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-04-24-welcome-to-the-geminispace.html'>2021-04-24 Welcome to the Geminispace</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2022-08-27-gemtexter-1.1.0-lets-gemtext-again.html'>2022-08-27 Gemtexter 1.1.0 - Let's Gemtext again</a><br />
<a class='textlink' href='./2023-03-25-gemtexter-2.0.0-lets-gemtext-again-2.html'>2023-03-25 Gemtexter 2.0.0 - Let's Gemtext again² (You are currently reading this)</a><br />
<a class='textlink' href='./2023-07-21-gemtexter-2.1.0-lets-gemtext-again-3.html'>2023-07-21 Gemtexter 2.1.0 - Let's Gemtext again³</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>'The Pragmatic Programmer' book notes</title>
<link href="gemini://foo.zone/gemfeed/2023-03-16-the-pragmatic-programmer-book-notes.gmi" />
<id>gemini://foo.zone/gemfeed/2023-03-16-the-pragmatic-programmer-book-notes.gmi</id>
<updated>2023-03-16T00:55:20+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>These are my personal takeaways after reading 'The Pragmatic Programmer' by David Thomas and Andrew Hunt. Note that the book contains much more knowledge wisdom and that these notes only contain points I personally found worth writing down. This is mainly for my own use, but you might find it helpful too.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='ThePragmaticProgrammerbooknotes'>"The Pragmatic Programmer" book notes</h1><br />
<br />
<span class='quote'>Published at 2023-03-16T00:55:20+02:00</span><br />
<br />
<span>These are my personal takeaways after reading "The Pragmatic Programmer" by David Thomas and Andrew Hunt. Note that the book contains much more knowledge wisdom and that these notes only contain points I personally found worth writing down. This is mainly for my own use, but you might find it helpful too.</span><br />
<br />
<pre>
,.......... ..........,
,..,' '.' ',..,
,' ,' : ', ',
,' ,' : ', ',
,' ,' : ', ',
,' ,'............., : ,.............', ',
,' '............ '.' ............' ',
'''''''''''''''''';''';''''''''''''''''''
'''
</pre>
<br />
<span>Think about your work while doing it - every day on every project. Have a feeling of continuous improvement. </span><br />
<br />
<ul>
<li>Be a realist.</li>
<li>Smell challenges.</li>
<li>Care about your craft.</li>
<li>Code can always be flawed, but it can meet the requirements.</li>
<li>You should be proud of your code, though.</li>
</ul><br />
<span>No one writes perfect code, including you. However:</span><br />
<br />
<ul>
<li>Paranoia is good thinking.</li>
<li>Practice defensive programming and crash early.</li>
<li>Crashing is often the best thing you can do. </li>
<li>Changes should be reversible.</li>
</ul><br />
<span>Erlang: Defensive programming is a waste of time. Let it crash. "This can never happen" - don't practise that kind of self-deception when programming. </span><br />
<br />
<span>Leave assertions in the code, even in production. Only leave out the assertions causing the performance issues.</span><br />
<br />
<span>Take small steps, always. Get feedback, too, for each of the steps the code does. Avoid fortune telling. If you have to involve in it, then the step is too large.</span><br />
<br />
<span>Decouple the code (e.g. OOP or functional programming). Prefer interfaces for types and mixins for a class extension over class inheritance.</span><br />
<br />
<ul>
<li>Refactor now and not later.</li>
<li>Later, it will be even more painful.</li>
</ul><br />
<span>Don't think outside the box. Find the box. The box is more extensive than you think. Think about the hard problem at hand. Do you have to do it a certain way, or do you have to do it at all?</span><br />
<br />
<span>Do what works and not what's fashionable. E.g. does SCRUM make sense? The goal is to deliver deliverables and not to "become" agile.</span><br />
<br />
<h2 style='display: inline' id='Continuouslearning'>Continuous learning</h2><br />
<br />
<span>Add new tools to your repertoire every day and keep the momentum up. Learning new things is your most crucial aspect. Invest regularly in your knowledge portfolio. The learning process extends your thinking. It does not matter if you will never use it.</span><br />
<br />
<ul>
<li>Learn a new programming language every year.</li>
<li>Read a technical book every month.</li>
<li>Take courses.</li>
</ul><br />
<span>Think critically about everything you learn. Use paper for your notes. There is something special about it.</span><br />
<br />
<h2 style='display: inline' id='Stayconnected'>Stay connected</h2><br />
<br />
<span>It's your life, and you own it. Bruce Lee once said: </span><br />
<br />
<span class='quote'>"I am not on the world to life after your expectations, neither are you to life after mine."</span><br />
<br />
<ul>
<li>Go to meet-ups and actively engage.</li>
<li>Stay current.</li>
<li>Dealing with computers is hard. Dealing with people is harder. </li>
</ul><br />
<span>It's your life. Share it, celebrate it, be proud and have fun.</span><br />
<br />
<h2 style='display: inline' id='Thestoryofstonesoup'>The story of stone soup</h2><br />
<br />
<span>How to motivate others to contribute something (e.g. ideas to a startup):</span><br />
<br />
<span class='quote'>A kindly, old stranger was walking through the land when he came upon a village. As he entered, the villagers moved towards their homes, locking doors and windows. The stranger smiled and asked, why are you all so frightened. I am a simple traveler, looking for a soft place to stay for the night and a warm place for a meal. "There's not a bite to eat in the whole province," he was told. "We are weak and our children are starving. Better keep moving on." "Oh, I have everything I need," he said. "In fact, I was thinking of making some stone soup to share with all of you." He pulled an iron cauldron from his cloak, filled it with water, and began to build a fire under it. Then, with great ceremony, he drew an ordinary-looking stone from a silken bag and dropped it into the water. By now, hearing the rumor of food, most of the villagers had come out of their homes or watched from their windows. As the stranger sniffed the "broth" and licked his lips in anticipation, hunger began to overcome their fear. "Ahh," the stranger said to himself rather loudly, "I do like a tasty stone soup. Of course, stone soup with cabbage -- that's hard to beat." Soon a villager approached hesitantly, holding a small cabbage he'd retrieved from its hiding place, and added it to the pot. "Wonderful!!" cried the stranger. "You know, I once had stone soup with cabbage and a bit of salt beef as well, and it was fit for a king." The village butcher managed to find some salt beef . . . And so it went, through potatoes, onions, carrots, mushrooms, and so on, until there was indeed a delicious meal for everyone in the village to share. The village elder offered the stranger a great deal of money for the magic stone, but he refused to sell it and traveled on the next day. As he left, the stranger came upon a group of village children standing near the road. He gave the silken bag containing the stone to the youngest child, whispering to a group, "It was not the stone, but the villagers that had performed the magic." </span><br />
<br />
<span>By working together, everyone contributes what they can, achieving a greater good together.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other book notes of mine are:</span><br />
<br />
<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 "The Pragmatic Programmer" book notes (You are currently reading this)</a><br />
<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 "Never split the difference" book notes</a><br />
<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 "The Obstacle is the Way" book notes</a><br />
<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes</a><br />
<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 "Mind Management" book notes</a><br />
<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 "Slow Productivity" book notes</a><br />
<br />
<a class='textlink' href='../resources.html'>More books and other resources I found useful.</a><br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>How to shut down after work</title>
<link href="gemini://foo.zone/gemfeed/2023-02-26-how-to-shut-down-after-work.gmi" />
<id>gemini://foo.zone/gemfeed/2023-02-26-how-to-shut-down-after-work.gmi</id>
<updated>2023-02-26T23:48:01+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Do you need help fully discharging from work in the evenings or for the weekend? Shutting down from work won't just improve your work-life balance; it will also significantly improve the quality of your personal life and work. After a restful weekend, you will be much more energized and productive the next working day. So it should not just be in your own, but also your employers' interest that you fully relax and shut down after work. </summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>How to shut down after work</h1><br />
<br />
<span class='quote'>Published at 2023-02-26T23:48:01+02:00</span><br />
<br />
<pre>
|\ "Music should be heard not only with the ears, but also the soul."
|---|--\-----------------------|-----------------------------------------|
| | |\ | |@ |\ |
|---|---|--\-------------------|-------------/|----|------|--\----|------|
| @| | |\ |O | 3 / | |@ | | |
|---|--@|---|--\--------|------|---------/----|----|------|-------|------|
| @| @| \ |O | / | | |@ @| @|. |
|-----------|-----|-----|------|-----/---|---@|----|--------------|------|
| @| | |O | | | | @|. |
|-----------|----@|-----|------|----|---@|------------------------|------|
@| | | Larry Komro @|.
-@- [kom...@uwec.edu]
</pre>
<br />
<span>Do you need help fully discharging from work in the evenings or for the weekend? Shutting down from work won't just improve your work-life balance; it will also significantly improve the quality of your personal life and work. After a restful weekend, you will be much more energized and productive the next working day. So it should not just be in your own, but also your employers' interest that you fully relax and shut down after work. </span><br />
<br />
<h2 style='display: inline'>Have a shutdown routine</h2><br />
<br />
<span>Have a routine. Try to finish work around the same time every day. Write any outstanding tasks down for the next day, so you are sure you will remember them. Writing them down brings wonders as you can remove them from your mind for the remainder of the day (or the upcoming weekend) as you know you will surely pick them up the next working day. Tidying up your workplace could also count toward your daily shutdown routine. </span><br />
<br />
<span>A commute home from the office also greatly helps, as it disconnects your work from your personal life. Don't work on your commute home, though! If you don't commute but work from home, then it helps to walk around the block or in a nearby park to disconnect from work. </span><br />
<br />
<h2 style='display: inline'>Don't work when you officially don't work</h2><br />
<br />
<span>Unless you are self-employed, you have likely signed an N-hour per week contract with your employer, and your regular working times are from X o'clock in the morning to Y o'clock in the evening (with M minutes lunch break in the middle). And there might be some flexibility in your working times, too. But that kind of flexibility (e.g. extending the lunch break so that there is time to pick up a family member from the airport) will be agreed upon, and you will counteract it, for example, by starting working earlier the next day or working late, that one exception. But overall, your weekly working time will stay N hours. </span><br />
<br />
<span>Another exception would be when you are on an on-call schedule and are expected to watch your work notifications out-of-office times. But that is usually only a few days per month and, therefore, not the norm. And it should also be compensated accordingly. </span><br />
<br />
<span>There might be some maintenance work you must carry out, which can only be done over the weekend, but it should be explicitly agreed upon and compensated for. Also, there might be a scenario that a production incident comes up shortly before the end of the work day, requiring you (and your colleagues) to stay a bit longer. But this should be an exceptional case.</span><br />
<br />
<span>Other than that, there is no reason why you should work out-of-office hours. I know many people who suffer "the fear of missing out", so slack messages and E-Mails are checked until late in the evening, during weekends or holidays. I have been improving here personally a lot over the last couple of months, but still, I fall into this trap occasionally. </span><br />
<br />
<span>Also, when you respond to slack messages and E-Mails, your colleagues can think that you have nothing better to do. They also will take it for granted and keep slacking and messaging you out of regular office times. </span><br />
<br />
<span>Checking for your messages constantly outside of regular office times makes it impossible to shut down and relax from work altogether. </span><br />
<br />
<h2 style='display: inline'>Distract your mind</h2><br />
<br />
<span>Often, your mind goes back to work-related stuff even after work. That's normal as you concentrated highly on your work throughout the day. The brain unconsciously continues to work and will automatically present you with random work-related thoughts. You can counteract this by focusing on non-work stuff, which may include:</span><br />
<br />
<ul>
<li>Exercise. A half an hour workout or yoga session, followed by some stretching, helps to calm your mind after work. </li>
<li>Play (with your family, pets, friends, or video game)</li>
<li>Mindfully listen to music. When have you ever "really" listened to music? I mean, not just as a background stimulation but really paid attention to the melody, rhythm, voice and lyrics? That requires focused attention and distracts you from other thoughts. </li>
<li>Think of or work on that fun passion project. I currently, for example, like to learn and code a bit in Rakulang. </li>
<li>Read. Nothing beats reading a good Science Fiction Novel (or whatever you prefer) before falling asleep.</li>
</ul><br />
<span>Some of these can be habit-stacked: Exercise could be combined with watching videos about your passion project (e.g. watching lectures about that new programming language you are currently learning for fun). With walking, for example, you could combine listening to an Audiobook or music, or you could also think about your passion project during that walk. </span><br />
<br />
<h2 style='display: inline'>Get a pet</h2><br />
<br />
<span>Even if you have children, it helps wonders to get a pet. My cat, for example, will remind me a few times daily to take a few minute's breaks to pet, play or give food. So my cat not only helps me after work but throughout the day.</span><br />
<br />
<span>My neighbour also works from home, and he has dogs, which he regularly has to take out to the park.</span><br />
<br />
<h2 style='display: inline'>Journal your day</h2><br />
<br />
<span>If you are upset about something, making it impossible to shut down from work, write down everything (e.g., with a pen in a paper journal). Writing things down helps you to "get rid" of the negative. Especially after conflicts with colleagues or company decisions, you don't agree on. This kind of self-therapy is excellent. Brainstorm all your emotions and (even if opinionated) opinions so you have everything on paper. Once done, you don't think about it so much anymore, as you know you can access that information if required. But stopping ruminating about it will be much easier now. You will likely never access that information again, though. But at least writing the thoughts down saved your day. </span><br />
<br />
<span>Write down three things which went well for the day. This helps you to appreciate the day. </span><br />
<br />
<h2 style='display: inline'>Don't stress about what your employer expects from you</h2><br />
<br />
<span>Think about what's fun and motivates you. Maybe the next promotion to Principal or a Manager role isn't for you. Many fall into the trap of stressing themselves out to satisfy the employer so that the next upgrade will happen and think about it constantly, even after work. But it is more important that you enjoy your craftsmanship. Work on what you expect from yourself. Ideally, your goals should be aligned with your employer. I am not saying you should abandon everything what your manager is asking you to do, but it is, after all, your life. And you have to decide where and on what you want to work. But don't sell yourself short. Keep track of your accomplishments.</span><br />
<br />
<h2 style='display: inline'>Call it a day</h2><br />
<br />
<span>Every day you gave your best was good; the day's outcome doesn't matter. What matters is that you know you gave your best and are closer to your goals than the previous day. This gives you a sense of progress and accomplishment.</span><br />
<br />
<span>There are some days at work you feel drained afterwards and think you didn't progress towards your goals at all. It's more challenging to shut down from work after such a day. A quick hack is to work on a quick win before the end of the day, giving you a sense of accomplishment after all. Another way is to make progress on your fun passion project after work. It must not be work-related, but a sense of accomplishment will still be there.</span><br />
<span> </span><br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Why GrapheneOS rox</title>
<link href="gemini://foo.zone/gemfeed/2023-01-23-why-grapheneos-rox.gmi" />
<id>gemini://foo.zone/gemfeed/2023-01-23-why-grapheneos-rox.gmi</id>
<updated>2023-01-23T15:31:52+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Art by Joan Stark</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>Why GrapheneOS rox</h1><br />
<br />
<span class='quote'>Published at 2023-01-23T15:31:52+02:00</span><br />
<br />
<pre>
Art by Joan Stark
_.===========================._
.'` .- - __- - - -- --__--- -. `'.
__ / ,'` _|--|_________|--|_ `'. \
/'--| ; _.'\ | ' ' | /'._ ; |
// | |_.-' .-'.' ___ '.'-. '-._| |
(\) \"` _.-` / .-'`_ `'-. \ `-._ `"/
(\) `-' | .' .-'" "'-. '. | `-`
(\) | / .'(3)(2)(1)'. \ |
(\) | / / (4) .-. \ \ |
(\) | | |(5) ( )'==,J | |
(\) | \ \ (6) '-' (0) / / |
(\) | \ '.(7)(8)(9).' / |
(\) ___| '. '-.._..-' .' |
(\) /.--| '-._____.-' |
(\) (\) |\_ _ __ _ __ __/|
(\) (\) | |
(\)_._._.__(\) | |
(\\\\jgs\\\) '.___________________.'
'-'-'-'--'
</pre>
<br />
<span>In 2021 I wrote "On Being Pedantic about Open-Source", and there was a section "What about mobile?" where I expressed the dilemma about the necessity of using proprietary mobile operating systems. With GrapheneOS, I found my perfect solution for personal mobile phone use. </span><br />
<br />
<a class='textlink' href='./2021-08-01-on-being-pedantic-about-open-source.html'>On Being Pedantic about Open-Source</a><br />
<br />
<span>What is GrapheneOS?</span><br />
<br />
<span class='quote'>GrapheneOS is a privacy and security-focused mobile OS with Android app compatibility developed as a non-profit open-source project. It's focused on the research and development of privacy and security technologies, including substantial improvements to sandboxing, exploits mitigations and the permission model.</span><br />
<br />
<span>GrapheneOS is an independent Android distribution based on the Android Open Source Project (AOSP) but hardened in multiple ways. Other independent Android distributions, like LineageOS, are also based on AOSP, but GrapheneOS takes it further so that it can be my daily driver on my phone.</span><br />
<br />
<a class='textlink' href='https://GrapheneOS.org'>https://GrapheneOS.org</a><br />
<a class='textlink' href='https://LineageOS.org'>https://LineageOS.org</a><br />
<br />
<h2 style='display: inline'>User Profiles</h2><br />
<br />
<span>GrapheneOS allows configuring up to 32 user profiles (including a guest profile) on a single phone. A profile is a completely different environment within the phone, and it is possible to switch between them instantly. Sessions of a profile can continue running in the background or be fully terminated. Each profile can have completely different settings and different applications installed.</span><br />
<br />
<span>I use my default profile with primarily open-source applications installed, which I trust. I use another profile for banking (PayPal, various proprietary bank apps, Amazon store app, etc.) and another profile for various Google services (which I try to avoid, but I have to use once in a while). Furthermore, I have configured a profile for Social Media use (that one isn't in my default profile, as otherwise I am tempted to scroll social media all the time, which I try to avoid and only want to do intentionally when switching to the corresponding profile!).</span><br />
<br />
<span>The neat thing about the profiles is that some can run a sandboxed version of Google Play (see later in this post), while others don't. So some profiles can entirely operate without any Google Play, and only some profiles (to which I rarely switch) have Google Play enabled. </span><br />
<br />
<span>You notice how much longer (multiple days) your phone can be on a single charge when Google Play Services isn't running in the background. This tells a lot about the background activities and indicates that using Google Play shouldn't be the norm.</span><br />
<br />
<h2 style='display: inline'>Proxying some of the Google offerings </h2><br />
<br />
<span>There's also the case that I am using an app from the Google Play store (as the app isn't available from F-Droid), which doesn't require Google Play Services to run in the background. Here's where I use the Aurora Android store. The Aurora store can be installed through F-Droid. Aurora acts as an anonymous proxy from your phone to the Google Play Store and lets you install apps from there. No Google credentials are required for that!</span><br />
<br />
<a class='textlink' href='https://f-droid.org'>https://f-droid.org</a><br />
<br />
<span>There's a similar solution for watching videos on YouTube. You can use the NewPipe app (also from F-Droid), which acts as an anonymous proxy for watching videos from YouTube. So there isn't any need to install the official YouTube app, and there isn't any need to login to your Google account. What's so bad about the official app? You don't know which data it is sending about you to Google, so it is a privacy concern. </span><br />
<br />
<h2 style='display: inline'>Google Play Sandboxing </h2><br />
<br />
<span>Before switching to GrapheneOS, I had been using LineageOS on one of my phones for a couple of years. Still, I always had to have a secondary personal phone with all of these proprietary apps which (partially) only work with Google Play on the phone (e.g. Banking, Navigation, various travel apps from various Airlines, etc.) somewhere around as I didn't install Google Play on my LineageOS phone due to privacy concerns and only installed apps from the F-Droid store on it. When travelling, I always had to carry around a second phone with Google Play on it, as without it; life would become inconvenient pretty soon. </span><br />
<br />
<span>With GrapheneOS, it is different. Here, I do not just have a separate user profile, "Google", for various Google apps where Google Play runs, but Google Play also runs in a sandbox!!!</span><br />
<br />
<span class='quote'>GrapheneOS has a compatibility layer providing the option to install and use the official releases of Google Play in the standard app sandbox. Google Play receives no special access or privileges on GrapheneOS instead of bypassing the app sandbox and receiving a massive amount of highly privileged access. Instead, the compatibility layer teaches it how to work within the full app sandbox. It also isn't used as a backend for the OS services as it would be elsewhere since GrapheneOS doesn't use Google Play even when it's installed.</span><br />
<br />
<span>When I need to access Google Play, I can switch to the "Google" profile. Even there, Google is sandboxed to the absolute minimum permissions required to be operational, which gives additional privacy protection.</span><br />
<br />
<span>The sad truth is that Google Maps is still the best navigation app. When driving unknown routes, I can switch to my Google profile to use Google Maps. I don't need to do that when going streets I know about, but it is crucial (for me) to have Google Maps around when driving to a new destination.</span><br />
<br />
<span>Also, Google Translate and Google Lens are still the best translation apps I know. I just recently relocated to another country, where I am still learning the language, so Google Lens has been proven very helpful on various occasions by ad-hoc translating text into English or German for me.</span><br />
<br />
<span>The same applies to banking. Many banking apps require Google Play to be available (It might be even more secure to only use banking apps from the Google Play store due to official support and security updates). I rarely need to access my mobile banking app, but once in a while, I need to. As you have guessed by now, I can switch to my banking profile (with Google Play enabled), do what I need to do, and then terminate the session and go back to my default profile, and then my life can go on :-). </span><br />
<br />
<span>It is great to have the flexibility to use any proprietary Android app when needed. That only applies to around 1% of my phone usage time, but you often don't always know when you need "that one app now". So it's perfect that it's covered with the phone you always have with you. </span><br />
<br />
<h2 style='display: inline'>The camera and the cloud </h2><br />
<br />
<span>I really want my phone to shoot good looking pictures, so that I can later upload them to the Irregular Ninja:</span><br />
<br />
<a class='textlink' href='https://irregular.ninja'>https://irregular.ninja</a><br />
<br />
<span>The stock camera app of the OASP could be better. Photos usually look washed out, and the app lacks features. With GrapheneOS, there are two options:</span><br />
<br />
<ul>
<li>Use the official Google camera app with sandboxed Google Play Services running. You will get the full Google experience here.</li>
<li>Or, just use the default GrapheneOS camera app.</li>
</ul><br />
<span>The GrapheneOS camera app is much better than the stock OASP camera app. I have been comparing the photo quality of my Pixel phone under LineageOS and GrapheneOS, and the differences are pronounced. I didn't compare the quality with the official Google camera app, but I have seen some comparison videos and the differences seem like they aren't groundbreaking. </span><br />
<br />
<span>For automatic backups of my photos, I am relying on a self-hosted instance of NextCloud (with a client app available via F-Droid). So there isn't any need to rely on any Google apps and services (Google Play Photos or Google Camera app) anymore, and that's great!</span><br />
<br />
<a class='textlink' href='https://nextcloud.com'>https://nextcloud.com</a><br />
<br />
<span>I also use NextCloud to synchronize my notes (NextCloud Notes), my RSS news feeds (NextCloud News) and contacts (DAVx5). All apps required are available in the F-Droid store.</span><br />
<br />
<h2 style='display: inline'>Fine granular permissions</h2><br />
<br />
<span>Another great thing about GrapheneOS is that, besides putting your apps into different profiles, you can also restrict network access and configure storage scopes per app individually.</span><br />
<br />
<span>For example, let's say you are installing that one proprietary app from the Google Play Store through the Aurora store, and then you want to ensure that the app doesn't send data "home" through the internet. Nothing is easier to do than that. Just remove network access permissions from that only app.</span><br />
<br />
<span>The app also wants to store and read some data from your phone (e.g. it could be a proprietary app for enhancing photos, and therefore storage access to a photo folder would be required). In GrapheneOS, you can configure a storage scope for that particular app, e.g. only read and write from one folder but still forbid access to all other folders on your phone.</span><br />
<br />
<h2 style='display: inline'>Termux</h2><br />
<br />
<span>Termux can be installed on any Android phone through F-Droid, so it doesn't need to be a GrapheneOS phone. But I have to mention Termux here as it significantly adds value to my phone experience. </span><br />
<br />
<span class='quote'>Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required. A minimal base system is installed automatically - additional packages are available using the APT package manager.</span><br />
<br />
<a class='textlink' href='https://termux.dev'>https://termux.dev</a><br />
<br />
<span>In short, Termux is an entire Linux environment running on your Android phone. Just pair your phone with a Bluetooth keyboard, and you will have the whole Linux experience. I am only using terminal Linux applications with Termux, though. What makes it especially great is that I could write on a new blog post (in Neovim through Termux on my phone) or do some coding whilst travelling (e.g. during a flight), or look up my passwords or some other personal documents (through my terminal-based password manager). All changes I commit to Git can be synced to the server with a simple <span class='inlinecode'>git push</span> once online (e.g. after the plane landed) again.</span><br />
<br />
<span>There are Pixel phones with a screen size of 6", and that's decent enough for occasional use like that, and everything (the phone, the BT keyboard, maybe an external battery pack) all fit nicely in a small travel pocket.</span><br />
<br />
<h2 style='display: inline'>So, why not use a pure Linux phone?</h2><br />
<br />
<span>Strictly speaking, an Android phone is a Linux phone, but it's heavily modified and customized. For me, a "pure" Linux phone is a more streamlined Linux kernel running in a distribution like Ubuntu Touch or Mobian. </span><br />
<br />
<span>A pure Linux phone, e.g. with Ubuntu Touch installed, e.g. on a PinePhone, Fairphone, the Librem 5 or the Volla phone, is very appealing to me. And they would also provide an even better Linux experience than Termux does. Some support running LineageOS within an Anbox, enabling you to run various proprietary Android apps occasionally within Linux.</span><br />
<br />
<a class='textlink' href='https://ubuntu-touch.io/'>Ubuntu Touch</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Linux_for_mobile_devices'>More Linux distributions for mobile devices </a><br />
<br />
<span>But here, Google Play would not be sandboxed; you could not configure individual network permissions and storage scopes like in GrapheneOS. Pure Linux-compatible phones usually come with a crappy camera, and the battery life is generally pretty bad (only a few hours). Also, no big tech company pushes the development of Linux phones. Everything relies on hobbyists, whereas multiple big tech companies put a lot of effort into the Android project, and a lot of code also goes into the Android Open-Source project. </span><br />
<br />
<span>Currently, pure Linux phones are only a nice toy to tinker with but are still not ready (will they ever?) to be the daily driver. SailfishOS may be an exception; I played around with it in the past. It is pretty usable, but it's not an option for me as it is partial a proprietary operating system.</span><br />
<br />
<a class='textlink' href='https://sailfishos.org'>SailfishOS</a><br />
<br />
<h2 style='display: inline'>Small GrapheneOS downsides </h2><br />
<br />
<span>Sometimes, switching a profile to use a different app is annoying, and you can't copy and paste from the system clipboard from one profile to another. But that's a small price I am willing to pay!</span><br />
<br />
<span>Another thing is that GrapheneOS can only run on Google Pixel phones, whereas LineageOS can be installed on a much larger variety of hardware. But on the other hand, GrapheneOS works very well on Pixel phones. The GrapheneOS team can concentrate their development efforts on a smaller set of hardware which then improves the software's quality (best example: The camera app).</span><br />
<br />
<span>And, of course, GrapheneOS is an open-source project. This is a good thing; however, on the other side, nobody can guarantee that the OS will not break or will not damage your phone. You have to trust the GrapheneOS project and donate to the project so they can keep up with the great work. But I rather trust the GrapheneOS team than big tech. </span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>(Re)learning Java - My takeaways</title>
<link href="gemini://foo.zone/gemfeed/2022-12-24-ultrarelearning-java-my-takeaways.gmi" />
<id>gemini://foo.zone/gemfeed/2022-12-24-ultrarelearning-java-my-takeaways.gmi</id>
<updated>2022-12-24T23:18:40+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>As a regular participant in the annual Pet Project competition at work, I always try to find a project where I can learn something new. In this post, I would like to share my takeaways after revisiting Java. You can read about my motivations in my 'Creative universe' post:</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>(Re)learning Java - My takeaways</h1><br />
<br />
<span class='quote'>Published at 2022-12-24T23:18:40+02:00</span><br />
<br />
<a href='./2022-12-24-ultrarelearning-java-my-takeaways/learnjava.jpg'><img src='./2022-12-24-ultrarelearning-java-my-takeaways/learnjava.jpg' /></a><br />
<br />
<span>As a regular participant in the annual Pet Project competition at work, I always try to find a project where I can learn something new. In this post, I would like to share my takeaways after revisiting Java. You can read about my motivations in my "Creative universe" post:</span><br />
<br />
<a class='textlink' href='./2022-04-10-creative-universe.html'>Creative universe</a><br />
<br />
<span>I have been programming in Java back in the days as a university student, and even my Diploma Thesis I implemented in Java (it would require some overhaul so that it is fully compatible with a recent version of Java, though - It still compiles and runs, but with a lot of warnings, though!):</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/vs-sim'>VS-Sim: Distributed systems simulator</a><br />
<br />
<span>However, after that, I became a Linux Sysadmin and mainly continued programming in Perl, Puppet, bash, and a little Python. For personal use, I also programmed a bit in Haskell and C. After my Sysadmin role, I moved to London and became a Site Reliability Engineer (SRE), where I mainly programmed in Ruby, bash, Puppet and Golang and a little bit of C. </span><br />
<br />
<span>At my workplace, as an SRE, I don't do Java a lot. I have been reading Java code to understand the software better so I can apply and suggest workarounds or fixes to existing issues and bugs. However, most of our stack is in Java, and our Software Engineers use Java as their primary programming language.</span><br />
<br />
<h2 style='display: inline'>Stuck at Java 1.4</h2><br />
<br />
<span>Over time, I had been missing out on many new features that were added to the language since Java 1.4, so I decided to implement my next Pet Project in Java and learn every further aspect of the language as my main goal. Of course, I still liked the idea of winning a Pet Project Prize, but my main objective was to level up my Java skills.</span><br />
<br />
<h2 style='display: inline'>(Re)learning & upskilling to Java 18</h2><br />
<br />
<h3 style='display: inline'>Effective Java</h3><br />
<br />
<span>This book was recommended by my brother and also by at least another colleague at work to be one of the best, if not the best, book about Java programming. I read the whole book from the beginning to the end and immersed myself in it. I fully agree; this is a great book. Every Java developer or Java software engineer should read it!</span><br />
<br />
<a href='./2022-12-24-ultrarelearning-java-my-takeaways/effective-java.jpg'><img src='./2022-12-24-ultrarelearning-java-my-takeaways/effective-java.jpg' /></a><br />
<br />
<span>I recommend reading the 90-part effective Java Series on <span class='inlinecode'>dev.to</span>. It's a perfect companion to the book as it explains all the chapters again but from a slightly different perspective and helps you to really understand the content.</span><br />
<br />
<a class='textlink' href='https://dev.to/kylec32/series/2292'>Kyle Carter's 90-part Effective Java Series </a><br />
<br />
<h3 style='display: inline'>Java Pub House</h3><br />
<br />
<span>During my lunch breaks, I usually have a walk around the block or in a nearby park. I used that time to listen to the Java Pub House podcast. I listened to *every* episode and learned tons of new stuff. I can highly recommend this podcast. Especially GraalVM, a high-performance JDK distribution written for Java and other JVM languages, captured my attention. GraalVM can compile Java code into native binaries, improving performance and easing the distribution of Java programs. Because of the latter, I should release a VS-Sim GraalVM edition one day through a Linux AppImage ;-).</span><br />
<br />
<a class='textlink' href='https://www.javapubhouse.com'>https://www.javapubhouse.com</a><br />
<a class='textlink' href='https://www.graalvm.org'>https://www.graalvm.org</a><br />
<br />
<h3 style='display: inline'>Java Concurrency course</h3><br />
<br />
<span>I also watched a course on O'Reilly Safari Books online about Java Concurrency. That gave an excellent refresher on how the Java thread pools work and what were the concurrency primitives available in the standard library.</span><br />
<br />
<h3 style='display: inline'>Read a lot of Java code</h3><br />
<br />
<span>First, the source code is often the best documentation (if programmed nicely), and second, it helps to get the hang of the language and standard practices. I started to read more and more Java code at work. I did that whenever I had to understand how something, in particular, worked (e.g. while troubleshooting and debugging an issue). </span><br />
<br />
<h3 style='display: inline'>Observed Java code reviews</h3><br />
<br />
<span>Another great way to get the hang of Java again was to sneak into the code reviews of the Software Engineer colleagues. They are the expert on the matter and are a great source to copy knowledge. It's OK to stay passive and only follow the reviews. Sometimes, it's OK to step up and take ownership of the review. The developers will also always be happy to answer any naive questions which come up.</span><br />
<br />
<h3 style='display: inline'>Took ownership of a roadmap-Java project</h3><br />
<br />
<span>Besides my Pet Project, I also took ownership of a regular roadmap Java project at work, making an internal Java service capable of running in Kubernetes. This was a bunch of minor changes and adding a bunch of classes and unit tests dealing with the statelessness and a persistent job queue in Redis. The job also involved reading and understanding a lot of already existing Java code. It wasn't part of my job description, but it was fun, and I learned a lot. The service runs smoothly in production now. Of course, all of my code got reviewed by my Software Engineering colleagues.</span><br />
<br />
<h2 style='display: inline'>The good</h2><br />
<br />
<span>From the new language features and syntaxes, there are many personal takeaways, and I can't possibly list them all, but here are some of my personal highlights:</span><br />
<br />
<ul>
<li>Static factory methods and public constructors both have their uses, and it pays to understand their relative merits. Often static factories are preferable (cleaner and easier to read), so avoid the reflex to provide public constructors without first considering static factories.</li>
<li>Java streams were utterly new to me. I love how they can help to produce more compact code. But it's challenging to set the line of when enough is enough. Overusing streams can have the opposite effect: Code becomes more complex and challenging to understand. And it is so easy to parallelize the computation of streams by "just" marking the stream as <span class='inlinecode'>.parallel()</span> (more on that later in this post).</li>
<li>Overall, object-oriented languages tend to include more and more functional paradigms. The functional interfaces, which Java provides now, are fantastic. Their full powers shine in combination with the use of streams. An entire book can be written about Java functional interfaces, so I leave it to you to do any further digging.</li>
<li>Local type inference help to reduce even more boilerplate code. E.g. instead of <span class='inlinecode'>Hash<String,Hash<String,String>> foo = new Hash<String,Hash<String,String>>();</span> it's possible to just write <span class='inlinecode'>var foo = new Hash<String,Hash<String,String>>();</span></li>
<li>Class inheritance isn't the preferred way anymore to structure reusable code. Now, it's composition over inheritance. E.g. use dependency injection (inject one object to another object through its constructor) or prefer interfaces (which now also support default implementations of methods) over class inheritance. This makes sense to me as I do that already when I program in Ruby. </li>
<li>I learned the <span class='inlinecode'>try-with-resources</span> pattern. Very useful in ensuring closing resources again correctly. No need anymore for complicated and nested <span class='inlinecode'>finally</span>-blocks, which used to be almost impossible to get right previously in case of an error condition (e.g. I/O error somewhere deeply nested in an input or output stream).</li>
<li>Optimize only when required. It's considered to be cleaner to prefer immutable variables (declaring them as <span class='inlinecode'>final</span>). I knew that already, but for Java, it always seemed to be a waste of resources (creating entirely new objects whenever states change), but apparently, it's okay. Java also does many internal tricks for performance optimization here, e.g. interning strings.</li>
<li>I learned about the concept of static member classes and the difference between non-static member classes (also sometimes known as inner classes). Non-static member classes have full access to all members of their outer class (think of closure). In contrast, static member classes act like completely separate classes without such access but provide the benefit of a nested name that can help group functionality in the code.</li>
<li>I learned about the existence of thread-local variables. These are only available to the current thread and aren't shared with other threads.</li>
<li>I learned about the concept of Java modules, which help to structure larger code bases better. The traditional Java packages are different. </li>
<li>I learned to love the new <span class='inlinecode'>Optional</span> type. I already knew the concept from Haskell, where <span class='inlinecode'>Maybe</span> would be the corresponding type. <span class='inlinecode'>Optional</span> helps to avoid <span class='inlinecode'>null</span>-pointers but comes with some (minimal) performance penalty. So, in the end, you end up with both <span class='inlinecode'>Optional</span> types and <span class='inlinecode'>null</span>-pointers in your code (depending on the requirements). But I like to prefer <span class='inlinecode'>Optional</span> over <span class='inlinecode'>null</span>-pointer when "no result" is a valid return value from a method.</li>
<li>The <span class='inlinecode'>enum</span> type is way more powerful than I thought. Initially, I felt an <span class='inlinecode'>enum</span> could only be used to define a list of constants and then to compare an instance to another instance of the same. An <span class='inlinecode'>enum</span> is still there to define a list of constants, but it's also almost like a <span class='inlinecode'>class</span> (you can implement constructors, and methods, inherit from other enums). There are quite a lot of possible use cases.</li>
<li>A small but almost the most helpful thing I learned is always to use the <span class='inlinecode'>@Override</span> annotation when overriding a method from a parent class. If done, Java helps to detect any typos or type errors when overriding methods. That's useful and spares a lot of time debugging where a method was mistakenly overloaded but not overridden.</li>
<li>Lambdas are much cleaner, shorter and easier to read than anonymous classes. Many Java libraries require passing instances of (anonymous) classes (e.g. in Swing) to other objects. Lambdas are so lovely because they are primarily compatible with the passing of anonymous classes, so they are a 1:1 replacement in many instances. Lambdas also play very nicely together with the Java functional interfaces, as each Lambda got a type, and the type can be an already existing functional interface (or, if you got a particular case, you could define your custom functional interface for your own set of Lambdas, of course).</li>
<li>I love the concept of Java records. You can think of a record as an immutable object holding some data (as members). They are ideal for pipe and stream processing. They are much easier to define (with much less boilerplate) and come with write protection out of the box.</li>
</ul><br />
<h2 style='display: inline'>The bad and the ugly</h2><br />
<br />
<span>There are also many ugly corners in Java. Many are doomed to stay there forever due to historical decisions and ensuring backward compatibility with older versions of the Java language and the Java standard library. </span><br />
<br />
<ul>
<li>Finalizers and cleaners seem obsolete, fragile and still, you can use them.</li>
<li>In many cases, extreme caution needs to be taken to minimize the accessibility of class members. You might think that Java provides the best "out-of-the-box" solution for proper encapsulation, but the language has many loopholes.</li>
<li>In the early days, Java didn't support generics yet. So what you would use is to cast everything to <span class='inlinecode'>Object</span>. Java now fully supports generics (for a while already), but you can still cast everything to <span class='inlinecode'>Object</span> and back to whatever type you want. That can lead to nasty runtime errors. Also, there's a particular case to convert between an Array of Object to an Array of String or from an Array of String to a List of String. Java can't convert between these types automatically, and extreme caution needs to be taken when enforcing so (e.g. through explicit type casts). In many of these cases, Java would print out warnings that need to be manually suppressed via annotations. Programming that way, converting data between old and new best practices, is clunky.</li>
<li>If you don't know what you do, Java streams can be all wrong. Side effects in functions used in streams can be nasty to debug. Also, don't just blindly add a <span class='inlinecode'>.parallel()</span> to your stream. You need to understand what the stream does and how it exactly works; otherwise, parallelizing a stream can impact the performance drastically (in a negative way). There need to be language constructs preventing you from doing the wrong things. That's so much easier to do it right in a purely functional programming language like Haskell.</li>
<li>Java is a pretty old language (already), so there are many obstacles to consider. There are too many exceptions and different outcomes of how Java code can behave. In most cases, when you write an API, every method you program needs to be documented so the user won't encounter any surprises using your code. Writing and reading a lot of documentation seems to be quite the overhead when the method name is already descriptive.</li>
<li>Java serialization is broken. It works, and the language still supports it, but you better not use Java's native way of object serialization and deserialization. Unbelievable how much can get wrong here, especially regarding security (injecting arbitrary code).</li>
<li>Being a bit spoiled by Golang's Goroutines, I was shocked about the limitations of the Java threads. They are resource hungry, and you can't just spin up millions of them as you would with Goroutines. I knew this limitation of threads already (as it's not a problem of the language but of how threads work in the OS), but still, I was pretty shocked when I got reminded of them again. Of course, there's a workaround: Use asynchronous sockets so that you don't waste a whole thread on a single I/O operation (in my case, waiting for a network response). Golang's runtime does that automatically for you: An OS thread will be re-used for other tasks until the network socket unblocks. Every modern programming language should support lightweight threads or Coroutines like Go's Goroutines. </li>
</ul><br />
<br />
<h2 style='display: inline'>Conclusion</h2><br />
<br />
<span>While (re)learning Java, I felt like a student again and was quite enthusiastic about it initially. I invested around half a year, immersing myself intensively in Java (again). The last time I did that was many years ago as a university student. I even won a Silver Prize at work, implementing a project this year (2022 as of writing this). I feel confident now with understanding, debugging and patching Java code at work, which boosted my debugging and troubleshooting skills. </span><br />
<br />
<span>I don't hate Java, but I don't love programming in it, either. I will, I guess, always see Java as the necessary to get stuff done (reading code to understand how the service works, adding a tiny feature to make my life easier, adding a quick bug fix to overcome an obstacle...).</span><br />
<br />
<span>Although Java has significantly improved since 1.4, its code still tends to be more boilerplate. Not mainly because due to lines of code (Golang code tends to be quite repetitive, primarily when no generics are used), but due to the levels of abstractions it uses. Class hierarchies can be ten classes or deeper, and it is challenging to understand what the code is doing. Good test coverage and much documentation can mitigate the problem partially. Big enterprises use Java, and that also reflects to the language. There are too many libraries and too many abstractions that are bundled with too many legacy abstractions and interfaces and too many exceptions in the library APIs. There's even an external library named Lombok, which aims to reduce Java boilerplate code. Why is there a need for an external library? It should be all part of Java itself.</span><br />
<br />
<a class='textlink' href='https://projectlombok.org/'>https://projectlombok.org/</a><br />
<br />
<span>Java needs a clean cut. The clean cut shall be incompatible with previous versions of Java and only promote modern best practices without all the legacy burden carried around. The same can be said for other languages, e.g. Perl, but in Perl, they already attack the problem with the use of flags which change the behaviour of the language to more modern standards. Or do it like Python, where they had a hard (incompatible) cut from version 2 to version 3. It will be painful, for sure. But that would be the only way I would enjoy using that language as one of my primary languages to code new stuff regularly. Currently, my Java will stay limited to very few projects and the more minor things already mentioned in this post. </span><br />
<br />
<span>Am I a Java expert now? No, by far not. But I am better now than before :-).</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>I tried (Doom) Emacs, but I switched back to (Neo)Vim</title>
<link href="gemini://foo.zone/gemfeed/2022-11-24-i-tried-emacs-but-i-switched-back-to-neovim.gmi" />
<id>gemini://foo.zone/gemfeed/2022-11-24-i-tried-emacs-but-i-switched-back-to-neovim.gmi</id>
<updated>2022-11-24T11:17:15+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Art by \ \_! / __!</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>I tried (Doom) Emacs, but I switched back to (Neo)Vim</h1><br />
<br />
<span class='quote'>Published at 2022-11-24T11:17:15+02:00; Updated at 2022-11-26</span><br />
<br />
<pre>
_/ \ _(\(o
/ \ / _ ^^^o
/ ! \/ ! '!!!v'
! ! \ _' ( \____
! . \ _!\ \===^\)
Art by \ \_! / __!
Gunnar Z. \! / \ <--- Emacs is a giant dragon
(\_ _/ _\ )
\ ^^--^^ __-^ /(__
^^----^^ "^--v'
</pre>
<br />
<span>As a long-lasting user of Vim (and NeoVim), I always wondered what GNU Emacs is really about, so I decided to try it. I didn't try vanilla GNU Emacs, but Doom Emacs. I chose Doom Emacs as it is a neat distribution of Emacs with Evil mode enabled by default. Evil mode allows Vi(m) key bindings (so to speak, it's emulating Vim within Emacs), and I am pretty sure I won't be ready to give up all the muscle memory I have built over more than a decade.</span><br />
<br />
<a class='textlink' href='https://www.gnu.org/software/emacs/'>GNU Emacs</a><br />
<a class='textlink' href='https://github.com/doomemacs/'>Doom Emacs</a><br />
<br />
<span>I used Doom Emacs for around two months. Still, ultimately I decided to switch back to NeoVim as my primary editor and IDE and Vim (usually pre-installed on Linux-based systems) and Nvi (usually pre-installed on *BSD systems) as my "always available editor" for quick edits. (It is worth mentioning that I don't have a high opinion on whether Vim or NeoVim is the better editor, I prefer NeoVim as it comes with better defaults out of the box, but there is no real blocker to use Vim instead).</span><br />
<br />
<a class='textlink' href='https://www.vim.org'>Vim</a><br />
<a class='textlink' href='https://neovim.io'>NeoVim</a><br />
<br />
<span>So why did I switch back to the Vi-family?</span><br />
<br />
<h2 style='display: inline'>Emacs is a giant dragon</h2><br />
<br />
<span>Emacs feels like a giant dragon as it is much more than an editor or an integrated development environment. Emacs is a whole platform on its own. There's an E-Mail client, an IRC client, or even games you can run within Emacs. And you can also change Emacs within Emacs using its own Lisp dialect, Emacs Lisp (Emacs is programmed in Emacs Lisp). Therefore, Emacs is also its own programming language. You can change every aspect of Emacs within Emacs itself. People jokingly state Emacs is an operating system and that you should directly use it as the <span class='inlinecode'>init 1</span> process (if you don't know what the <span class='inlinecode'>init 1</span> process is: Under UNIX and similar operating systems, it's the very first userland processed launched. That's usually <span class='inlinecode'>systemd</span> on Linux-based systems, <span class='inlinecode'>launchd</span> on macOS, or any other init script or init system used by the OS)!</span><br />
<br />
<span>In many aspects, Emacs is like shooting at everything with a bazooka! However, I prefer it simple. I only wanted Emacs to be a good editor (which it is, too), but there's too much other stuff in Emacs that I don't need to care about! Vim and NeoVim do one thing excellent: Being great text editors and, when loaded with plugins, decent IDEs, too. </span><br />
<br />
<h2 style='display: inline'>Magit love</h2><br />
<br />
<span>I almost fell in love with Magit, an integrated Git client for Emacs. But I think the best way to interact with Git is to use the <span class='inlinecode'>git</span> command line directly. I don't worry about typing out all the commands, as the most commonly used commands are in my shell history. Other useful Git programs I use frequently are <span class='inlinecode'>bit</span> and <span class='inlinecode'>tig</span>. Also, get a mechanical keyboard that makes hammering whole commands into the terminal even more enjoyable.</span><br />
<br />
<a class='textlink' href='https://magit.vc/'>Magit</a><br />
<a class='textlink' href='https://github.com/jonas/tig'>Tig</a><br />
<br />
<span>Magit is pretty neat for basic Git operations, but I found myself searching the internet for the correct sub-commands to do the things I wanted to do in Git. Mainly, the way how branches are managed is confusing. Often, I fell back to the command line to fix up the mess I produced with Magit (e.g. accidentally pushing to the wrong remote branch, so I found myself fixing things manually on the terminal with the <span class='inlinecode'>git</span> command with forced pushes....). Magit is hotkey driven, and common commands are quickly explorable through built-in hotkey menus. Still, I found it challenging to navigate to more advanced Git sub-commands that way which was much easier accomplished by using the <span class='inlinecode'>git</span> command directly.</span><br />
<br />
<h2 style='display: inline'>Graphical UI</h2><br />
<br />
<span>If there is one thing I envy about Emacs is that it's a graphical program, whereas the Vi-family of editors are purely terminal-based. I see the benefits of being a graphical program as this enables the use of multiple fonts simultaneously to embed pictures and graphs (that would be neat as a Markdown preview, for example). There's also GVim (Vim with GTK UI), but that's more of an afterthought.</span><br />
<br />
<span>There are now graphical front-end clients for NeoVim, but I still need to dig into them. Let me know your experience if you have one. Luckily, I don't rely on something graphical in my text editor, but it would improve how the editor looks and feels. UTF8 can already do a lot in the terminal, and terminal emulators also allow you to use TrueType fonts. Still, you will always be limited to one TTF font for the whole terminal, and it isn't possible to have, for example, a different font for headings, paragraphs, etc... you get the idea. TTF+UTF8 can't beat authentic graphics. </span><br />
<br />
<h2 style='display: inline'>Scripting it</h2><br />
<br />
<span>It is possible to customize every aspect of Emacs through Emacs Lisp. I have done some Elk Scheme programming in the past (a dialect of Lisp), but that was a long time ago, and I am not willing to dive here again to customize my environment. I would instead take the pragmatic approach and script what I need in VimScript (a terrible language, but it gets the job done!). I watched Damian Conway's VimScript course on O'Reilly Safari Books Online, which I greatly recommend. Yes, VimScript feels clunky, funky and weird and is far less elegant than Lisp, but it gets its job done - in most cases! (That reminds me that the Vim team has announced a new major version of VimScript with improvements and language changes made - I haven't gotten to it yet - but I assume that VimScript will always stay VimScript).</span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Emacs_Lisp'>Emacs Lisp</a><br />
<a class='textlink' href='http://sam.zoy.org/elk/'>Elk Scheme</a><br />
<a class='textlink' href='http://vimscript.org/'>VimScript</a><br />
<a class='textlink' href='https://www.oreilly.com/library/view/scripting-vim/9781491996287/'>Scripting Vim by Damian Conway</a><br />
<br />
<span>NeoVim is also programmable with Lua, which seems to be a step up and Vim comes with a Perl plugin API (which was removed from NeoVim, but that is a different story - why would someone remove the most potent mature text manipulation programming language from one of the most powerful text editors?).</span><br />
<br />
<a class='textlink' href='https://neovim.io/doc/user/lua.html'>NeoVim Lua API</a><br />
<br />
<span>One example is my workflow of how I compose my blog articles (e.g. this one you are currently reading): I am writing everything in NeoVim, but I also want to have every paragraph checked against Grammarly (as English is not my first language). So I write a whole paragraph, then I select the entire paragraph via visual selection with <span class='inlinecode'>SHIFT+v</span>, and then I press <span class='inlinecode'>,y</span> to yank the paragraph to the systems clipboard, then I paste the paragraph to Grammarly's browser window with <span class='inlinecode'>CTRL+v</span>, let Grammarly suggest the improvements, and then I copy the result back with <span class='inlinecode'>CTRL+c</span> to the system clipboard and in NeoVim I type <span class='inlinecode'>,i</span> to insert the result back overriding the old paragraph (which is still selected in visual mode) with the new content. That all sounds a bit complicated, but it's surprisingly natural and efficient.</span><br />
<br />
<span>To come back to the example, for the clipboard integration, I use this small VimScript snippet, and I didn't have to dig into any Lisp or Perl for this:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#9A1900">" Clipboard</font></i>
vnoremap ,<b><font color="#0000FF">y</font></b> !pbcopy<font color="#FF6600"><CR></font>ugv
vnoremap ,<b><font color="#0000FF">i</font></b> !pbpaste<font color="#FF6600"><CR></font>
nmap ,<b><font color="#0000FF">i</font></b> !wpbpaste<font color="#FF6600"><CR></font>
</pre>
<br />
<span>That's only a very few lines and does precisely what I want. It's quick and dirty but get's the job done! If VimScript becomes too cumbersome, I can use Lua for NeoVim scripting.</span><br />
<br />
<h2 style='display: inline'>The famous Emacs Org mode</h2><br />
<br />
<span>Org-mode is an Emacs mode for keeping notes, authoring documents, computational notebooks, literate programming, maintaining to-do lists, planning projects, and more — in a fast and effective plain-text system. There's even a dedicated website for it:</span><br />
<br />
<a class='textlink' href='https://orgmode.org/'>https://orgmode.org/</a><br />
<br />
<span>In short, Org-mode is an "interactive markup language" that helps you organize everything mentioned above. I rarely touched the surface during my two-month experiment with Emacs, and I am impressed by it, so I see the benefits of having that. But it's not for me.</span><br />
<br />
<span>I use "Dead Tree Mode" to organize my work and notes. Dead tree? Yeah, I use an actual pen and a real paper journal (Leuchtturm or a Moleskine and a set of coloured <span class='inlinecode'>0.5</span> Muji Pens are excellent choices). That's far more immersive and flexible than a computer program can ever be. Yes, some automation and interaction with the computer (like calendar scheduling etc.) are missing. Still, an actual paper journal forces you to stay simple and focus on the actual work rather than tinkering with your computer program. (But I could not resist, and I wrote a VimScript which parses a table of contents page in Markdown format of my scanned paper journals, and NeoVim allows me to select a topic so that the corresponding PDF scan on the right journal page gets opened in an external PDF viewer (the PDF viewer is <span class='inlinecode'>zathura</span>, it uses Vi-keybindings, of course) :-). (See the appendix of this blog post for that script).</span><br />
<br />
<a class='textlink' href='https://pwmt.org/projects/zathura/'>Zathura</a><br />
<br />
<span>On the road, I also write some of my notes in Markdown format to NextCloud Notes, which is editable from my phone and via NeoVim on my computers. Markdown is much less powerful than Org-mode, but I prefer it the simple way. There's a neat terminal application, <span class='inlinecode'>ranger</span>, which I use to browse my NextCloud Notes when they are synced to a local folder on my machine. <span class='inlinecode'>ranger</span> is a file manager inspired by Vim and therefore makes use of Vim keybindings and it feels just natural to me. </span><br />
<br />
<a class='textlink' href='https://github.com/ranger/ranger'>Ranger - A Vim inspired file manager</a><br />
<span>Did I mention that I also use my <span class='inlinecode'>zsh</span> (my default shell) and my <span class='inlinecode'>tmux</span> (terminal multiplexer) in Vi-mode?</span><br />
<br />
<a class='textlink' href='https://zsh.sourceforge.io/'>Z shell</a><br />
<a class='textlink' href='https://github.com/tmux/tmux'>tmux terminal multiplexer</a><br />
<br />
<h2 style='display: inline'>Seeking simplicity</h2><br />
<br />
<span>I am not ready to dive deep into the whole world of Emacs. I prefer small and simple tools as opposed to complex tools. Emacs comes with many features out of the box, whereas in Vim/NeoVim, you would need to install many plugins to replicate some of the behaviour. Yes, I need to invest time managing all the Vim/NeoVim plugins I use, but I feel more in control compared to Doom Emacs, where a framework around vanilla Emacs manages all the plugins. I could use vanilla Emacs and manage all my plugins the vanilla way, but for me, it's not worth the effort to learn and dive into that as all that I want to do I can already do with Vim/NeoVim.</span><br />
<br />
<span>I am not saying that Vim/NeoVim are simple programs, but they are much simpler than Emacs with much smaller footprints; furthermore, they appear to be more straightforward as I am used to them. I only need Vim/NeoVim to be an editor, an IDE (through some plugins), and nothing more.</span><br />
<br />
<h2 style='display: inline'>Conclusion</h2><br />
<br />
<span>I understand the Emacs users now. Emacs is an incredibly powerful platform for almost everything, not just text editing. With Emacs, you can do nearly everything (Writing, editing, programming, calendar scheduling and note taking, Jira integration, playing games, listening to music, reading/writing emails, browsing the web, using as a calculator, generating HTML pages, configuring interactive menus, jumping around between every feature and every file within one single session, chat on IRC, surf the Gopherspace, ... the options are endless....). If you want to have one piece of software which rules it all and you are happy to invest a large part of your time in your platform: Pick Emacs, and over time Emacs will become "your" Emacs, customized to your own needs and change the way it works, which makes the Emacs users stick even more to it.</span><br />
<br />
<span>Vim/NeoVim also comes with a very high degree of customization options, but to a lesser extreme than Emacs (but still, a much higher degree than most other editors out there). If you want the best text editor in the world, which can also be tweaked to be a decent IDE, you are only looking for: Pick Vim or NeoVim! You would also need to invest a lot of time in learning, tweaking and customizing Vim/NeoVim, but that's a little more straightforward, and the result is much more lightweight once you get used to the "Vi way of doing things" you never would want to change back. I haven't tried the Emacs vanilla keystrokes, but they are terrible (that's probably one of the reasons why Doom Emacs uses Vim keybindings by default).</span><br />
<br />
<span class='quote'>Update: One reader recommended to have a look at NvChad. NvChad is a NeoVim config written in Lua aiming to provide a base configuration with very beautiful UI and blazing fast startuptime (around <span class='inlinecode'>0.02</span> secs ~ <span class='inlinecode'>0.07</span> secs). They tweak UI plugins such as telescope, nvim-tree, bufferline etc well to provide an aesthetic UI experience. That sounds interesting!</span><br />
<br />
<a class='textlink' href='https://github.com/NvChad/NvChad'>https://github.com/NvChad/NvChad</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Installing DTail on OpenBSD</title>
<link href="gemini://foo.zone/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi" />
<id>gemini://foo.zone/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi</id>
<updated>2022-10-30T11:03:19+02:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This will be a quick blog post, as I am busy with my personal life now. I have relocated to a different country and am still busy arranging things. So bear with me :-)</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='InstallingDTailonOpenBSD'>Installing DTail on OpenBSD</h1><br />
<br />
<span class='quote'>Published at 2022-10-30T11:03:19+02:00</span><br />
<br />
<pre>
,_---~~~~~----._
_,,_,*^____ _____``*g*\"*,
/ __/ /' ^. / \ ^@q f
@f | | | | 0 _/
\`/ \~__((@/ __ \__((@/ \
| _l__l_ I <--- The Go Gopher
} [______] I
] | | | |
] ~ ~ |
| |
| |
| | A ;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~,--,-/ \---,-/|~~,~~~~~~~~~~~~~~~~~~~~~~~~~~~
_|\,'. /| /| `/|-.
\`.' /| , `;.
,'\ A A A A _ /| `.;
,/ _ A _ / _ /| ;
/\ / \ , , A / / `/|
/_| | _ \ , , ,/ \
// | |/ `.\ ,- , , ,/ ,/ \/
/ @| |@ / /' \ \ , > /| ,--.
|\_/ \_/ / | | , ,/ \ ./' __:..
| __ __ | | | .--. , > > |-' / `
,/| / ' \ | | | \ , | /
/ |<--.__,->| | | . `. > > / (
/_,' \\ ^ / \ / / `. >-- /^\ |
\\___/ \ / / \__' \ \ \/ \ |
`. |/ , , /`\ \ )
\ ' |/ , V \ / `-\
OpenBSD Puffy ---> `|/ ' V V \ \.' \_
'`-. V V \./'\
`|/-. \ / \ /,---`\ kat
/ `._____V_____V'
' '
</pre>
<br />
<span>This will be a quick blog post, as I am busy with my personal life now. I have relocated to a different country and am still busy arranging things. So bear with me :-)</span><br />
<br />
<span> In this post, I want to give a quick overview (or how-to) about installing DTail on OpenBSD, as the official documentation only covers Red Hat and Fedora Linux! And this blog post will also be used as my reference!</span><br />
<br />
<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br />
<br />
<span>I am using Rexify for my OpenBSD automation. Check out the following article covering my Rex setup in a little bit more detail:</span><br />
<br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br />
<br />
<span>I will also mention some relevant <span class='inlinecode'>Rexfile</span> snippets in this post!</span><br />
<br />
<h2 style='display: inline' id='Compileit'>Compile it</h2><br />
<br />
<span>First of all, DTail needs to be downloaded and compiled. For that, <span class='inlinecode'>git</span>, <span class='inlinecode'>go</span>, and <span class='inlinecode'>gmake</span> are required:</span><br />
<br />
<pre>
$ doas pkg_add git go gmake
</pre>
<br />
<span>I am happy that the Go Programming Language is readily available in the OpenBSD packaging system. Once the dependencies got installed, clone DTail and compile it:</span><br />
<br />
<pre>
$ mkdir git
$ cd git
$ git clone https://github.com/mimecast/dtail
$ cd dtail
$ gmake
</pre>
<br />
<span>You can verify the version by running the following command:</span><br />
<br />
<pre>
$ ./dtail --version
DTail 4.1.0 Protocol 4.1 Have a lot of fun!
$ file dtail
dtail: ELF 64-bit LSB executable, x86-64, version 1
</pre>
<br />
<span>Now, there isn't any need anymore to keep <span class='inlinecode'>git</span>, <span class='inlinecode'>go</span> and <span class='inlinecode'>gmake</span>, so they can be deinstalled now:</span><br />
<br />
<pre>
$ doas pkg_delete git go gmake
</pre>
<br />
<span>One day I shall create an official OpenBSD port for DTail.</span><br />
<br />
<h2 style='display: inline' id='Installit'>Install it</h2><br />
<br />
<span>Installing the binaries is now just a matter of copying them to <span class='inlinecode'>/usr/local/bin</span> as follows:</span><br />
<br />
<pre>
$ for bin in dserver dcat dgrep dmap dtail dtailhealth; do
doas cp -p $bin /usr/local/bin/$bin
doas chown root:wheel /usr/local/bin/$bin
done
</pre>
<br />
<span>Also, we will be creating the <span class='inlinecode'>_dserver</span> service user:</span><br />
<br />
<pre>
$ doas adduser -class nologin -group _dserver -batch _dserver
$ doas usermod -d /var/run/dserver/ _dserver
</pre>
<br />
<span>The OpenBSD init script is created from scratch (not part of the official DTail project). Run the following to install the bespoke script:</span><br />
<br />
<pre>
$ cat <<'END' | doas tee /etc/rc.d/dserver
#!/bin/ksh
daemon="/usr/local/bin/dserver"
daemon_flags="-cfg /etc/dserver/dtail.json"
daemon_user="_dserver"
. /etc/rc.d/rc.subr
rc_reload=NO
rc_pre() {
install -d -o _dserver /var/log/dserver
install -d -o _dserver /var/run/dserver/cache
}
rc_cmd $1 &
END
$ doas chmod 755 /etc/rc.d/dserver
</pre>
<br />
<h3 style='display: inline' id='Rexification'>Rexification</h3><br />
<br />
<span>This is the task for setting it up via Rex. Note the <span class='inlinecode'>. . . .</span>, that's a placeholder which we will fill up more and more during this blog post:</span><br />
<br />
<pre>
desc 'Setup DTail';
task 'dtail', group => 'frontends',
sub {
my $restart = FALSE;
file '/etc/rc.d/dserver':
content => template('./etc/rc.d/dserver.tpl'),
owner => 'root',
group => 'wheel',
mode => '755',
on_change => sub { $restart = TRUE };
.
.
.
.
service 'dserver' => 'restart' if $restart;
service 'dserver', ensure => 'started';
};
</pre>
<br />
<h2 style='display: inline' id='Configureit'>Configure it</h2><br />
<br />
<span>Now, DTail is fully installed but still needs to be configured. Grab the default config file from GitHub ...</span><br />
<br />
<pre>
$ doas mkdir /etc/dserver
$ curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dtail.json.examples |
doas tee /etc/dserver/dtail.json
</pre>
<br />
<span>... and then edit it and adjust <span class='inlinecode'>LogDir</span> in the <span class='inlinecode'>Common</span> section to <span class='inlinecode'>/var/log/dserver</span>. The result will look like this:</span><br />
<br />
<pre>
"Common": {
"LogDir": "/var/log/dserver",
"Logger": "Fout",
"LogRotation": "Daily",
"CacheDir": "cache",
"SSHPort": 2222,
"LogLevel": "Info"
}
</pre>
<br />
<h3 style='display: inline' id='Rexification'>Rexification</h3><br />
<br />
<span>That's as simple as adding the following to the Rex task:</span><br />
<br />
<pre>
file '/etc/dserver',
ensure => 'directory';
file '/etc/dserver/dtail.json',
content => template('./etc/dserver/dtail.json.tpl'),
owner => 'root',
group => 'wheel',
mode => '755',
on_change => sub { $restart = TRUE };
</pre>
<br />
<h2 style='display: inline' id='Updatethekeycacheforit'>Update the key cache for it</h2><br />
<br />
<span>DTail relies on SSH for secure authentication and communication. However, the system user <span class='inlinecode'>_dserver</span> has no permission to read the SSH public keys from the user's home directories, so the DTail server also checks for available public keys in an alternative path <span class='inlinecode'>/var/run/dserver/cache</span>. </span><br />
<br />
<span>The following script, populating the DTail server key cache, can be run periodically via <span class='inlinecode'>CRON</span>:</span><br />
<br />
<pre>
$ cat <<'END' | doas tee /usr/local/bin/dserver-update-key-cache.sh
#!/bin/ksh
CACHEDIR=/var/run/dserver/cache
DSERVER_USER=_dserver
DSERVER_GROUP=_dserver
echo 'Updating SSH key cache'
ls /home/ | while read remoteuser; do
keysfile=/home/$remoteuser/.ssh/authorized_keys
if [ -f $keysfile ]; then
cachefile=$CACHEDIR/$remoteuser.authorized_keys
echo "Caching $keysfile -> $cachefile"
cp $keysfile $cachefile
chown $DSERVER_USER:$DSERVER_GROUP $cachefile
chmod 600 $cachefile
fi
done
# Cleanup obsolete public SSH keys
find $CACHEDIR -name \*.authorized_keys -type f |
while read cachefile; do
remoteuser=$(basename $cachefile | cut -d. -f1)
keysfile=/home/$remoteuser/.ssh/authorized_keys
if [ ! -f $keysfile ]; then
echo 'Deleting obsolete cache file $cachefile'
rm $cachefile
fi
done
echo 'All set...'
END
$ doas chmod 500 /usr/local/bin/dserver-update-key-cache.sh
</pre>
<br />
<span>Note that the script above is a slight variation of the official DTail script. The official DTail one is a <span class='inlinecode'>bash</span> script, but on OpenBSD, there's <span class='inlinecode'>ksh</span>. I run it once daily by adding it to the <span class='inlinecode'>daily.local</span>:</span><br />
<br />
<pre>
$ echo /usr/local/bin/dserver-update-key-cache.sh | doas tee -a /etc/daily.local
/usr/local/bin/dserver-update-key-cache.sh
</pre>
<br />
<h3 style='display: inline' id='Rexification'>Rexification</h3><br />
<br />
<span>That's done by adding ...</span><br />
<br />
<pre>
file '/usr/local/bin/dserver-update-key-cache.sh',
content => template('./scripts/dserver-update-key-cache.sh.tpl'),
owner => 'root',
group => 'wheel',
mode => '500';
append_if_no_such_line '/etc/daily.local', '/usr/local/bin/dserver-update-key-cache.sh';
</pre>
<br />
<span>... to the Rex task!</span><br />
<br />
<h2 style='display: inline' id='Startit'>Start it</h2><br />
<br />
<span>Now, it's time to enable and start the DTail server:</span><br />
<br />
<pre>
$ sudo rcctl enable dserver
$ sudo rcctl start dserver
$ tail -f /var/log/dserver/*.log
INFO|1022-090634|Starting scheduled job runner after 2s
INFO|1022-090634|Starting continuous job runner after 2s
INFO|1022-090644|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
INFO|1022-090654|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
INFO|1022-090719|Starting server|DTail 4.1.0 Protocol 4.1 Have a lot of fun!
INFO|1022-090719|Generating private server RSA host key
INFO|1022-090719|Starting server
INFO|1022-090719|Binding server|0.0.0.0:2222
INFO|1022-090719|Starting scheduled job runner after 2s
INFO|1022-090719|Starting continuous job runner after 2s
INFO|1022-090729|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
INFO|1022-090739|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnect
.
.
.
Ctr+C
</pre>
<br />
<span>As we don't want to wait until tomorrow, let's populate the key cache manually:</span><br />
<br />
<pre>
$ doas /usr/local/bin/dserver-update-key-cache.sh
Updating SSH key cache
Caching /home/_dserver/.ssh/authorized_keys -> /var/cache/dserver/_dserver.authorized_keys
Caching /home/admin/.ssh/authorized_keys -> /var/cache/dserver/admin.authorized_keys
Caching /home/failunderd/.ssh/authorized_keys -> /var/cache/dserver/failunderd.authorized_keys
Caching /home/git/.ssh/authorized_keys -> /var/cache/dserver/git.authorized_keys
Caching /home/paul/.ssh/authorized_keys -> /var/cache/dserver/paul.authorized_keys
Caching /home/rex/.ssh/authorized_keys -> /var/cache/dserver/rex.authorized_keys
All set...
</pre>
<br />
<h2 style='display: inline' id='Useit'>Use it</h2><br />
<br />
<span>The DTail server is now ready to serve connections. You can use any DTail commands, such as <span class='inlinecode'>dtail</span>, <span class='inlinecode'>dgrep</span>, <span class='inlinecode'>dmap</span>, <span class='inlinecode'>dcat</span>, <span class='inlinecode'>dtailhealth</span>, to do so. Checkout out all the usage examples on the official DTail page.</span><br />
<br />
<span>I have installed DTail server this way on my personal OpenBSD frontends <span class='inlinecode'>blowfish</span>, and <span class='inlinecode'>fishfinger</span>, and the following command connects as user <span class='inlinecode'>rex</span> to both machines and greps the file <span class='inlinecode'>/etc/fstab</span> for the string <span class='inlinecode'>local</span>:</span><br />
<br />
<pre>
❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
CLIENT|earth|WARN|Encountered unknown host|{blowfish.buetow.org:2222 0xc0000a00f0 0xc0000a61e0 [blowfish.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN [23.88.35.144]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN 0xc0000a2180}
CLIENT|earth|WARN|Encountered unknown host|{fishfinger.buetow.org:2222 0xc0000a0150 0xc000460110 [fishfinger.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L [46.23.94.99]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L 0xc0000a2240}
Encountered 2 unknown hosts: 'blowfish.buetow.org:2222,fishfinger.buetow.org:2222'
Do you want to trust these hosts?? (y=yes,a=all,n=no,d=details): a
CLIENT|earth|INFO|STATS:STATS|cgocalls=11|cpu=8|connected=2|servers=2|connected%=100|new=2|throttle=0|goroutines=19
CLIENT|earth|INFO|Added hosts to known hosts file|/home/paul/.ssh/known_hosts
REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
</pre>
<br />
<span>Running it the second time, and given that you trusted the keys the first time, it won't prompt you for the host keys anymore:</span><br />
<br />
<pre>
❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
</pre>
<br />
<h2 style='display: inline' id='Conclusions'>Conclusions</h2><br />
<br />
<span>It's a bit of manual work, but it's ok on this small scale! I shall invest time in creating an official OpenBSD port, though. That would render most of the manual steps obsolete, as outlined in this post!</span><br />
<br />
<span>Check out the following for more information:</span><br />
<br />
<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br />
<a class='textlink' href='https://github.com/mimecast/dtail'>https://github.com/mimecast/dtail</a><br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-04-22-dtail-the-distributed-log-tail-program.html'>2021-04-22 DTail - The distributed log tail program</a><br />
<a class='textlink' href='./2022-03-06-the-release-of-dtail-4.0.0.html'>2022-03-06 The release of DTail 4.0.0</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD (You are currently reading this)</a><br />
<a class='textlink' href='./2023-09-25-dtail-usage-examples.html'>2023-09-25 DTail usage examples</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>After a bad night's sleep</title>
<link href="gemini://foo.zone/gemfeed/2022-09-30-after-a-bad-nights-sleep.gmi" />
<id>gemini://foo.zone/gemfeed/2022-09-30-after-a-bad-nights-sleep.gmi</id>
<updated>2022-09-30T09:53:23+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Everyone has it once in a while: A bad night's sleep. Here I attempt to list valuable tips on how to deal with it.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>After a bad night's sleep</h1><br />
<br />
<span class='quote'>Published at 2022-09-30T09:53:23+03:00; Updated at 2022-10-12</span><br />
<br />
<pre>
z
z
Z
.--. Z Z
/ _(c\ .-. __
| / / '-; \'-'` `\______
\_\/'/ __/ ) / ) | \--,
| \`""`__-/ .'--/ /--------\ \
\\` ///-\/ / /---;-. '-'
jgs (________\ \
'-'
</pre>
<br />
<span>Everyone has it once in a while: A bad night's sleep. Here I attempt to list valuable tips on how to deal with it.</span><br />
<br />
<h2 style='display: inline'>Don't take the day off.</h2><br />
<br />
<span>Don't take a day off after not sleeping enough the previous night. That would be wasting the holiday allowance. It wouldn't be possible to enjoy my free time anyway, so why not just work? There's still a way for an IT Engineer to be productive (sometimes even more) with half or less of the concentration power available!</span><br />
<br />
<h2 style='display: inline'>Start work early</h2><br />
<br />
<span>Probably I am already awake early and am unable to fall asleep again. My strategy here is to "attack" the day: Start work early and finish early. The early bird will also encounter fewer distractions from colleagues.</span><br />
<br />
<h2 style='display: inline'>Sweat the small stuff</h2><br />
<br />
<span>There's never a shortage of small items to hook off my list. Most of these items don't require my full concentration power, and I will be happy to get them off my list so that the next day, after a good night's sleep, I can immerse myself again in focused, deep work with all concentration powers at hand.</span><br />
<br />
<span>Examples of "small work items" are:</span><br />
<br />
<ul>
<li>Tidying up the workspace.</li>
<li>Installing pending computer software updates.</li>
<li>Going through the work backlog: Create new tickets, close obsolete ones, and roughly pre-plan upcoming work.</li>
<li>Finishing off the easy tickets from the current sprint.</li>
<li>Going through any tedious paperwork.</li>
<li>Catch up with the journal and mark off all trivial action items.</li>
</ul><br />
<h2 style='display: inline'>Enter the flow state</h2><br />
<br />
<span>I find it easy to enter the "flow state" after a bad night's sleep. All I need to do is to put on some ambient music (preferably instrumental chill house) and start to work on a not-too-difficult ticket.</span><br />
<br />
<span>Usually, the "flow state" is associated with deep-focused work, but deep-focused work isn't easily possible under sleep deprivation. It's still possible to be in the flow by working on more manageable tasks and leaving the difficult ones for the next day.</span><br />
<br />
<h2 style='display: inline'>Reschedule meetings</h2><br />
<br />
<span>I find engaging in discussions and demanding meetings challenging after a lousy night's sleep. I still attend the sessions I am invited to as "only" a participant, but I prefer to reschedule all meetings I am the primary driver of.</span><br />
<br />
<span>This, unfortunately, also includes interviews. Interviews require full concentration power. So for interviews, I would find a colleague to step in for me or ask to reschedule the interview altogether. Everything else wouldn't make it justice and would waste everyone's time!</span><br />
<br />
<h2 style='display: inline'>Invent</h2><br />
<br />
<span>The mind works differently under sleep deprivation: It's easier to invent new stuff as it's easier to have a look at things from different perspectives. Until an hour ago, I didn't know yet what I would be blogging about for this month, and then I just started writing this, and it took me only half an hour to write the first draft of this blog post!</span><br />
<br />
<h2 style='display: inline'>Fast</h2><br />
<br />
<span>I don't eat breakfast, and I don't eat lunch on these days. I only have dinner. Not eating means my mind doesn't get foggy, and I keep up the work momentum. This is called intermittent fasting, which not only generally helps to keep the weight under control and boosts the concentration power. Furthermore, intermittent fasting is healthy. You should include it in your routine, even after a good night's sleep.</span><br />
<br />
<h2 style='display: inline'>Stretch</h2><br />
<br />
<span>I won't have enough energy for strenuous physical exercise on those days, but a 30 to a 60-minute stretching session can make the day. Stretching will even hurt less under sleep deprivation! The stretching could also be substituted with a light Yoga session.</span><br />
<br />
<h2 style='display: inline'>Walk</h2><br />
<br />
<span>Walking is healthy, and the time can be used to listen to interesting podcasts. The available concentration power might not be enough for more sophisticated audio literature. I will have enough energy for one or two daily walks (~10k steps for the day in total). Sometimes, I listen to music during walks. I also try to catch the bright sunlight.</span><br />
<br />
<h2 style='display: inline'>Red Bull</h2><br />
<br />
<span>I don't think that Red Bull is a healthy drink. But once in a while, a can in the early afternoon brings wonders, and productivity will skyrocket. Other than Red Bull, drink a lot of water throughout the day. Don't forget to drink the sugar-free version; otherwise, your intermittent fast will be broken.</span><br />
<br />
<h2 style='display: inline'>Power nap</h2><br />
<br />
<span>I don't know how to "enforce" a nap, but sometimes I manage to power nap, and it helps wonders. A 30-minute nap sometimes brings me back to normal. If you don't tend to fast as you are too hungry, it helps to try to nap approximately 30 minutes after eating something.</span><br />
<br />
<h2 style='display: inline'>Don't take anything personally.</h2><br />
<br />
<span>It's much more challenging to keep the mind "under control" in this state. Every annoyance can potentially upset, which could reflect on the work colleagues. It is wise to attempt to go with a positive attitude into the day, always smile and be polite to the family and colleagues at work. Don't let anything drop out to the people next; they don't deserve it as they didn't do anything wrong! Also, remember, it can't be controlled at all. It's time to let go of the annoyances for the day.</span><br />
<br />
<h2 style='display: inline'>Meditate</h2><br />
<br />
<span>To keep the good vibe, it helps to meditate for 10 minutes. Meditation must nothing be fancy. It can be just lying on the sofa and observing your thoughts as they come and go. Don't judge your thoughts, as that could put you in a negative mood. It's not necessary to sit in an uncomfortable Yoga pose, and it is not required to chant "Ohhmmmmm".</span><br />
<br />
<h2 style='display: inline'>Write things down</h2><br />
<br />
<span>Sometimes something requiring more concentration power demands time. This is where it helps to write a note in a journal and return to it another day. This doesn't mean slacking off but managing the rarely available concentration power for the day. I might repeat myself: Today, sweat all the small stuff. Tomorrow, do the deep-focused work on that crucial project again.</span><br />
<br />
<span>It's easier to forget things on those days, so everything should be written down so that it can be worked off later. Things written down will not be overlooked!</span><br />
<br />
<h2 style='display: inline'>Social media</h2><br />
<br />
<span>I wouldn't say I like checking social media, as it can consume a lot of time and can become addictive. But once in a while, I want to catch up with my "networks". After a bad night's sleep, it's the perfect time to check your social media. Once done, you don't have to do it anymore for the next couple of days!</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Gemtexter 1.1.0 - Let's Gemtext again</title>
<link href="gemini://foo.zone/gemfeed/2022-08-27-gemtexter-1.1.0-lets-gemtext-again.gmi" />
<id>gemini://foo.zone/gemfeed/2022-08-27-gemtexter-1.1.0-lets-gemtext-again.gmi</id>
<updated>2022-08-27T18:25:57+01:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I proudly announce that I've released Gemtexter version `1.1.0`. What is Gemtexter? It's my minimalist static site generator for Gemini Gemtext, HTML and Markdown written in GNU Bash.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='Gemtexter110LetsGemtextagain'>Gemtexter 1.1.0 - Let's Gemtext again</h1><br />
<br />
<span class='quote'>Published at 2022-08-27T18:25:57+01:00</span><br />
<br />
<pre>
-=[ typewriter ]=- 1/98
.-------.
_|~~ ~~ |_
=(_|_______|_)=
|:::::::::|
|:::::::[]|
|o=======.|
jgs `"""""""""`
</pre>
<br />
<span>I proudly announce that I've released Gemtexter version <span class='inlinecode'>1.1.0</span>. What is Gemtexter? It's my minimalist static site generator for Gemini Gemtext, HTML and Markdown written in GNU Bash.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/gemtexter'>https://codeberg.org/snonux/gemtexter</a><br />
<br />
<span>It has been around a year since I released the first version <span class='inlinecode'>1.0.0</span>. Although, there aren't any groundbreaking changes, there have been a couple of smaller commits and adjustments. I was quite surprised that I received a bunch of feedback and requests about Gemtexter so it means that I am not the only person in the universe actually using it.</span><br />
<br />
<h2 style='display: inline' id='Whatsnew'>What's new?</h2><br />
<br />
<h3 style='display: inline' id='AutomaticcheckforGNUversionrequirements'>Automatic check for GNU version requirements</h3><br />
<br />
<span>Gemtexter relies on the GNU versions of the tools <span class='inlinecode'>grep</span>, <span class='inlinecode'>sed</span> and <span class='inlinecode'>date</span> and it also requires the Bash shell in version 5 at least. That's now done in the <span class='inlinecode'>check_dependencies()</span> function:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#7bc710">check_dependencies ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><i><font color="#ababab"># At least, Bash 5 is required</font></i>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -i </font><font color="#ff0000">required_version</font><font color="#F3E651">=</font><font color="#bb00ff">5</font>
<font color="#ff0000"> </font><font color="#ff0000">IFS</font><font color="#F3E651">=.</font><font color="#ff0000"> </font><b><font color="#ffffff">read</font></b><font color="#ff0000"> -ra version </font><font color="#F3E651"><<<</font><font color="#ff0000"> </font><font color="#bb00ff">"$BASH_VERSION"</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#bb00ff">"${version[0]}"</font><font color="#ff0000"> -lt </font><font color="#ff0000">$required_version</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> log ERROR </font><font color="#bb00ff">"ERROR, </font><font color="#ffffff">\"</font><font color="#bb00ff">bash</font><font color="#ffffff">\"</font><font color="#bb00ff"> must be at least at major version $required_version!"</font>
<font color="#ff0000"> </font><b><font color="#ffffff">exit</font></b><font color="#ff0000"> </font><font color="#bb00ff">2</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><i><font color="#ababab"># These must be the GNU versions of the commands</font></i>
<font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> tool </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> </font><font color="#ff0000">$DATE</font><font color="#ff0000"> </font><font color="#ff0000">$SED</font><font color="#ff0000"> </font><font color="#ff0000">$GREP</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> </font><font color="#ff0000">$tool</font><font color="#ff0000"> --version </font><font color="#F3E651">|</font><font color="#ff0000"> grep -q GNU</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> log ERROR </font><font color="#bb00ff">"ERROR, </font><font color="#ffffff">\"</font><font color="#bb00ff">$tool</font><font color="#ffffff">\"</font><font color="#bb00ff"> command is not the GNU version, please install!"</font>
<font color="#ff0000"> </font><b><font color="#ffffff">exit</font></b><font color="#ff0000"> </font><font color="#bb00ff">2</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">done</font></b>
<font color="#ff0000">}</font>
</pre>
<br />
<span>Especially macOS users didn't read the <span class='inlinecode'>README</span> carefully enough to install GNU Grep, GNU Sed and GNU Date before using Gemtexter.</span><br />
<br />
<h3 style='display: inline' id='BackticksnowproduceinlinecodeblocksintheHTMLoutput'>Backticks now produce <span class='inlinecode'>inline code blocks</span> in the HTML output</h3><br />
<br />
<span>The Gemtext format doesn't support inline code blocks, but Gemtexter now produces <span class='inlinecode'>inline code blocks</span> (means, small code fragments can be placed in the middle of a paragraph) in the HTML output when the code block is enclosed with Backticks. There were no adjustments required for the Markdown output format, because Markdown supports it already out of the box.</span><br />
<br />
<h3 style='display: inline' id='CacheforAtomfeedgeneration'>Cache for Atom feed generation</h3><br />
<br />
<span>The Bash is not the most performant language. Gemtexter already takes a couple of seconds only to generate the Atom feed for around two hand full of articles on my slightly underpowered Surface Go 2 Linux tablet. Therefore, I introduced a cache, so that subsequent Atom feed generation runs finish much quicker. The cache uses a checksum of the Gemtext <span class='inlinecode'>.gmi</span> file to decide whether anything of the content has changed or not.</span><br />
<br />
<h3 style='display: inline' id='Inputfiltersupport'>Input filter support</h3><br />
<br />
<span>Once your capsule reaches a certain size, it can become annoying to re-generate everything if you only want to preview the HTML or Markdown output of one single content file. The following will add a filter to only generate the files matching a regular expression:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#F3E651">.</font><font color="#ff0000">/gemtexter --generate </font><font color="#bb00ff">'.*hello.*'</font>
</pre>
<br />
<h3 style='display: inline' id='Revampedgitsupport'>Revamped <span class='inlinecode'>git</span> support</h3><br />
<br />
<span>The Git support has been completely rewritten. It's now more reliable and faster too. Have a look at the <span class='inlinecode'>README</span> for more information.</span><br />
<br />
<h3 style='display: inline' id='Additionofhtmlextrasandwebfontsupport'>Addition of <span class='inlinecode'>htmlextras</span> and web font support</h3><br />
<br />
<span>The <span class='inlinecode'>htmlextras</span> folder now contains all extra files required for the HTML output format such as cascading style sheet (CSS) files and web fonts.</span><br />
<br />
<h3 style='display: inline' id='Subsectionsupport'>Sub-section support</h3><br />
<br />
<span>It's now possible to define sub-sections within a Gemtexter capsule. For the HTML output, each sub-section can use its own CSS and web font definitions. E.g.:</span><br />
<br />
<a class='textlink' href='https://foo.zone'>The foo.zone main site</a><br />
<a class='textlink' href='https://foo.zone/notes'>The notes sub-section (with different fonts)</a><br />
<br />
<h3 style='display: inline' id='More'>More</h3><br />
<br />
<span>Additionally, there were a couple of bug fixes, refactorings and overall improvements in the documentation made. </span><br />
<br />
<span>Overall I think it's a pretty solid <span class='inlinecode'>1.1.0</span> release without anything groundbreaking (therefore no major version jump). But I am happy about it.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-04-24-welcome-to-the-geminispace.html'>2021-04-24 Welcome to the Geminispace</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2022-08-27-gemtexter-1.1.0-lets-gemtext-again.html'>2022-08-27 Gemtexter 1.1.0 - Let's Gemtext again (You are currently reading this)</a><br />
<a class='textlink' href='./2023-03-25-gemtexter-2.0.0-lets-gemtext-again-2.html'>2023-03-25 Gemtexter 2.0.0 - Let's Gemtext again²</a><br />
<a class='textlink' href='./2023-07-21-gemtexter-2.1.0-lets-gemtext-again-3.html'>2023-07-21 Gemtexter 2.1.0 - Let's Gemtext again³</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Let's Encrypt with OpenBSD and Rex</title>
<link href="gemini://foo.zone/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi" />
<id>gemini://foo.zone/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi</id>
<updated>2022-07-30T12:14:31+01:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I was amazed at how easy it is to automatically generate and update Let's Encrypt certificates with OpenBSD.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='LetsEncryptwithOpenBSDandRex'>Let's Encrypt with OpenBSD and Rex</h1><br />
<br />
<span class='quote'>Published at 2022-07-30T12:14:31+01:00</span><br />
<br />
<pre>
/ _ \
The Hebern Machine \ ." ". /
___ / \
.."" "".. | O |
/ \ | |
/ \ | |
---------------------------------
_/ o (O) o _ |
_/ ." ". |
I/ _________________/ \ |
_/I ." | |
===== / I / / |
===== | | | \ | _________________." |
===== | | | | | / \ / _|_|__|_|_ __ |
| | | | | | | \ "._." / o o \ ." ". |
| --| --| -| / \ _/ / \ |
\____\____\__| \ ______ | / | | |
-------- --- / | | |
( ) (O) / \ / |
----------------------- ".__." |
_|__________________________________________|_
/ \
/________________________________________________\
ASCII Art by John Savard
</pre>
<br />
<span>I was amazed at how easy it is to automatically generate and update Let's Encrypt certificates with OpenBSD.</span><br />
<br />
<h2 style='display: inline' id='WhatsLetsEncrypt'>What's Let's Encrypt?</h2><br />
<br />
<span class='quote'>Let's Encrypt is a non-profit certificate authority run by Internet Security Research Group that provides X.509 certificates for Transport Layer Security (TLS) encryption at no charge. It is the world's largest certificate authority, used by more than 265 million websites, with the goal of all websites being secure and using HTTPS.</span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Let's_Encrypt'>Source: Wikipedia</a><br />
<br />
<span>In short, it gives away TLS certificates for your website - for free! The catch is, that the certificates are only valid for three months. So it is better to automate certificate generation and renewals.</span><br />
<br />
<h2 style='display: inline' id='Meetacmeclient'>Meet <span class='inlinecode'>acme-client</span></h2><br />
<br />
<span><span class='inlinecode'>acme-client</span> is the default Automatic Certifcate Management Environment (ACME) client on OpenBSD and part of the OpenBSD base system. </span><br />
<br />
<span>When invoked, the client first checks whether certificates actually require to be generated.</span><br />
<br />
<ul>
<li>It first checks whether a certificate already exists; if not, it will attempt to generate a new one.</li>
<li>If the certificate already exists but expires within the next 30 days, it will renew it.</li>
<li>Otherwise, <span class='inlinecode'>acme-client</span> won't do anything.</li>
</ul><br />
<span>Oversimplified, the following steps are undertaken by <span class='inlinecode'>acme-client</span> for generating a new certificate:</span><br />
<br />
<ul>
<li>Reading its config file <span class='inlinecode'>/etc/acme-client.conf</span> for a list of hosts (and their alternative names) to generate certificates. So it means you can also have certificates for arbitrary subdomains!</li>
<li>Automatic generation of the private certificate part (the certificate key) and the certificate signing request (CSR) to <span class='inlinecode'>/etc/ssl/...</span>.</li>
<li>Requesting Let's Encrypt to sign the certificate. This also includes providing a set of temporary files requested by Let's Encrypt in the next step for verification.</li>
<li>Let's Encrypt then will contact the hostname for the certificate through a particular URL (e.g. <span class='inlinecode'>http://foo.zone/.well-known/acme-challenge/...</span>) to verify that the requester is the valid owner of the host.</li>
<li>Let's Encrypt generates a certificate, which then is downloaded to <span class='inlinecode'>/etc/ssl/...</span>.</li>
</ul><br />
<h2 style='display: inline' id='Configuration'>Configuration</h2><br />
<br />
<span>There is some (but easy) configuration required to make that all work on OpenBSD.</span><br />
<br />
<h3 style='display: inline' id='acmeclientconf'>acme-client.conf</h3><br />
<br />
<span>This is how my <span class='inlinecode'>/etc/acme-client.conf</span> looks like (I copied a template from <span class='inlinecode'>/etc/examples/acme-client.conf</span> to <span class='inlinecode'>/etc/acme-client.conf</span> and added my domains to the bottom:</span><br />
<br />
<pre>
#
# $OpenBSD: acme-client.conf,v 1.4 2020/09/17 09:13:06 florian Exp $
#
authority letsencrypt {
api url "https://acme-v02.api.letsencrypt.org/directory"
account key "/etc/acme/letsencrypt-privkey.pem"
}
authority letsencrypt-staging {
api url "https://acme-staging-v02.api.letsencrypt.org/directory"
account key "/etc/acme/letsencrypt-staging-privkey.pem"
}
authority buypass {
api url "https://api.buypass.com/acme/directory"
account key "/etc/acme/buypass-privkey.pem"
contact "mailto:me@example.com"
}
authority buypass-test {
api url "https://api.test4.buypass.no/acme/directory"
account key "/etc/acme/buypass-test-privkey.pem"
contact "mailto:me@example.com"
}
domain buetow.org {
alternative names { www.buetow.org paul.buetow.org }
domain key "/etc/ssl/private/buetow.org.key"
domain full chain certificate "/etc/ssl/buetow.org.fullchain.pem"
sign with letsencrypt
}
domain dtail.dev {
alternative names { www.dtail.dev }
domain key "/etc/ssl/private/dtail.dev.key"
domain full chain certificate "/etc/ssl/dtail.dev.fullchain.pem"
sign with letsencrypt
}
domain foo.zone {
alternative names { www.foo.zone }
domain key "/etc/ssl/private/foo.zone.key"
domain full chain certificate "/etc/ssl/foo.zone.fullchain.pem"
sign with letsencrypt
}
domain irregular.ninja {
alternative names { www.irregular.ninja }
domain key "/etc/ssl/private/irregular.ninja.key"
domain full chain certificate "/etc/ssl/irregular.ninja.fullchain.pem"
sign with letsencrypt
}
domain snonux.land {
alternative names { www.snonux.land }
domain key "/etc/ssl/private/snonux.land.key"
domain full chain certificate "/etc/ssl/snonux.land.fullchain.pem"
sign with letsencrypt
}
</pre>
<br />
<h3 style='display: inline' id='httpdconf'>httpd.conf</h3><br />
<br />
<span>For ACME to work, you will need to configure the HTTP daemon so that the "special" ACME requests from Let's Encrypt are served correctly. I am using the standard OpenBSD <span class='inlinecode'>httpd</span> here. These are the snippets I use for the <span class='inlinecode'>foo.zone</span> host in <span class='inlinecode'>/etc/httpd.conf</span> (of course, you need a similar setup for all other hosts as well):</span><br />
<br />
<pre>
server "foo.zone" {
listen on * port 80
location "/.well-known/acme-challenge/*" {
root "/acme"
request strip 2
}
location * {
block return 302 "https://$HTTP_HOST$REQUEST_URI"
}
}
server "foo.zone" {
listen on * tls port 443
tls {
certificate "/etc/ssl/foo.zone.fullchain.pem"
key "/etc/ssl/private/foo.zone.key"
}
location * {
root "/htdocs/gemtexter/foo.zone"
directory auto index
}
}
</pre>
<br />
<span>As you see, plain HTTP only serves the ACME challenge path. Otherwise, it redirects the requests to TLS. The TLS section then attempts to use the Let's Encrypt certificates.</span><br />
<br />
<span>It is worth noticing that <span class='inlinecode'>httpd</span> will start without the certificates being present. This will cause a certificate error when you try to reach the HTTPS endpoint, but it helps to bootstrap Let's Encrypt. As you saw in the config snippet above, Let's Encrypt only requests the plain HTTP endpoint for the verification process, so HTTPS doesn't need to be operational yet at this stage. But once the certificates are generated, you will have to reload or restart <span class='inlinecode'>httpd</span> to use any new certificate.</span><br />
<br />
<h3 style='display: inline' id='CRONjob'>CRON job</h3><br />
<br />
<span>You could now run <span class='inlinecode'>doas acme-client foo.zone</span> to generate the certificate or to renew it. Or you could automate it with CRON.</span><br />
<br />
<span>I have created a script <span class='inlinecode'>/usr/local/bin/acme.sh</span> for that for all of my domains:</span><br />
<br />
<pre>
#!/bin/sh
function handle_cert {
host=$1
# Create symlink, so that relayd also can read it.
crt_path=/etc/ssl/$host
if [ -e $crt_path.crt ]; then
rm $crt_path.crt
fi
ln -s $crt_path.fullchain.pem $crt_path.crt
# Requesting and renewing certificate.
/usr/sbin/acme-client -v $host
}
has_update=no
handle_cert www.buetow.org
if [ $? -eq 0 ]; then
has_update=yes
fi
handle_cert www.paul.buetow.org
if [ $? -eq 0 ]; then
has_update=yes
fi
handle_cert www.tmp.buetow.org
if [ $? -eq 0 ]; then
has_update=yes
fi
handle_cert www.dtail.dev
if [ $? -eq 0 ]; then
has_update=yes
fi
handle_cert www.foo.zone
if [ $? -eq 0 ]; then
has_update=yes
fi
handle_cert www.irregular.ninja
if [ $? -eq 0 ]; then
has_update=yes
fi
handle_cert www.snonux.land
if [ $? -eq 0 ]; then
has_update=yes
fi
# Pick up the new certs.
if [ $has_update = yes ]; then
/usr/sbin/rcctl reload httpd
/usr/sbin/rcctl reload relayd
/usr/sbin/rcctl restart smtpd
fi
</pre>
<br />
<span>And added the following line to <span class='inlinecode'>/etc/daily.local</span> to run the script once daily so that certificates will be renewed fully automatically:</span><br />
<br />
<pre>
/usr/local/bin/acme.sh
</pre>
<br />
<span>I am receiving a daily output via E-Mail like this now:</span><br />
<br />
<pre>
Running daily.local:
acme-client: /etc/ssl/buetow.org.fullchain.pem: certificate valid: 80 days left
acme-client: /etc/ssl/paul.buetow.org.fullchain.pem: certificate valid: 80 days left
acme-client: /etc/ssl/tmp.buetow.org.fullchain.pem: certificate valid: 80 days left
acme-client: /etc/ssl/dtail.dev.fullchain.pem: certificate valid: 80 days left
acme-client: /etc/ssl/foo.zone.fullchain.pem: certificate valid: 80 days left
acme-client: /etc/ssl/irregular.ninja.fullchain.pem: certificate valid: 80 days left
acme-client: /etc/ssl/snonux.land.fullchain.pem: certificate valid: 79 days left
</pre>
<br />
<h2 style='display: inline' id='relaydconfandsmtpdconf'>relayd.conf and smtpd.conf</h2><br />
<br />
<span>Besides <span class='inlinecode'>httpd</span>, <span class='inlinecode'>relayd</span> (mainly for Gemini) and <span class='inlinecode'>smtpd</span> (for mail, of course) also use TLS certificates. And as you can see in <span class='inlinecode'>acme.sh</span>, the services are reloaded or restarted (<span class='inlinecode'>smtpd</span> doesn't support reload) whenever a certificate is generated or updated.</span><br />
<br />
<h2 style='display: inline' id='Rexification'>Rexification</h2><br />
<br />
<span>I didn't write all these configuration files by hand. As a matter of fact, everything is automated with the Rex configuration management system.</span><br />
<br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<br />
<span>At the top of the <span class='inlinecode'>Rexfile</span> I define all my hosts:</span><br />
<br />
<pre>
our @acme_hosts = qw/buetow.org paul.buetow.org tmp.buetow.org dtail.dev foo.zone irregular.ninja snonux.land/;
</pre>
<br />
<h3 style='display: inline' id='GeneralACMEclientconfiguration'>General ACME client configuration</h3><br />
<br />
<span>ACME will be installed into the frontend group of hosts. Here, blowfish is the primary, and twofish is the secondary OpenBSD box.</span><br />
<br />
<pre>
group frontends => 'blowfish.buetow.org', 'twofish.buetow.org';
</pre>
<br />
<span>This is my Rex task for the general ACME configuration:</span><br />
<br />
<pre>
desc 'Configure ACME client';
task 'acme', group => 'frontends',
sub {
file '/etc/acme-client.conf',
content => template('./etc/acme-client.conf.tpl',
acme_hosts => \@acme_hosts,
is_primary => $is_primary),
owner => 'root',
group => 'wheel',
mode => '644';
file '/usr/local/bin/acme.sh',
content => template('./scripts/acme.sh.tpl',
acme_hosts => \@acme_hosts,
is_primary => $is_primary),
owner => 'root',
group => 'wheel',
mode => '744';
file '/etc/daily.local',
ensure => 'present',
owner => 'root',
group => 'wheel',
mode => '644';
append_if_no_such_line '/etc/daily.local', '/usr/local/bin/acme.sh';
};
</pre>
<br />
<span>And there is also a Rex task just to run the ACME script remotely:</span><br />
<br />
<pre>
desc 'Invoke ACME client';
task 'acme_invoke', group => 'frontends',
sub {
say run '/usr/local/bin/acme.sh';
};
</pre>
<br />
<span>Furthermore, this snippet (also at the top of the Rexfile) helps to determine whether the current server is the primary server (all hosts will be without the <span class='inlinecode'>www.</span> prefix) or the secondary server (all hosts will be with the <span class='inlinecode'>www.</span> prefix):</span><br />
<br />
<pre>
# Bootstrapping the FQDN based on the server IP as the hostname and domain
# facts aren't set yet due to the myname file in the first place.
our $fqdns = sub {
my $ipv4 = shift;
return 'blowfish.buetow.org' if $ipv4 eq '23.88.35.144';
return 'twofish.buetow.org' if $ipv4 eq '108.160.134.135';
Rex::Logger::info("Unable to determine hostname for $ipv4", 'error');
return 'HOSTNAME-UNKNOWN.buetow.org';
};
# To determine whether the server is the primary or the secondary.
our $is_primary = sub {
my $ipv4 = shift;
$fqdns->($ipv4) eq 'blowfish.buetow.org';
};
</pre>
<br />
<span>The following is the <span class='inlinecode'>acme-client.conf.tpl</span> Rex template file used for the automation. You see that the <span class='inlinecode'>www.</span> prefix isn't sent for the primary server. E.g. <span class='inlinecode'>foo.zone</span> will be served by the primary server (in my case, a server located in Germany) and <span class='inlinecode'>www.foo.zone</span> by the secondary server (in my case, a server located in Japan):</span><br />
<br />
<pre>
#
# $OpenBSD: acme-client.conf,v 1.4 2020/09/17 09:13:06 florian Exp $
#
authority letsencrypt {
api url "https://acme-v02.api.letsencrypt.org/directory"
account key "/etc/acme/letsencrypt-privkey.pem"
}
authority letsencrypt-staging {
api url "https://acme-staging-v02.api.letsencrypt.org/directory"
account key "/etc/acme/letsencrypt-staging-privkey.pem"
}
authority buypass {
api url "https://api.buypass.com/acme/directory"
account key "/etc/acme/buypass-privkey.pem"
contact "mailto:me@example.com"
}
authority buypass-test {
api url "https://api.test4.buypass.no/acme/directory"
account key "/etc/acme/buypass-test-privkey.pem"
contact "mailto:me@example.com"
}
<%
our $primary = $is_primary->($vio0_ip);
our $prefix = $primary ? '' : 'www.';
%>
<% for my $host (@$acme_hosts) { %>
domain <%= $prefix.$host %> {
domain key "/etc/ssl/private/<%= $prefix.$host %>.key"
domain full chain certificate "/etc/ssl/<%= $prefix.$host %>.fullchain.pem"
sign with letsencrypt
}
<% } %>
</pre>
<br />
<span>And this is the <span class='inlinecode'>acme.sh.tpl</span>:</span><br />
<br />
<pre>
#!/bin/sh
<%
our $primary = $is_primary->($vio0_ip);
our $prefix = $primary ? '' : 'www.';
-%>
function handle_cert {
host=$1
# Create symlink, so that relayd also can read it.
crt_path=/etc/ssl/$host
if [ -e $crt_path.crt ]; then
rm $crt_path.crt
fi
ln -s $crt_path.fullchain.pem $crt_path.crt
# Requesting and renewing certificate.
/usr/sbin/acme-client -v $host
}
has_update=no
<% for my $host (@$acme_hosts) { -%>
handle_cert <%= $prefix.$host %>
if [ $? -eq 0 ]; then
has_update=yes
fi
<% } -%>
# Pick up the new certs.
if [ $has_update = yes ]; then
/usr/sbin/rcctl reload httpd
/usr/sbin/rcctl reload relayd
/usr/sbin/rcctl restart smtpd
fi
</pre>
<br />
<h3 style='display: inline' id='Servicerexification'>Service rexification </h3><br />
<br />
<span>These are the Rex tasks setting up <span class='inlinecode'>httpd</span>, <span class='inlinecode'>relayd</span> and <span class='inlinecode'>smtpd</span> services:</span><br />
<br />
<pre>
desc 'Setup httpd';
task 'httpd', group => 'frontends',
sub {
append_if_no_such_line '/etc/rc.conf.local', 'httpd_flags=';
file '/etc/httpd.conf',
content => template('./etc/httpd.conf.tpl',
acme_hosts => \@acme_hosts,
is_primary => $is_primary),
owner => 'root',
group => 'wheel',
mode => '644',
on_change => sub { service 'httpd' => 'restart' };
service 'httpd', ensure => 'started';
};
desc 'Setup relayd';
task 'relayd', group => 'frontends',
sub {
append_if_no_such_line '/etc/rc.conf.local', 'relayd_flags=';
file '/etc/relayd.conf',
content => template('./etc/relayd.conf.tpl',
ipv6address => $ipv6address,
is_primary => $is_primary),
owner => 'root',
group => 'wheel',
mode => '600',
on_change => sub { service 'relayd' => 'restart' };
service 'relayd', ensure => 'started';
};
desc 'Setup OpenSMTPD';
task 'smtpd', group => 'frontends',
sub {
Rex::Logger::info('Dealing with mail aliases');
file '/etc/mail/aliases',
source => './etc/mail/aliases',
owner => 'root',
group => 'wheel',
mode => '644',
on_change => sub { say run 'newaliases' };
Rex::Logger::info('Dealing with mail virtual domains');
file '/etc/mail/virtualdomains',
source => './etc/mail/virtualdomains',
owner => 'root',
group => 'wheel',
mode => '644',
on_change => sub { service 'smtpd' => 'restart' };
Rex::Logger::info('Dealing with mail virtual users');
file '/etc/mail/virtualusers',
source => './etc/mail/virtualusers',
owner => 'root',
group => 'wheel',
mode => '644',
on_change => sub { service 'smtpd' => 'restart' };
Rex::Logger::info('Dealing with smtpd.conf');
file '/etc/mail/smtpd.conf',
content => template('./etc/mail/smtpd.conf.tpl',
is_primary => $is_primary),
owner => 'root',
group => 'wheel',
mode => '644',
on_change => sub { service 'smtpd' => 'restart' };
service 'smtpd', ensure => 'started';
};
</pre>
<br />
<span>This is the <span class='inlinecode'>httpd.conf.tpl</span>:</span><br />
<br />
<pre>
<%
our $primary = $is_primary->($vio0_ip);
our $prefix = $primary ? '' : 'www.';
%>
# Plain HTTP for ACME and HTTPS redirect
<% for my $host (@$acme_hosts) { %>
server "<%= $prefix.$host %>" {
listen on * port 80
location "/.well-known/acme-challenge/*" {
root "/acme"
request strip 2
}
location * {
block return 302 "https://$HTTP_HOST$REQUEST_URI"
}
}
<% } %>
# Gemtexter hosts
<% for my $host (qw/foo.zone snonux.land/) { %>
server "<%= $prefix.$host %>" {
listen on * tls port 443
tls {
certificate "/etc/ssl/<%= $prefix.$host %>.fullchain.pem"
key "/etc/ssl/private/<%= $prefix.$host %>.key"
}
location * {
root "/htdocs/gemtexter/<%= $host %>"
directory auto index
}
}
<% } %>
# DTail special host
server "<%= $prefix %>dtail.dev" {
listen on * tls port 443
tls {
certificate "/etc/ssl/<%= $prefix %>dtail.dev.fullchain.pem"
key "/etc/ssl/private/<%= $prefix %>dtail.dev.key"
}
location * {
block return 302 "https://github.dtail.dev$REQUEST_URI"
}
}
# Irregular Ninja special host
server "<%= $prefix %>irregular.ninja" {
listen on * tls port 443
tls {
certificate "/etc/ssl/<%= $prefix %>irregular.ninja.fullchain.pem"
key "/etc/ssl/private/<%= $prefix %>irregular.ninja.key"
}
location * {
root "/htdocs/irregular.ninja"
directory auto index
}
}
# buetow.org special host.
server "<%= $prefix %>buetow.org" {
listen on * tls port 443
tls {
certificate "/etc/ssl/<%= $prefix %>buetow.org.fullchain.pem"
key "/etc/ssl/private/<%= $prefix %>buetow.org.key"
}
block return 302 "https://paul.buetow.org"
}
server "<%= $prefix %>paul.buetow.org" {
listen on * tls port 443
tls {
certificate "/etc/ssl/<%= $prefix %>paul.buetow.org.fullchain.pem"
key "/etc/ssl/private/<%= $prefix %>paul.buetow.org.key"
}
block return 302 "https://foo.zone/contact-information.html"
}
server "<%= $prefix %>tmp.buetow.org" {
listen on * tls port 443
tls {
certificate "/etc/ssl/<%= $prefix %>tmp.buetow.org.fullchain.pem"
key "/etc/ssl/private/<%= $prefix %>tmp.buetow.org.key"
}
root "/htdocs/buetow.org/tmp"
directory auto index
}
</pre>
<br />
<span>and this the <span class='inlinecode'>relayd.conf.tpl</span>:</span><br />
<br />
<pre>
<%
our $primary = $is_primary->($vio0_ip);
our $prefix = $primary ? '' : 'www.';
%>
log connection
tcp protocol "gemini" {
tls keypair <%= $prefix %>foo.zone
tls keypair <%= $prefix %>buetow.org
}
relay "gemini4" {
listen on <%= $vio0_ip %> port 1965 tls
protocol "gemini"
forward to 127.0.0.1 port 11965
}
relay "gemini6" {
listen on <%= $ipv6address->($hostname) %> port 1965 tls
protocol "gemini"
forward to 127.0.0.1 port 11965
}
</pre>
<br />
<span>And last but not least, this is the <span class='inlinecode'>smtpd.conf.tpl</span>:</span><br />
<br />
<pre>
<%
our $primary = $is_primary->($vio0_ip);
our $prefix = $primary ? '' : 'www.';
%>
pki "buetow_org_tls" cert "/etc/ssl/<%= $prefix %>buetow.org.fullchain.pem"
pki "buetow_org_tls" key "/etc/ssl/private/<%= $prefix %>buetow.org.key"
table aliases file:/etc/mail/aliases
table virtualdomains file:/etc/mail/virtualdomains
table virtualusers file:/etc/mail/virtualusers
listen on socket
listen on all tls pki "buetow_org_tls" hostname "<%= $prefix %>buetow.org"
#listen on all
action localmail mbox alias <aliases>
action receive mbox virtual <virtualusers>
action outbound relay
match from any for domain <virtualdomains> action receive
match from local for local action localmail
match from local for any action outbound
</pre>
<br />
<h2 style='display: inline' id='Allpiecestogether'>All pieces together</h2><br />
<br />
<span>For the complete <span class='inlinecode'>Rexfile</span> example and all the templates, please look at the Git repository:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/rexfiles'>https://codeberg.org/snonux/rexfiles</a><br />
<br />
<span>Besides ACME, other things, such as DNS servers, are also rexified. The following command will run all the Rex tasks and configure everything on my frontend machines automatically:</span><br />
<br />
<pre>
rex commons
</pre>
<br />
<span>The <span class='inlinecode'>commons</span> is a group of tasks I specified which combines a set of common tasks I always want to execute on all frontend machines. This also includes the ACME tasks mentioned in this article!</span><br />
<br />
<h2 style='display: inline' id='Conclusion'>Conclusion</h2><br />
<br />
<span>ACME and Let's Encrypt greatly help reduce recurring manual maintenance work (creating and renewing certificates). Furthermore, all the certificates are free of cost! I love to use OpenBSD and Rex to automate all of this.</span><br />
<br />
<span>OpenBSD suits perfectly here as all the tools are already part of the base installation. But I like underdogs. Rex is not as powerful and popular as other configuration management systems (e.g. Puppet, Chef, SALT or even Ansible). It is more of an underdog, and the community is small.</span><br />
<br />
<span>Why re-inventing the wheel? I love that a <span class='inlinecode'>Rexfile</span> is just a Perl DSL. Also, OpenBSD comes with Perl in the base system. So no new programming language had to be added to my mix for the configuration management system. Also, the <span class='inlinecode'>acme.sh</span> shell script is not a Bash but a standard Bourne shell script, so I didn't have to install an additional shell as OpenBSD does not come with the Bash pre-installed.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other *BSD related posts are:</span><br />
<br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex (You are currently reading this)</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Sweating the small stuff - Tiny projects of mine</title>
<link href="gemini://foo.zone/gemfeed/2022-06-15-sweating-the-small-stuff.gmi" />
<id>gemini://foo.zone/gemfeed/2022-06-15-sweating-the-small-stuff.gmi</id>
<updated>2022-06-15T08:47:44+01:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This blog post is a bit different from the others. It consists of multiple but smaller projects worth mentioning. I got inspired by Julia Evan's 'Tiny programs' blog post and the side projects of The Sephist, so I thought I would also write a blog posts listing a couple of small projects of mine:</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>Sweating the small stuff - Tiny projects of mine</h1><br />
<br />
<span class='quote'>Published at 2022-06-15T08:47:44+01:00; Updated at 2022-06-18</span><br />
<br />
<pre>
_
/_/_ .'''.
=O(_)))) ...' `.
jgs \_\ `. .'''
`..'
</pre>
<br />
<span>This blog post is a bit different from the others. It consists of multiple but smaller projects worth mentioning. I got inspired by Julia Evan's "Tiny programs" blog post and the side projects of The Sephist, so I thought I would also write a blog posts listing a couple of small projects of mine:</span><br />
<br />
<a class='textlink' href='https://jvns.ca/blog/2022/03/08/tiny-programs/'>Tiny programs</a><br />
<a class='textlink' href='https://thesephist.com/projects/'>The Sephist's project list</a><br />
<br />
<span>Working on tiny projects is a lot of fun as you don't need to worry about any standards or code reviews and you decide how and when you work on it. There aren't restrictions regarding technologies used. You are likely the only person working on these tiny projects and that means that there is no conflict with any other developers. This is complete freedom :-).</span><br />
<br />
<span>But before going through the tiny projects let's take a paragraph for the <span class='inlinecode'>1y</span> anniversary retrospective.</span><br />
<br />
<h2 style='display: inline'><span class='inlinecode'>1y</span> anniversary</h2><br />
<br />
<span>It has been one year since I started posting regularly (at least once monthly) on this blog again. It has been a lot of fun (and work) doing so for various reasons:</span><br />
<br />
<ul>
<li>I practice English writing (I am not a native speaker). I am far from being a novelist, but this blog helps improves my writing skills. I also tried out tools like Grammarly.com and Languagetool.org and also worked with <span class='inlinecode'>:spell</span> in Vim or the LibreOffice checker. This post was checked with the <span class='inlinecode'>write-better</span> Node application. </li>
<li>I force myself to "finish" some kind of project worth writing about every month. If its not a project, then its still a topic which requires research and deep thinking. Producing 2k words of text can actually be challenging.</li>
<li>It's fun to rely on KISS (keep it simple & stupid) tools. E.g. use of Gemtexter and not WordPress, use of Vim instead of an office suite or a rich web editor.</li>
</ul><br />
<span>Retrospectively, these have been the most popular blog posts of mine over the last year:</span><br />
<br />
<a class='textlink' href='./2021-09-12-keep-it-simple-and-stupid.html'>Keep it simple and stupid</a><br />
<a class='textlink' href='./2022-04-10-creative-universe.html'>Creative universe</a><br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>Bash Golf series</a><br />
<a class='textlink' href='./2021-12-26-how-to-stay-sane-as-a-devops-person.html'>How to stay sane as a DevOps person</a><br />
<a class='textlink' href='./2022-05-27-perl-is-still-a-great-choice.html'>Perl is still a great choice</a><br />
<br />
<span>But now, let's continue with the small projects worth mentioning :-)</span><br />
<br />
<h2 style='display: inline'>Static photo album generator</h2><br />
<br />
<span><span class='inlinecode'>photoalbum.sh</span> is a minimal static HTML photo album generator. I use it to drive "The Irregular Ninja" site and for some ad-hoc (personal) albums to share photos with the family and friends.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/photoalbum'>https://codeberg.org/snonux/photoalbum</a><br />
<br />
<h3 style='display: inline'>The Irregular Ninja</h3><br />
<br />
<span>Photography is one of my casual hobbies. I love to capture interesting perspectives and motifs. I love to walk new streets and neighbourhoods I never walked before so I can capture those unexpected motifs, colours and moments. Unfortunately, because of time constraints (and sometime weather constraints), I do that on a pretty infrequent basis.</span><br />
<br />
<a href='./2022-06-15-sweating-the-small-stuff/ninja.jpg'><img src='./2022-06-15-sweating-the-small-stuff/ninja.jpg' /></a><br />
<br />
<span>More than 10 years ago I wrote the bespoke small static photo album generator in Bash <span class='inlinecode'>photoalbum.sh</span> which I recently refactored to a modern Bash coding style and also freshened up the Cascading Style Sheets. Last but not least, the new domain name <span class='inlinecode'>irregular.ninja</span> has been registered.</span><br />
<br />
<span>The thumbnails are presented in a random order and there are also random CSS effects for each preview. There's also a simple background blur for each page generated. And that's all in less than 300 lines of Bash code! The script requires ImageMagick (available for all common Linux and *BSD distributions) to be installed.</span><br />
<br />
<span>As you can see, there is a lot of randomization and irregularity going on. Thus, the name "Irregular Ninja" was born.</span><br />
<br />
<a class='textlink' href='https://irregular.ninja'>https://irregular.ninja</a><br />
<br />
<span>I only use a digital compact camera or a smartphone to take the photos. I don't like the idea of carrying around a big camera with me "just in case" so I keep it small and simple. The best camera is the camera you have with you. :-)</span><br />
<br />
<span>I hope you like this photo site. It's worth checking it out again around once every other month!</span><br />
<br />
<h2 style='display: inline'>Random journal page extractor</h2><br />
<br />
<span>I bullet journal. I write my notes into a Leuchtturm paper notebook. Once full, I am scanning it to a PDF file and archive it. As of writing this, I am at journal #7 (each from 123 up to 251 pages in A5). It means that there is a lot of material already.</span><br />
<br />
<span>Once in a while I want to revisit older notes and ideas. For that I have written a simple Bash script <span class='inlinecode'>randomjournalpage.sh</span> which randomly picks a PDF file from a folder and extracts 42 pages from it at a random page offset and opens them in a PDF viewer (Evince in this case, as I am a GNOME user). </span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/randomjournalpage'>https://codeberg.org/snonux/randomjournalpage</a><br />
<br />
<span>There's also a weekly <span class='inlinecode'>CRON</span> job on my servers to send me a reminder that I might want to read in my old journals again. My laptop also runs this script each time it boots and saves the output to a NextCloud folder. From there, it's synchronized to the NextCloud server so I can pick it up from there with my smartphone later when I am "on the road".</span><br />
<br />
<h2 style='display: inline'>Global uptime records statistic generator</h2><br />
<br />
<span><span class='inlinecode'>guprecords</span> is a Perl script which reads multiple <span class='inlinecode'>uprecord</span> files (produced by <span class='inlinecode'>uptimed</span> - a widely available daemon for recording server uptimes) and generates uptime statistics of multiple hosts combined. I keep all the record files of all my personal computers in a Git repository (I even keep the records of the boxes I don't own or use anymore) and there's already quite a collection of it. It looks like this:</span><br />
<br />
<pre>
❯ perl ~/git/guprecords/src/guprecords --indir=./stats/ --count=20 --all
Pos | System | Kernel | Uptime | Boot time
1 | sun | FreeBSD 10.1-RELEA.. | 502d 03:29:19 | Sun Aug 16 15:56:40 2015
2 | vulcan | Linux 3.10.0-1160... | 313d 13:19:39 | Sun Jul 25 18:32:25 2021
3 | uugrn | FreeBSD 10.2-RELEASE | 303d 15:19:35 | Tue Dec 22 21:33:07 2015
4 | uugrn | FreeBSD 11.0-RELEA.. | 281d 14:38:04 | Fri Oct 21 15:22:02 2016
5 | deltavega | Linux 3.10.0-957.2.. | 279d 11:15:00 | Sun Jun 30 11:42:38 2019
6 | vulcan | Linux 3.10.0-957.2.. | 279d 11:12:14 | Sun Jun 30 11:43:41 2019
7 | deltavega | Linux 3.10.0-1160... | 253d 04:42:22 | Sat Apr 24 13:34:34 2021
8 | host0 | FreeBSD 6.2-RELEAS.. | 240d 02:23:23 | Wed Jan 31 20:34:46 2007
9 | uugrn | FreeBSD 11.1-RELEA.. | 202d 21:12:41 | Sun May 6 18:06:17 2018
10 | tauceti | Linux 3.2.0-4-amd64 | 197d 18:45:40 | Mon Dec 16 19:47:54 2013
11 | pluto | Linux 2.6.32-5-amd64 | 185d 11:53:04 | Wed Aug 1 07:34:10 2012
12 | sun | FreeBSD 10.3-RELEA.. | 164d 22:31:55 | Sat Jul 22 18:47:21 2017
13 | vulcan | Linux 3.10.0-1160... | 161d 07:08:43 | Sun Feb 14 10:05:38 2021
14 | sun | FreeBSD 10.3-RELEA.. | 158d 21:18:36 | Sat Jan 27 10:18:57 2018
15 | uugrn | FreeBSD 11.1-RELEA.. | 157d 20:57:24 | Fri Nov 3 05:02:54 2017
16 | tauceti-f | Linux 3.2.0-3-amd64 | 150d 04:12:38 | Mon Sep 16 09:02:58 2013
17 | tauceti | Linux 3.2.0-4-amd64 | 149d 09:21:43 | Mon Aug 11 09:47:50 2014
18 | pluto | Linux 3.2.0-4-amd64 | 142d 02:57:31 | Mon Sep 8 01:59:02 2014
19 | tauceti-f | Linux 3.2.0-3-amd64 | 132d 22:46:26 | Mon May 6 11:11:35 2013
20 | keppler-16b | Darwin 13.4.0 | 131d 08:17:12 | Thu Jun 11 10:44:25 2015
</pre>
<br />
<span>It can also sum up all uptimes for each host to generate a total per host uptime top list:</span><br />
<br />
<pre>
❯ perl ~/git/guprecords/src/guprecords --indir=./stats/ --count=20 --total
Pos | System | Kernel | Uptime |
1 | uranus | Linux 5.4.17-200.f.. | 1419d 19:05:39 |
2 | sun | FreeBSD 10.1-RELEA.. | 1363d 11:41:14 |
3 | vulcan | Linux 3.10.0-1160... | 1262d 20:27:48 |
4 | uugrn | FreeBSD 10.2-RELEASE | 1219d 15:10:16 |
5 | deltavega | Linux 3.10.0-957.2.. | 1115d 06:33:55 |
6 | pluto | Linux 2.6.32-5-amd64 | 1086d 10:44:05 |
7 | tauceti | Linux 3.2.0-4-amd64 | 846d 12:58:21 |
8 | tauceti-f | Linux 3.2.0-3-amd64 | 625d 07:16:39 |
9 | host0 | FreeBSD 6.2-RELEAS.. | 534d 19:50:13 |
10 | keppler-16b | Darwin 13.4.0 | 448d 06:15:00 |
11 | tauceti-e | Linux 3.2.0-4-amd64 | 415d 18:14:13 |
12 | moon | Darwin 18.7.0 | 326d 11:21:42 |
13 | callisto | Linux 4.0.4-303.fc.. | 303d 12:18:24 |
14 | alphacentauri | FreeBSD 10.1-RELEA.. | 300d 20:15:00 |
15 | earth | Linux 5.13.14-200... | 289d 08:05:05 |
16 | makemake | Linux 5.11.9-200.f.. | 286d 21:53:03 |
17 | london | Linux 3.2.0-4-amd64 | 258d 15:10:38 |
18 | fishbone | OpenBSD 4.1 .. | 223d 05:55:26 |
19 | sagittarius | Darwin 15.6.0 | 198d 23:53:59 |
20 | mars | Linux 3.2.0-4-amd64 | 190d 05:44:21 |
</pre>
<br />
<a class='textlink' href='https://codeberg.org/snonux/guprecords'>https://codeberg.org/snonux/guprecords</a><br />
<br />
<span>This all is of no real practical use but fun!</span><br />
<br />
<h2 style='display: inline'>Server configuration management</h2><br />
<br />
<span>The <span class='inlinecode'>rexfiles</span> project contains all Rex files for my (personal) server setup automation. A <span class='inlinecode'>Rexfile</span> is written in a Perl DSL run by the Rex configuration management system. It's pretty much KISS and that's why I love it. It suits my personal needs perfectly. </span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/rexfiles'>https://codeberg.org/snonux/rexfiles</a><br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<br />
<span>This is an E-Mail I posted to the Rex mailing list:</span><br />
<br />
<span class='quote'>Hi there! I was searching for a simple way to automate my personal OpenBSD setup. I found that configuration management systems like Puppet, Salt, Chef, etc.. were too bloated for my personal needs. So for a while I was configuring everything by hand. At one point I got fed up and started writing Shell scripts. But that was not the holy grail so that I looked at Ansible. I found that Ansible had some dependencies on Python on the target machine when you want to use all the features. Furthermore, I am not really familiar with Python. But then I remembered that there was also Rex. It's written in my beloved Perl. Also, OpenBSD comes with Perl in the base system out of the box which makes it integrate better than all my scripts (automation and also scripts deployed via the automation to the system) are all in the same language. Rex may not have all the features like other configuration management systems, but its easy to work-around or extend when you know Perl. Thanks!</span><br />
<br />
<h2 style='display: inline'>Fancy SSH execution loop</h2><br />
<br />
<span><span class='inlinecode'>rubyfy</span> is a fancy SSH loop wrapper written in Ruby for running shell commands on multiple remote servers at once. I also forked this project for work (under a different name) where I added even more features such as automatic server discovery. It's used by many colleagues on a frequent basis. Here are some examples:</span><br />
<br />
<pre>
# Run command 'hostname' on server foo.example.com
./rubyfy.rb -c 'hostname' <<< foo.example.com
# Run command 'id' as root (via sudo) on all servers listed in the list file
# Do it on 10 servers in parallel
./rubyfy.rb --parallel 10 --root --command 'id' < serverlist.txt
# Run a fancy script in background on 50 servers in parallel
./rubyfy.rb -p 50 -r -b -c '/usr/local/scripts/fancy.zsh' < serverlist.txt
# Grep for specific process on both servers and write output to ./out/grep.txt
echo {foo,bar}.example.com | ./rubyfy.rb -p 10 -c 'pgrep -lf httpd' -n grep.txt
# Reboot server only if file /var/run/maintenance.lock does NOT exist!
echo foo.example.com |
./rubyfy.rb --root --command reboot --precondition /var/run/maintenance.lock
</pre>
<br />
<a class='textlink' href='https://codeberg.org/snonux/rubyfy'>https://codeberg.org/snonux/rubyfy</a><br />
<br />
<h2 style='display: inline'>A KISS dynamic DNS solution</h2><br />
<br />
<span><span class='inlinecode'>dyndns</span> is a tiny shell script which implements "your" own DynDNS service. It relies on SSH access to the authoritative DNS server and the <span class='inlinecode'>nsupdate</span> command. There is really no need to use any of the "other" free DynDNS services out there.</span><br />
<br />
<span>Syntax (this must run from the client connecting to the DNS server through SSH): </span><br />
<br />
<pre>
ssh dyndns@dyndnsserver /path/to/dyndns-update \
your.host.name. TYPE new-entry TIMEOUT
</pre>
<br />
<span>This is a real world example: </span><br />
<br />
<pre>
ssh dyndns@dyndnsserver /path/to/dyndns-update \
local.buetow.org. A 137.226.50.91 30
</pre>
<br />
<a class='textlink' href='https://codeberg.org/snonux/dyndns'>https://codeberg.org/snonux/dyndns</a><br />
<br />
<h2 style='display: inline'>CPU information gatherer for Linux</h2><br />
<br />
<span>This is a tiny GNU Awk script for Linux which displays information about the CPU. All what it does is presenting <span class='inlinecode'>/proc/cpuinfo</span> in an easier to read way. The output is somewhat more compact than the standard <span class='inlinecode'>lscpu</span> command you find commonly on Linux distributions.</span><br />
<br />
<pre>
❯ ./cpuinfo
cpuinfo (c) 1.0.2 Paul Buetow
11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz GenuineIntel 12288 KB cache
p = 001 Physical processors
c = 004 Cores
s = 008 Siblings (Hyper-Threading enabled if s != c)
v = 008 [v = p*c*(s != c ? 2 : 1)] Total logical CPUs
Hyper-Threading is enabled
0003000 MHz each core
0012000 MHz total
0005990 Bogomips each processor (including virtual)
0023961 Bogomips total
</pre>
<br />
<a class='textlink' href='https://codeberg.org/snonux/cpuinfo'>https://codeberg.org/snonux/cpuinfo</a><br />
<br />
<h2 style='display: inline'>Show differences of two files over the network</h2><br />
<br />
<span>This is a shell wrapper to use the standard diff tool over the network to compare a file between two computers. It uses NetCat for the network part and also encrypts all traffic using OpenSSL. This is how its used:</span><br />
<br />
<span>1. Open two terminal windows and login to two different hosts (you could use ClusterSSH or <span class='inlinecode'>tmux</span> here). 2. Run on the first host <span class='inlinecode'>netdiff otherhost.example.org /file/to/diff.txt</span> and run on the second host <span class='inlinecode'>netdiff firsthost.example.org /file/to/diff.txt</span>. 3. You then will see the file differences.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/netdiff'>https://codeberg.org/snonux/netdiff</a><br />
<br />
<h2 style='display: inline'>Delay sending out E-Mails with Mutt</h2><br />
<br />
<span>This is a shell script for the Mutt email client for delaying sending out E-Mails. For example, you want to write an email on Saturday but don't want to bother the recipient earlier than Monday. It relies on CRON.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/muttdelay'>https://codeberg.org/snonux/muttdelay</a><br />
<br />
<h2 style='display: inline'>Graphical UI for sending text messages</h2><br />
<br />
<span><span class='inlinecode'>jsmstrade</span> is a minimalistic graphical Java swing client for sending SMS messages over the SMStrade service.</span><br />
<br />
<a href='./2022-06-15-sweating-the-small-stuff/jsmstrade.png'><img src='./2022-06-15-sweating-the-small-stuff/jsmstrade.png' /></a><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/jsmstrade'>https://codeberg.org/snonux/jsmstrade</a><br />
<a class='textlink' href='https://smstrade.de'>https://smstrade.de</a><br />
<br />
<h2 style='display: inline'>IPv6 and IPv4 connectivity testing site</h2><br />
<br />
<span><span class='inlinecode'>ipv6test</span> is a quick and dirty Perl CGI script for testing whether your browser connects via IPv4 or IPv6. It requires you to setup three sub-domains: One reachable only via IPv4 (e.g. <span class='inlinecode'>test4.ipv6.buetow.org</span>), another reachable only via IPv6 (e.g. <span class='inlinecode'>test6.ipv6.buetow.org</span>) and the main one reachable through both protocols (e.g. <span class='inlinecode'>ipv6.buetow.org</span>).</span><br />
<br />
<span>I don't have it running on any of my servers at the moment. This means that there is no demo to show now. Sorry!</span><br />
<br />
<h2 style='display: inline'>List open Jira tickets in the terminal</h2><br />
<br />
<span><span class='inlinecode'>japi</span> s a small Perl script for listing open Jira issues. It might be broken by now as the Jira APIs may have changed. Sorry! But feel free to fork and modernize it. :-)</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/jsmstrade'>https://codeberg.org/snonux/jsmstrade</a><br />
<span> </span><br />
<h2 style='display: inline'>Debian running on "your" Android phone</h2><br />
<br />
<span>Debroid is a tutorial and a set of scripts to install and to run a Debian <span class='inlinecode'>chroot</span> on an Android phone.</span><br />
<br />
<a class='textlink' href='./2015-12-05-run-debian-on-your-phone-with-debroid.html'>Check out my previous post about it</a><br />
<br />
<span>I am not using Debroid anymore as I have switched to Termux now.</span><br />
<br />
<a class='textlink' href='https://termux.com'>https://termux.com</a><br />
<br />
<h2 style='display: inline'>Perl service framework</h2><br />
<br />
<span>PerlDaemon is a minimal daemon for Linux and other Unix like operating systems programmed in Perl. It is a minimal but pretty functional and fairly generic service framework. This means that it does not do anything useful other than providing a framework for starting, stopping, configuring and logging. To do something useful, a module (written in Perl) must be provided.</span><br />
<br />
<a class='textlink' href='./2011-05-07-perl-daemon-service-framework.html'>Checkout my previous post about it</a><br />
<br />
<h2 style='display: inline'>More</h2><br />
<br />
<span>There are more projects on my Codeberg page but they aren't as tiny as the ones mentioned in this post or aren't finished yet so I won't bother listing them here. However, there also a few more scripts used frequently by me (not publicly accessible (yet?)) which I would like to mention here:</span><br />
<br />
<h3 style='display: inline'>Work time tracker</h3><br />
<br />
<span><span class='inlinecode'>worktime.rb</span>, for example, is a command line Ruby script I use to track my time spent working. This is to make sure that I don't overwork (in particular useful when working from home). It also generates some daily and weekly stats and carries over work time (surpluses or minuses) to the next work day, week or even year.</span><br />
<br />
<span>It has some special features such as tracking time for self-improvement/development, days off and time spent at the lunch break and time spent on Pet Projects.</span><br />
<br />
<span>An example weekly report looks like this (I often don't track my lunch time but what I do instead I stop the work timer when I go out for lunch and start the work timer once back at the desk):</span><br />
<br />
<pre>
Mon 20211213 50: work:5.92h
Tue 20211214 50: work:7.47h lunch:0.50h pet:0.42h
Wed 20211215 50: work:8.86h pet:0.50h
Thu 20211216 50: work:8.02h pet:0.50h
Fri 20211217 50: work:9.81h
* Sat 20211218 50: work:0.00h selfdevelopment:1.00h
* Sun 20211219 50: work:2.08h pet:1.00h selfdevelopment:-2.08h
================================================
balance:0.06h work:42.15h lunch:0.50h pet:2.42h selfdevelopment:-1.08h buffer:8.38h
</pre>
<br />
<span>All I do when I start work is to run the <span class='inlinecode'>wtlogin</span> command and after finishing work to run the <span class='inlinecode'>wtlogout</span> command. My shell will remind me when I work without having logged in. It uses a simple JSON database which is editable with <span class='inlinecode'>wtedit</span> (this opens the JSON in Vim). The report shown above can be generated with <span class='inlinecode'>wtreport</span>. Any out-of-bounds reporting can be added with the <span class='inlinecode'>wtadd</span> command.</span><br />
<br />
<h3 style='display: inline'>Password and document store</h3><br />
<br />
<span><span class='inlinecode'>geheim.rb</span> is my personal password and document store ("geheim" is the German word for secret). It's written in Ruby and heavily relies on Git, FZF (for search), Vim and standard encryption algorithms. Other than the standard <span class='inlinecode'>pass</span> Unix password manager, <span class='inlinecode'>geheim</span> also encrypts the file names and password titles.</span><br />
<br />
<span>The tool is command line driven but also provides an interactive shell when invoked with <span class='inlinecode'>geheim shell</span>. It also works on my Android phone via Termux so I have all my documents and passwords always with me. </span><br />
<br />
<h3 style='display: inline'>Backup procedure</h3><br />
<br />
<span><span class='inlinecode'>backup</span> is a Bash script which does run once daily (or every time on boot) on my home FreeBSD NAS server and performs backup related tasks such as creating a local backup of my remote NextCloud instance, creating encrypted (incremental) ZFS snapshots of everything what's stored on the NAS and synchronizes (via <span class='inlinecode'>rsync</span>) backups to a remote cloud storage. It also can synchronize backups to a local external USB drive.</span><br />
<br />
<a class='textlink' href='./2016-04-03-offsite-backup-with-zfs.html'>Check out my offsite backup series</a><br />
<br />
<h2 style='display: inline'>konpeito.media</h2><br />
<br />
<span>Here's a bonus...</span><br />
<br />
<pre>
▄ █ ▄ ▄ █ ▄ ▄ █ ▄
▄▀█▀▄ ▄▀█▀▄ ▄▀█▀▄
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ ▀ ▀ ▀
█ ▄▄ ▄▄ █
█ █ █▀▀▀█ █ █ █ ▄▀ ▄▀▀▀▀▄ █▄ █ █▀▀▀▀▀▄ ▄▀▀▀▀▄ █ ▀▀▀█▀▀▀ ▄▀▀▀▀▄
█ ▀▀▀▀▀▀▀▀▀ █ █ ▄█ █ █ █ ▀▄ █ █▄▄▄▄▄▀ █▄▄▄▄▄▄█ █ █ █ █
█ ▄▀▀▀▀▀▀▀▀▀▀▀▄ █ █▀ ▀▄ ▀▄ ▄▀ █ ▀▄█ █ ▀▄ ▄ █ █ ▀▄ ▄▀
▀▄█▄█▄▄▄▄▄▄▄█▄█▄▀ ▀ ▀ ▀▀▀▀ ▀ ▀ ▀ ▀▀▀▀ ▀ ▀ ▀▀▀
</pre>
<br />
<span>*THIS ISN'T MY PROJECT* but I found KONPEITO an interesting Gemini capsule. It's a quarterly released Low-Fi music mix tape distributed only through Gemini (and not the web). </span><br />
<br />
<a class='textlink' href='gemini://konpeito.media'>gemini://konpeito.media</a><br />
<br />
<span>If you wonder what Gemini is:</span><br />
<br />
<a class='textlink' href='./2021-04-24-welcome-to-the-geminispace.html'>Welcome to the Geminispae</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Perl is still a great choice</title>
<link href="gemini://foo.zone/gemfeed/2022-05-27-perl-is-still-a-great-choice.gmi" />
<id>gemini://foo.zone/gemfeed/2022-05-27-perl-is-still-a-great-choice.gmi</id>
<updated>2022-05-27T07:50:12+01:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Perl (the Practical Extraction and Report Language) is a battle-tested, mature, multi-paradigm dynamic programming language. Note that it's not called PERL, neither P.E.R.L. nor Pearl. 'Perl' is the name of the language and `perl` the name of the interpreter or the interpreter command.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='Perlisstillagreatchoice'>Perl is still a great choice</h1><br />
<br />
<span class='quote'>Published at 2022-05-27T07:50:12+01:00; Updated at 2023-01-28</span><br />
<br />
<a href='./2022-05-27-perl-is-still-a-great-choice/regular_expressions.png'><img alt='Comic source: XKCD' title='Comic source: XKCD' src='./2022-05-27-perl-is-still-a-great-choice/regular_expressions.png' /></a><br />
<br />
<span>Perl (the Practical Extraction and Report Language) is a battle-tested, mature, multi-paradigm dynamic programming language. Note that it's not called PERL, neither P.E.R.L. nor Pearl. "Perl" is the name of the language and <span class='inlinecode'>perl</span> the name of the interpreter or the interpreter command.</span><br />
<br />
<span>Unfortunately (it makes me sad), Perl's popularity has been declining over the last years as Google trends shows:</span><br />
<br />
<a href='./2022-05-27-perl-is-still-a-great-choice/googletrendsperl.jpg'><img src='./2022-05-27-perl-is-still-a-great-choice/googletrendsperl.jpg' /></a><br />
<br />
<span>So why is that? Once the de-facto standard super-glue language for the web nowadays seems to have a bad reputation. Often, people state:</span><br />
<br />
<ul>
<li>Perl is a write-only language. Nobody can read Perl code.</li>
<li>Perl? Isn't it abandoned? It's still at version 5!</li>
<li>Why use Perl as there are better alternatives?</li>
<li>Why all the sigils? It looks like an exploding ASCII factory!!</li>
</ul><br />
<h2 style='display: inline' id='Writeonlylanguage'>Write-only language</h2><br />
<br />
<span>Is Perl really a write-only language? You have to understand that Perl 5 was released in 1994 (28 years ago as of this writing) and when we refer to Perl we usually mean Perl 5. That's many years, and there are many old scripts not following the modern Perl best practices (as they didn't exist yet). So yes, legacy scripts may be difficult to read. Japanese may be difficult to read too if you don't know Japanese, though.</span><br />
<br />
<span>To come back to the question: Is Perl a write-only language? I don't think so. Like in any other language, you have to apply best practices in order to keep your code maintainable. Some other programming languages enforce best practices, but that makes these languages less expressive. Perl follows the principles "there is more than one way to do it" (aka TIMTOWDI) and "making easy things easy and hard things possible".</span><br />
<br />
<span>Perl gives the programmer more flexibility in how to do things, and this results in a stronger learning curve than for lesser expressive languages like for example Go or Python. But, like in everything in life, common sense has to be applied. You should not take TIMTOWDI to the extreme in a production piece of code. In my personal opinion, it is also more satisfying to program in an expressive language.</span><br />
<br />
<span>Some good books on "good" Perl I can recommend are:</span><br />
<br />
<a class='textlink' href='http://modernperlbooks.com'>Modern Perl</a><br />
<a class='textlink' href='https://hop.perl.plover.com'>Higher Order Perl</a><br />
<br />
<span>Due to Perl's expressiveness you will find a lot of obscure code in the interweb in form of obfuscation, fancy email signatures (JAPHs), art, polyglots and even poetry in Perl syntax. But that's not what you will find in production code. That's only people having fun with the language which is different to "getting things done". The expressiveness is a bonus. It makes the Perl programmers love Perl.</span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Just_another_Perl_hacker'>JAPH</a><br />
<a class='textlink' href='http://www.cpan.org/misc/japh'>http://www.cpan.org/misc/japh</a><br />
<a class='textlink' href='https://www.perlmonks.org/index.pl?next=20;node_id=1590'>Perl Poetry</a><br />
<br />
<span>Even I personally have written some poetry in Perl and experimented with a polyglot script:</span><br />
<br />
<a class='textlink' href='./2008-06-26-perl-poetry.html'>My very own Perl Poetry</a><br />
<a class='textlink' href='./2014-03-24-the-fibonacci.pl.c-polyglot.html'>A Perl-Raku-C polyglot generating the Fibonacci sequence</a><br />
<br />
<span>This all doesn't mean that you can't "get things done" with Perl. Quite the opposite is the case. Perl is a very pragmatic programming language and is suitable very well for rapid prototyping and any kind of small to medium-sized scripts and programs. You can write large enterprise scale application in Perl too, but that wasn't the original intend of why Perl was invented (more on that later).</span><br />
<br />
<h2 style='display: inline' id='IsPerlabandoned'>Is Perl abandoned?</h2><br />
<br />
<span>As I pointed out in the previous section, Perl 5 is around for quite some time without any new major version released. This can lead to the impression that development is not progressing and that the project is abandoned. Nothing can be further from the truth. Perl 5.000 was released in 1994 and the latest version (as of this writing) Perl 5.34.1 was released two months ago in 2022. You can check the version history on Wikipedia. You will notice releases being made regularly:</span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Perl_5_version_history'>Perl 5 version history</a><br />
<br />
<span>As you can see, Perl 5 is under active development. I can also recommend to have a look at the following book, it summarizes all new Perl features which showed up after Perl v5.10:</span><br />
<br />
<a class='textlink' href='https://perlschool.com/books/perl-new-features/'>Perl New Features by Joshua McAdams and brian d foy</a><br />
<br />
<span>Actually, Perl is a family of two high-level, general-purpose, interpreted, dynamic programming languages. "Perl" refers to Perl 5, but from 2000 to 2019 it also referred to its redesigned "sister language", Perl 6, before the latter's name was officially changed to Raku in October 2019 as the differences between Perl 5 and Perl 6 were too groundbreaking. Raku would be a different topic (mostly out of scope of this blog article) but I at least wanted it to mention here. In my opinion, Raku is the "most powerful" programming language out there (I recently started learning it and intend to use it for some of my future personal programming projects):</span><br />
<br />
<a class='textlink' href='https://raku.org'>The Raku Programming Language</a><br />
<br />
<span>So it means that Perl and Raku now exist in parallel. They influence each other, but are different programming languages now. So why not just all use Raku instead of Perl? There are still a couple of reasons of why to choose Perl over Raku:</span><br />
<br />
<ul>
<li>Many programmers already know Perl and many scripts are already written in Perl. It's possible to call Perl code from Raku (either inline or as a library) and it is also possible to auto-convert Perl code into Raku code, but that's either a workaround or involves some kind of additional work.</li>
<li>Perl 5 comes with a great backwards compatibility. Perl scripts from 5.000 will generally still work on a recent version of Perl. New features usually have to be enabled via a so-called "use pragmas". For example, in order to enable sub signatures, <span class='inlinecode'>use signatures;</span> has to be specified.</li>
<li>Perl is pre-installed almost everywhere. Fancy running a quick one-off script? In almost all cases, there's no need to install Perl first - it's already there on almost any Linux or *BSD or Unix or other Unix like operating system!</li>
<li>Perl has been ported to "zillions" of platforms. One day I found myself on a VMS box. Perl doesn't come installed by default on VMS, but the admin installed Perl there already. The whole operating system was very strange to me, but I was able to write "shell scripts" in Perl and became productive pretty quickly on VMS without knowing almost anything about VMS :-).</li>
<li>Perl is reliable. It has been proven itself "millions" of times, over and over again. Large enterprises, such as booking.com, heavily rely on Perl. Did you know that the package manager of the OpenBSD operating system is programmed in Perl, too?</li>
<li>Perl is a great language to program in (given that you follow the modern best practices). Don't get confused when Perl is doing some things differently than other programming languages.</li>
</ul><br />
<a class='textlink' href='https://perldoc.perl.org/feature'>Perl feature pragmas</a><br />
<a class='textlink' href='https://www.OpenBSD.org'>The OpenBSD Operating System</a><br />
<a class='textlink' href='https://news.ycombinator.com/item?id=23360338'>Why does OpenBSD still include Perl in its base installation?</a><br />
<br />
<span>The renaming of Perl 6 to Raku has now opened the door for a future Perl 7. As far as I understand, Perl 7 will be Perl 5 but with modern features enabled by default (e.g. pragmas <span class='inlinecode'>use strict;</span>, <span class='inlinecode'>use warnings;</span>, <span class='inlinecode'>use signatures;</span> and so on. Also, the hope is that a Perl 7 with modern standards will attract more beginners. There aren't many Perl jobs out there nowadays. That's mostly due to Perl's bad (bad for no real reasons) reputation.</span><br />
<br />
<span class='quote'>Update 2022-12-10: A reader pointed out, that <span class='inlinecode'>use v5.36;</span> already turns strict, warnings and signatures pragmas automatically on! </span><br />
<br />
<a class='textlink' href='https://www.perl.com/article/announcing-perl-7/'>Announcing Perl 7</a><br />
<a class='textlink' href='http://blogs.perl.org/users/psc/2022/05/what-happened-to-perl-7.html'>What happened to Perl 7? (maybe have to use <span class='inlinecode'>use v7;</span>)</a><br />
<br />
<span class='quote'>Update 2022-12-10: A reader pointed out, that Perl 7 needs to provide a big improvement to earn and keep the attention for a major version bump.</span><br />
<br />
<span class='quote'>Update 2023-01-28: Meanwhile, I was also reading brian d foy's Perl New Feature book. It nicely presents all new features added to Perl since <span class='inlinecode'>v5.10</span>.</span><br />
<br />
<a class='textlink' href='https://www.leanpub.com/perl_new_features'>Perl New Features</a><br />
<br />
<h2 style='display: inline' id='WhyusePerlastherearebetteralternatives'>Why use Perl as there are better alternatives?</h2><br />
<br />
<span>Here, common sense must be applied. I don't believe there is anything like "the perfect" programming language. Everyone has got his preferred (or a set of preferred) programming language to chose from. All programming languages come with their own set of strengths and weaknesses. These are the strengths making Perl shine, and you (technically) don't need to bother to look for "better" alternatives:</span><br />
<br />
<ul>
<li>Perl is better than Shell/AWK/SED scripts. There's a point where shell scripts become fairly complex. The next step-up is to switch to Perl. There are many different versions of shells and AWK and SED interpreters. Do you always know which versions (<span class='inlinecode'>mawk</span>, <span class='inlinecode'>nawk</span>, <span class='inlinecode'>gawk</span>, <span class='inlinecode'>sed</span>, <span class='inlinecode'>gsed</span>, <span class='inlinecode'>grep</span>, <span class='inlinecode'>ggrep</span>...) are currently installed? These commands aren't fully compatible to each other. However, there is only one Perl 5. Simply: Perl is faster, more powerful, more expressive than any shell script can ever be, and it is also extendible through CPAN. Perl can directly talk to databases, which shell scripts can't.</li>
<li>Perl code tends to be compact so that it's much better suitable for "shell scripting" and quick "one-liners" than other languages. In my own experience: Ruby and Python code tends to blow up quickly. It doesn't mean that Ruby and Python are not suitable for this task, but I think Perl does much better.</li>
<li>Perl 5 has proven itself for decades and is a very stable/robust language. It is a battle-tested and mature as something can ever become.</li>
<li>Perl is the reference standard for regular expressions. Even so much that there is a PCRE library (Perl Compatible Regular Expressions) used by many other languages now. Perl fully integrates regular expression syntax into the language, which doesn't feel like an odd add-on like in most other languages.</li>
<li>Perl 5 is the master of text processing (well, maybe after Raku now. But you might not have the latest Raku available everywhere). The chief objective of developing the language was for text processing, and this is where Perl (Practical extraction and report language) really shines.</li>
<li>Perl is a "deep" language. That means Perl got a lot of features and syntactic sugar and magic. Depending on the perspective, this could be interpreted as a downside too. But IMHO mastery of a "deep" language brings big rewards. The code can be very compact, and it is fun to code in it.</li>
<li>Perl is the only language I know which can do "taint checking". Running a script in taint mode makes Perl sanitize all external input and that's a great security feature. Ruby used to have this feature too, but it got removed (as I understand there were some problems with the implementation not completely safe and it was easier just to remove it from the language than to fix it).</li>
</ul><br />
<span>About the first point, using Perl for better "shell" scripts was actually the original intend of why Perl was invented in the first place.</span><br />
<br />
<a class='textlink' href='https://nostarch.com/perloneliners'>Perl one-liners</a><br />
<a class='textlink' href='http://regex.info/book.html'>Mastering Regular Expressions</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Taint_checking'>Taint checking</a><br />
<br />
<span>Here are some reasons why not to chose Perl and look for "better" alternatives:</span><br />
<br />
<ul>
<li>If performance is your main objectives, then Perl might not be the language to use. Perl is a dynamic interpreted language, and it will generally never be as fast as statically typed languages compiled to native binaries (e.g. C/C++/Rust/Haskell) or statically typed languages run in a VM with JIT (e.g. Java) or languages like Golang (statically typed, compiled to a binary but still with a runtime in the binary). Perl might be still faster than the other language listed here in certain circumstances (e.g. faster startup time than Java or faster regular expressions engine), but usually it's not. It's not a problem of Perl, it's a problem of all dynamic scripting languages including Python, Ruby, ....</li>
<li>Don't use Perl (just yet) if you want to code object-oriented. Perl supports OOP, but it feels clunky and odd to use (blessed references to any data types are objects) and doesn't support real encapsulation out of the box. There are many (many) extensions available on CPAN to make OOP better, but that's totally fragmented. The most popular extension, Moose, comes with a huge dependency tree. But wait for Perl 7. It will maybe come with a new object system (an object system inspired by Raku).</li>
<li>It's possible to write large programs in Perl (make difficult things possible), but it might not be the best choice here. This also leads back to the clunky object system Perl has. You could write your projects in a procedural or functional style (Perl perfectly fits here), but OOP seems to be the gold standard for large projects nowadays. Functional programming requires a different mindset, and pure procedural programming lacks abstractions.</li>
<li>Apply common sense. What is the skill set your team has? What's already widely used and supported at work? Which languages comes with the best modules for the things you want to work on? Maybe Python is the answer (better machine learning modules). Maybe Perl is the better choice (better Bioinformatic modules). Perhaps Ruby is already the de-facto standard at work and everyone knows at least a little Ruby (as it happened to be at my workplace) and Ruby is "good enough" for all the tasks already. But that's not a hindrance to throw in a Perl one-liner once in a while :P.</li>
</ul><br />
<a class='textlink' href='https://github.com/Ovid/Cor'>Cor - Bringing modern OOP to the Perl Core</a><br />
<br />
<h2 style='display: inline' id='WhyallthesigilsItlookslikeanexplodingASCIIfactory'>Why all the sigils? It looks like an exploding ASCII factory!!</h2><br />
<br />
<span>The sigils <span class='inlinecode'>$ @ % &</span> (where Perl is famously known for) serve a purpose. They seem confusing at first, but they actually make the code better readable. <span class='inlinecode'>$scalar</span> is a scalar variable (holding a single value), <span class='inlinecode'>@array</span> is an array (holding a list of values), <span class='inlinecode'>%hash</span> holds a list of key-value pairs and <span class='inlinecode'>&sub</span> is for subroutines. A given variable <span class='inlinecode'>$ref</span> can also hold reference to something. <span class='inlinecode'>@$arrayref</span> dereferences a reference to an array, <span class='inlinecode'>%$hashref</span> to a hash, <span class='inlinecode'>$scalarref</span> to a scalar, <span class='inlinecode'>&$subref</span> dereferences a referene to a subroutine, etc. That can be encapsulated as deep as you want. (This paragraph only scratched the surface here of what Perl can do, and there is a lot of syntactic sugar not mentioned here).</span><br />
<br />
<span>In most other programming languages, you won't know instantly what's the "basic type" of a given variable without looking at the variable declaration or the variable name (If named intelligently, e.g. a variable name containing a list of cats is <span class='inlinecode'>cat_list</span>). Even Ruby makes some use of sigils (<span class='inlinecode'>@</span>, <span class='inlinecode'>@@</span> and <span class='inlinecode'>{body}lt;/span>), but that's for a different purpose than in Perl (in Ruby it is about object scope, class scope and global scope). Raku uses all the sigils Perl uses plus an additional bunch of twigils, e.g. <span class='inlinecode'>$.foo</span> for a scalar object variable with public accessors, <span class='inlinecode'>$!foo</span> for a private scalar object variable, <span class='inlinecode'>@.foo</span>, <span class='inlinecode'>@!foo</span>, <span class='inlinecode'>%.foo</span>, <span class='inlinecode'>%!foo</span> and so on. Sigils (and twigils) are very convenient once you get used to them. Don't let them scare you off - they are there to help you!</span><br />
<br />
<a class='textlink' href='https://www.perl.com/article/on-sigils/'>https://www.perl.com/article/on-sigils/</a><br />
<br />
<h2 style='display: inline' id='WheredoIpersonallystilluseperl'>Where do I personally still use perl?</h2><br />
<br />
<ul>
<li>I use Rexify for my OpenBSD server automation. Rexify is a configuration management system developed in Perl with similar features to Ansible but less bloated. It suits my personal needs perfectly.</li>
<li>I have written a couple of smaller to medium-sized Perl scripts which I (mostly) still use regularly. You can find them on my Codeberg page.</li>
<li>My day-to-day workflow heavily relies on "ack-grep". Ack is a tool developed in Perl aimed at programmers and can be used for quick searches on source code at the command line.</li>
<li>I aim to leave my OpenBSD servers as "vanilla" as possible (trying to rely only on the standard/base installation without installing additional software from the packaging system or ports tree). All my scripts are written either Bourne shell or in Perl here. So there is no need to install additional interpreters.</li>
<li>Here and there, I drop a Perl one-liner in order to get stuff done (work and personally). A wise Perl Monk would say: "One one-liner a day keeps the troubles away".</li>
</ul><br />
<span>Btw.: Did you know that the first version of PHP was a set of Perl snippets? Only later, PHP became an independent programming language.</span><br />
<br />
<a class='textlink' href='https://www.perl.org'>https://www.perl.org</a><br />
<br />
<span class='quote'>Update 2022-12-17: The following is another related post. I don't agree to the statement made there, that Python code tends to be shorter than Perl code, though!</span><br />
<br />
<a class='textlink' href='https://stackoverflow.blog/2022/07/06/why-perl-is-still-relevant-in-2022/'>Why Perl is still relevant in 2022</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2008-06-26-perl-poetry.html'>2008-06-26 Perl Poetry</a><br />
<a class='textlink' href='./2011-05-07-perl-daemon-service-framework.html'>2011-05-07 Perl Daemon (Service Framework)</a><br />
<a class='textlink' href='./2022-05-27-perl-is-still-a-great-choice.html'>2022-05-27 Perl is still a great choice (You are currently reading this)</a><br />
<a class='textlink' href='./2023-05-01-unveiling-guprecords:-uptime-records-with-raku.html'>2023-05-01 Unveiling <span class='inlinecode'>guprecords.raku</span>: Global Uptime Records with Raku</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Creative universe</title>
<link href="gemini://foo.zone/gemfeed/2022-04-10-creative-universe.gmi" />
<id>gemini://foo.zone/gemfeed/2022-04-10-creative-universe.gmi</id>
<updated>2022-04-10T10:09:11+01:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I have been participating in an annual work-internal project contest (we call it Pet Project contest) since I moved to London and switched jobs to my current employer. I am very happy to say that I won a 'silver' prize last week here 🎆. Over the last couple of years I have been a finalist in this contest six times and won some kind of prize five times. Some of my projects were also released as open source software. One had a magazine article published, and for another one I wrote an article on my employer's engineering blog. If you have followed all my posts on this blog (the one you are currently reading), then you have probably figured out what these projects were:</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>Creative universe</h1><br />
<br />
<span class='quote'>Published at 2022-04-10T10:09:11+01:00; Updated at 2022-04-18</span><br />
<br />
<pre>
. + . . . . . .
. . . *
. * . . . . . . + .
"You Are Here" . . + . . .
. | . . . . . .
| . . . +. + .
\|/ . . . .
. . V . * . . . . + .
+ . . . +
. . + .+. .
. . . + . . . . .
. . . . . . . . ! /
* . . . + . . - O -
. . . + . . * . . / |
. + . . . .. + .
. . . . * . * . +.. . *
. . . . . . . . + . . +
- the universe
</pre>
<br />
<h2 style='display: inline'>Prelude</h2><br />
<br />
<span>I have been participating in an annual work-internal project contest (we call it Pet Project contest) since I moved to London and switched jobs to my current employer. I am very happy to say that I won a "silver" prize last week here 🎆. Over the last couple of years I have been a finalist in this contest six times and won some kind of prize five times. Some of my projects were also released as open source software. One had a magazine article published, and for another one I wrote an article on my employer's engineering blog. If you have followed all my posts on this blog (the one you are currently reading), then you have probably figured out what these projects were:</span><br />
<br />
<a class='textlink' href='./2021-04-22-dtail-the-distributed-log-tail-program.html'>DTail - The distributed log tail program</a><br />
<a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>Realistic load testing with I/O Riot for linux</a><br />
<br />
<span>Note that my latest silver prize project isn't open source software and because of that there is no public material I can refer to. Maybe the next one again?</span><br />
<br />
<span>I want to point out that I never won the "gold" prize and it's the first time I won "silver", though. I believe, looking at the company's contest history, I am the employee with the most consecutive successful project submissions (my streak broke as I didn't participate last year) and am also the one with the highest successful project count in total. Sorry if this all sounds a bit self-promotional, but I think it is something to be proud of. Consistency beats a one-off success.</span><br />
<br />
<span>I often put endless hours and sometimes sleepless nights into such projects and all of that in my own time. I, an engineer whose native tongue is not English, also have to present such a project in front of the CEO, CTO and CPO, the Chief Scientist, the founders of the company, and, if it is not enough, to all other staff of the company too. I usually also demonstrate a working prototype live on a production grid during the presentation. 😓</span><br />
<br />
<span>So why would I sign up myself for such side projects? Isn't it a lot of stress and extra work? Besides the prize in form of money (you can not count on that, you may win or you may not win something) and recognition, there are also other motivational points:</span><br />
<br />
<ul>
<li>I want to learn new technologies or to deepen my knowledge of a given technology. I want to have a personal benefit from the project, even when I don't win any prize. So when the company is offering a contest, why not use it as a motivational trampoline? It's good to have a hard deadline for a project. And the project will also benefit the company in some way. So it's a win-win.</li>
<li>I love the idea of combining several old things into a new thing. You can call this creativity. At work, we call this sometimes Lego: Building new things from given blocks. But I also love to add something new and unique to the mix, something that didn't exist as a Lego block before and could not be built by using only the already existing blocks.</li>
</ul><br />
<h2 style='display: inline'>How to be creative</h2><br />
<br />
<span>How did I manage to be creative with all these Pet Projects? Unfortunately, there is no step-by-step guide I could point you to. But what I want to do in this blog post is share my personal experience so far.</span><br />
<br />
<h3 style='display: inline'>Know which problem you want to solve</h3><br />
<br />
<span>There must be a problem to be solved or a thing to be improved. It makes no sense to have a project without a goal. A problem might be obvious to you, and you don't even need to think about it. In that case, you are all set, and you can immerse yourself with the problem.</span><br />
<br />
<span>If, however, you don't know what problem you want to solve: Do you really need to be creative? All problems are solved anyway, correct? In that case, just go on with your work. As you immerse yourself with your daily work, you will find a project naturally after a while. I don't believe you should artificially find a project. It should come naturally to you. You should have an interest in the problem domain and a strong desire to find a proper solution for the problem. Artificially created projects come with the catch that you might give up on it rather sooner than later due to lack of motivation and desire.</span><br />
<br />
<h3 style='display: inline'>Immerse / deep dive</h3><br />
<br />
<span>If you want to be creative in a field, you must know a lot about it. The more you know about it, the more dots you can connect. When you are learning a new technology or if you are thinking about a tough problem, do it thoroughly. Don't let anything distract you. Read books, watch lectures, listen to podcasts or audiobooks about the topic, talk to other people working on similar topics. Immerse yourself for multiple hours per day, multiple days per week, multiple weeks and maybe even months. Create your own inner universe.</span><br />
<br />
<span>But once a day is over, shut your thoughts down. Hit the off-switch. Stop thinking about this problem for the remainder of the day. This can be difficult, as you didn't solve the problem- or didn't understand everything of the new technology yet, and you really want to get to the point. But be strict to yourself and stop thinking about it for a while.</span><br />
<br />
<span>You must understand that you are more than just your conscious thoughts. Your brain does a lot of work in the background that you aren't aware of consciously. What happens when you stop consciously thinking about a problem is that your brain continues processing it. You might have experienced the "AHA"-effect, where suddenly you had an idea out of nowhere (e.g. during a walk, in the shower, or in the morning when you woke up)? This is your conscious self downloading a result from the background thread of your brain. You can elevate this effect by immersing with the problem immensely before giving your conscious self a break.</span><br />
<br />
<span>Sometimes, depending on how deeply you were immersed, you may need to let the problem go for a couple of days (e.g. over a weekend) before you can download a new insight.</span><br />
<br />
<h3 style='display: inline'>Always have a notebook with you</h3><br />
<br />
<span>Wherever you go, ensure that you always have something to take notes with you. Once you have an idea from nowhere (or from your unconscious but volatile brain), you really want to write it down to persistent storage. It doesn't matter what kind of note-taking device you use here. It can be a paper journal, or it can be your smartphone. </span><br />
<br />
<span>My advice is to have a separate section where you put your notes of all of your ideas. At home or in the office, I write everything in my paper journal. When I am not at home, I use a digital note-taking app on my phone. Later, I copy the digital notes from it into a project-specific section of my paper journal.</span><br />
<br />
<span>I prefer taking notes on paper, as it gives you more freedom of how to structure it. You can use any colour, and you can also quickly create diagrams without the use of any complex computer program.</span><br />
<br />
<h3 style='display: inline'>When you didn't sleep enough</h3><br />
<br />
<span>I noticed while being sleep-deprived I am (obviously) unable to concentrate so much, and it is difficult to be immersed in a focused way. But on the other hand, I am a lot more creative compared to when I am not sleep-deprived. Then, my brain suddenly presents me with connections I have not thought of before. Here, I usually write any idea I have down on a sheet of paper or in my journal, so I can pick it up later. I then often continue to philosophise about a possible solution. Sometimes to the absurd, and sometimes to something pretty useful.</span><br />
<br />
<span>I am not saying that you should skip sleep. By all means, if you can sleep, then sleep. But there are some days when you don't manage to sleep (e.g. think too much about a project and didn't manage to hit the off switch). This is, where you can take advantage of your current state of mind. Disclaimer: Skipping sleep damages your health. So, please don't try this out on purpose. But in case you had a bad night, remember this trick.</span><br />
<br />
<h3 style='display: inline'>Have regular breaks and relax</h3><br />
<br />
<span>Have regular breaks. Don't skip your lunch break. Best, have a walk during lunchtime. And after work, do some kind of workout or visit a sports class. Do something completely unrelated to work before going to sleep (e.g. visit a parallel universe and read a Science Fiction novel). In short: Totally hit the off-switch after your work for the day is finished. You will be much more energised and motivated the next time you open your work laptop.</span><br />
<br />
<a class='textlink' href='../other-resources.html'>I personally love to read Science Fiction novels</a><br />
<br />
<span>I skip breakfast and lunch during the week. This means that on average, I intermittent fast on average 18-20 hours daily. It may sound odd to most people (who don't intermittent fast), but in a fasted state, I can be even more focused, thus helping me immerse myself in something even more. Not having breakfast and lunch also gives me back some time for other things (e.g. a nice walk, where I listen to podcasts or audiobooks or practise using my camera (street photography)). I relax my routine during the week ends, where I may enjoy a meal at any given time of the day.</span><br />
<br />
<span>It also helps a lot eat healthy. Healthy food makes your brain work more efficiently. But I won't go into more details here, as nothing is as contradictory as the health and food industry. Conduct your own research. Your opinion may be different from mine anyway, and everyone's body reacts to certain foods differently. What for one person works may not work for another person. But be aware that you will find a lot of wrong and also conflicting information on the internet. So always use multiple resources for your research.</span><br />
<br />
<h3 style='display: inline'>Upside-down approach</h3><br />
<br />
<span>It's easy to fall into the habit of "boxed" thinking, but creativity is exactly the opposite. Once in a while, make yourself think "Is A really required to do B?". Many assumptions are believed to be true. But are they really? A concrete example: "At work we only use the programming language L and framework F" and therefore, it is the standard we must use.</span><br />
<br />
<span>Another way to think about it is "Is there an alternative way to accomplish the desired result? What if there were no programming language L and framework F? What would I do instead?". Maybe you would use programming language X to implement your own domain-specific language, which does what framework F would have done but in exactly the way you want to + much more flexible than F! And maybe language X would be much better suitable than L to implement a DSL anyway. Conclusion: It never hurts to verify your assumptions.</span><br />
<br />
<span>Often, you will also find solutions to problems you never intended to solve and find new problems you never imagined to actually exist. That might not be a bad thing, but it might sidetrack you on your path to finding a solution for a particular problem. So be careful not to get sidetracked too much. In this case, just save a note for later reference (maybe your next Pet Project?) somewhere and go on with your actual problem.</span><br />
<br />
<span>Don't be afraid to think about weird and unconventional solutions. Sometimes, the most unconventional solution is the best solution to a problem. Also, try to keep to the basics. The best solutions are KISS.</span><br />
<br />
<a class='textlink' href='./2021-09-12-keep-it-simple-and-stupid.html'>Keep it simple and stupid</a><br />
<br />
<span>A small additional trick: you can train yourself to generate new and unconventional ideas. Just write down 20 random ideas every day. It doesn't matter what the ideas are about and whether they are useful or not. The purpose of this exercise is to make your brain think about something new and unconventional. These can be absurd ideas such as "Jump out of the window naked in the morning in order to wake up faster". Of course, you would never do that, but at least you had an idea and made your brain generate something.</span><br />
<br />
<h3 style='display: inline'>Don't be busy all the time</h3><br />
<br />
<span>Especially as a DevOps Engineer, you could be busy all the time with small, but frequent, ad hoc tasks. Don't lose yourself here. Yes, you should pay attention to your job and those tasks, but you should also make some room for creativity. Don't schedule meeting after ad hoc work after meeting after Jira ticket work after another Jira ticket. There should also be some "free" space in your calendar.</span><br />
<br />
<span>Use the "free" time to play around with your tech stack. Try out new options, explore the system metrics, explore new tools, etc. Dividends will pay off with new ideas, which you would have never come up with if you were "just busy" like a machine.</span><br />
<br />
<span>Sometimes, I pick a Unix manual page of a random command and start reading it. I have a bash helper function which will pick one for me:</span><br />
<br />
<pre>
❯ where learn
learn () {
man $(ls /bin /sbin /usr/bin /usr/sbin 2>/dev/null | shuf -n 1) |
sed -n "/^NAME/ { n;p;q }"
}
❯ learn
perltidy - a perl script indenter and reformatter
❯ learn
timedatectl - Control the system time and date
</pre>
<br />
<h2 style='display: inline'>Conclusion</h2><br />
<br />
<span>This all summarises advice I have, really. I hope this was interesting and helpful for you.</span><br />
<br />
<span>I have one more small tip: I never published a blog post the same day I wrote it. After finishing writing it, I always wait for a couple of days. In all cases so far, I had an additional idea to add or to fine tune the blog post.</span><br />
<br />
<span>Another article I found interesting and relevant is</span><br />
<br />
<a class='textlink' href='https://thesephist.com/posts/paradise/'>Creative Paradise by The Sephist</a><br />
<br />
<span>Relevant books I can recommend are:</span><br />
<br />
<ul>
<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li>
<li>Deep Work; Cal Newport; Piatkus</li>
<li>So Good They Can't Ignore You; Cal Newport; Business Plus</li>
<li>The Off Switch; Mark Cropley; Virgin Books</li>
<li>Ultralearning; Scott Young; Thorsons</li>
</ul><br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>The release of DTail 4.0.0</title>
<link href="gemini://foo.zone/gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi" />
<id>gemini://foo.zone/gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi</id>
<updated>2022-03-06T18:11:39+00:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I have recently released DTail 4.0.0 and this blog post goes through all the new goodies. If you want to jump directly to DTail, do it here (there are nice animated gifs which demonstrates the usage pretty well):</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='ThereleaseofDTail400'>The release of DTail 4.0.0</h1><br />
<br />
<span class='quote'>Published at 2022-03-06T18:11:39+00:00</span><br />
<br />
<pre>
,_---~~~~~----._
_,,_,*^____ _____``*g*\"*,
____ _____ _ _ / __/ /' ^. / \ ^@q f
| _ \_ _|_ _(_) | @f | @)) | | @)) l 0 _/
| | | || |/ _` | | | \`/ \~____ / __ \_____/ \
| |_| || | (_| | | | | _l__l_ I
|____/ |_|\__,_|_|_| } [______] I
] | | | |
] ~ ~ |
| |
| |
</pre>
<br />
<span>I have recently released DTail 4.0.0 and this blog post goes through all the new goodies. If you want to jump directly to DTail, do it here (there are nice animated gifs which demonstrates the usage pretty well):</span><br />
<br />
<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br />
<br />
<h2 style='display: inline' id='Sowhatsnewin400'>So, what's new in 4.0.0?</h2><br />
<br />
<h3 style='display: inline' id='Rewrittenlogging'>Rewritten logging</h3><br />
<br />
<span>For DTail 4, logging has been completely rewritten. The new package name is "internal/io/dlog". I rewrote the logging because DTail is a special case here: There are logs processed by DTail, there are logs produced by the DTail server itself, there are logs produced by a DTail client itself, there are logs only logged by a DTail client, there are logs only logged by the DTail server, and there are logs logged by both, server and client. There are also different logging levels and outputs involved.</span><br />
<br />
<span>As you can imagine, it becomes fairly complex. There is no ready Go off-shelf logging library which suits my needs and the logging code in DTail 3 was just one big source code file with global variables and it wasn't sustainable to maintain anymore. So why not rewrite it for profit and fun? </span><br />
<br />
<span>There's a are new log level structure now (The log level now can be specified with the "-logLevel" command line flag):</span><br />
<br />
<pre>
// Available log levels.
const (
None level = iota
Fatal level = iota
Error level = iota
Warn level = iota
Info level = iota
Default level = iota
Verbose level = iota
Debug level = iota
Devel level = iota
Trace level = iota
All level = iota
)
</pre>
<br />
<span>DTail also supports multiple log outputs (e.g. to file or to stdout). More are now easily pluggable with the new logging package. The output can also be "enriched" (default) or "plain" (read more about that further below).</span><br />
<br />
<h3 style='display: inline' id='Configurableterminalcolorcodes'>Configurable terminal color codes</h3><br />
<br />
<span>A complaint I received from the users of DTail 3 were the terminal colors used for the output. Under some circumstances (terminal configuration) it made the output difficult to read so that users defaulted to "--noColor" (disabling colored output completely). I toke it by heart and also rewrote the color handling. It's now possible to configure the foreground and background colors and an attribute (e.g. dim, bold, ...).</span><br />
<br />
<span>The example "dtail.json" configuration file represents the default (now, more reasonable default) color codes used, and it is free to the user to customize them:</span><br />
<br />
<pre>
{
"Client": {
"TermColorsEnable": true,
"TermColors": {
"Remote": {
"DelimiterAttr": "Dim",
"DelimiterBg": "Blue",
"DelimiterFg": "Cyan",
"RemoteAttr": "Dim",
"RemoteBg": "Blue",
"RemoteFg": "White",
"CountAttr": "Dim",
"CountBg": "Blue",
"CountFg": "White",
"HostnameAttr": "Bold",
"HostnameBg": "Blue",
"HostnameFg": "White",
"IDAttr": "Dim",
"IDBg": "Blue",
"IDFg": "White",
"StatsOkAttr": "None",
"StatsOkBg": "Green",
"StatsOkFg": "Black",
"StatsWarnAttr": "None",
"StatsWarnBg": "Red",
"StatsWarnFg": "White",
"TextAttr": "None",
"TextBg": "Black",
"TextFg": "White"
},
"Client": {
"DelimiterAttr": "Dim",
"DelimiterBg": "Yellow",
"DelimiterFg": "Black",
"ClientAttr": "Dim",
"ClientBg": "Yellow",
"ClientFg": "Black",
"HostnameAttr": "Dim",
"HostnameBg": "Yellow",
"HostnameFg": "Black",
"TextAttr": "None",
"TextBg": "Black",
"TextFg": "White"
},
"Server": {
"DelimiterAttr": "AttrDim",
"DelimiterBg": "BgCyan",
"DelimiterFg": "FgBlack",
"ServerAttr": "AttrDim",
"ServerBg": "BgCyan",
"ServerFg": "FgBlack",
"HostnameAttr": "AttrBold",
"HostnameBg": "BgCyan",
"HostnameFg": "FgBlack",
"TextAttr": "AttrNone",
"TextBg": "BgBlack",
"TextFg": "FgWhite"
},
"Common": {
"SeverityErrorAttr": "AttrBold",
"SeverityErrorBg": "BgRed",
"SeverityErrorFg": "FgWhite",
"SeverityFatalAttr": "AttrBold",
"SeverityFatalBg": "BgMagenta",
"SeverityFatalFg": "FgWhite",
"SeverityWarnAttr": "AttrBold",
"SeverityWarnBg": "BgBlack",
"SeverityWarnFg": "FgWhite"
},
"MaprTable": {
"DataAttr": "AttrNone",
"DataBg": "BgBlue",
"DataFg": "FgWhite",
"DelimiterAttr": "AttrDim",
"DelimiterBg": "BgBlue",
"DelimiterFg": "FgWhite",
"HeaderAttr": "AttrBold",
"HeaderBg": "BgBlue",
"HeaderFg": "FgWhite",
"HeaderDelimiterAttr": "AttrDim",
"HeaderDelimiterBg": "BgBlue",
"HeaderDelimiterFg": "FgWhite",
"HeaderSortKeyAttr": "AttrUnderline",
"HeaderGroupKeyAttr": "AttrReverse",
"RawQueryAttr": "AttrDim",
"RawQueryBg": "BgBlack",
"RawQueryFg": "FgCyan"
}
}
},
...
}
</pre>
<br />
<span>You notice the different sections - these are different contexts:</span><br />
<br />
<ul>
<li>Remote: Color configuration for all log lines sent remotely from the server to the client. </li>
<li>Client: Color configuration for all lines produced by a DTail client by itself (e.g. status information).</li>
<li>Server: Color configuration for all lines produced by the DTail server by itself and sent to the client (e.g. server warnings or errors)</li>
<li>MaprTable: Color configuration for the map-reduce table output.</li>
<li>Common: Common color configuration used in various places (e.g. when it's not clear what's the current context of a line).</li>
</ul><br />
<span>When you do so, make sure that you check your "dtail.json" against the JSON schema file. This is to ensure that you don't configure an invalid color accidentally (requires "jsonschema" to be installed on your computer). Furthermore, the schema file is also a good reference for all possible colors available:</span><br />
<br />
<pre>
jsonschema -i dtail.json schemas/dtail.schema.json
</pre>
<br />
<h3 style='display: inline' id='Serverlessmode'>Serverless mode</h3><br />
<br />
<span>All DTail commands can now operate on log files (and other text files) directly without any DTail server running. So there isn't a need anymore to install a DTail server when you are on the target server already anyway, like the following example shows:</span><br />
<br />
<pre>
% dtail --files /var/log/foo.log
</pre>
<br />
<span>or</span><br />
<br />
<pre>
% dmap --files /var/log/foo.log --query 'from TABLE select .... outfile result.csv'
</pre>
<br />
<span>The way it works in Go code is that a connection to a server is managed through an interface and in serverless mode DTail calls through that interface to the server code directly without any TCP/IP and SSH connection made in the background. This means, that the binaries are a bit larger (also ship with the code which normally would be executed by the server) but the increase of binary size is not much.</span><br />
<br />
<h3 style='display: inline' id='Shorthandflags'>Shorthand flags</h3><br />
<br />
<span>The "--files" from the previous example is now redundant. As a shorthand, It is now possible to do the following instead:</span><br />
<br />
<pre>
% dtail /var/log/foo.log
</pre>
<br />
<span>Of course, this also works with all other DTail client commands (dgrep, dcat, ... etc).</span><br />
<br />
<h3 style='display: inline' id='Spartanakaplainmode'>Spartan (aka plain) mode</h3><br />
<br />
<span>There's a plain mode, which makes DTail only print out the "plain" text of the files operated on (without any DTail specific enriched output). E.g.:</span><br />
<br />
<pre>
% dcat --plain /etc/passwd > /etc/test
% diff /etc/test /etc/passwd # Same content, no diff
</pre>
<br />
<span>This might be useful if you wanted to post-process the output. </span><br />
<br />
<h3 style='display: inline' id='Standardinputpipe'>Standard input pipe</h3><br />
<br />
<span>In serverless mode, you might want to process your data in a pipeline. You can do that now too through an input pipe:</span><br />
<br />
<pre>
% dgrep --plain --regex 'somethingspecial' /var/log/foo.log |
dmap --query 'from TABLE select .... outfile result.csv'
</pre>
<br />
<span>Or, use any other "standard" tool:</span><br />
<br />
<pre>
% awk '.....' < /some/file | dtail ....
</pre>
<br />
<h3 style='display: inline' id='Newcommanddtailhealth'>New command dtailhealth</h3><br />
<br />
<span>Prior to DTail 4, there was a flag for the "dtail" command to check the health of a remote DTail server (for use with monitoring system such as Nagios). That has been moved out to a separate binary to reduce complexity of the "dtail" command. The following checks whether DTail is operational on the current machine (you could also check a remote instance of DTail server, just adjust the server address).</span><br />
<br />
<pre>
% cat check_dtail.sh
#!/bin/sh
exec /usr/local/bin/dtailhealth --server localhost:2222
</pre>
<br />
<h3 style='display: inline' id='Improveddocumentation'>Improved documentation</h3><br />
<br />
<span>Some features, such as custom log formats and the map-reduce query language, are now documented. Also, the examples have been updated to reflect the new features added. This also includes the new animated example Gifs (plus documentation how they were created).</span><br />
<br />
<span>I must admit that not all features are documented yet:</span><br />
<br />
<ul>
<li>Server side scheduled map-reduce queries</li>
<li>Server side continuous map-reduce queries</li>
<li>Some more docs about terminal color customization</li>
<li>Some more docs about log levels</li>
</ul><br />
<span>That will be added in one of the future releases. </span><br />
<br />
<h3 style='display: inline' id='Integrationtestingsuite'>Integration testing suite</h3><br />
<br />
<span>DTail comes already with some unit tests, but what's new is a full integration testing suite which covers all common use cases of all the commands (dtail, dcat, dgrep, dmap) with a server backend and also in serverless mode.</span><br />
<br />
<span>How are the tests implemented? All integration tests are simply unit tests in the "./integrationtests" folder. They must be explicitly activated with:</span><br />
<br />
<pre>
% export DTAIL_INTEGRATION_TEST_RUN_MODE=yes
</pre>
<br />
<span>Once done, first compile all commands, and then run the integration tests:</span><br />
<br />
<pre>
% make
.
.
.
% go clean -testcache
% go test -race -v ./integrationtests
</pre>
<br />
<h3 style='display: inline' id='Improvedcode'>Improved code</h3><br />
<br />
<span>Not that the code quality of DTail has been bad (I have been using Go vet and Go lint for previous releases and will keep using these), but this time I had new tools (such as SonarQube and BlackDuck) in my arsenal to:</span><br />
<br />
<ul>
<li>Reduce the complexity of a couple of functions (splitting code up into several smaller functions)</li>
<li>Avoid repeating code (this version of DTail doesn't use Go generics yet, though).</li>
</ul><br />
<span>Other than that, a lot of other code has been refactored as I saw fit.</span><br />
<br />
<h3 style='display: inline' id='Useofmemorypools'>Use of memory pools</h3><br />
<br />
<span>DTail makes excessive use of string builder and byte buffer objects. For performance reasons, those are now re-used from memory pools.</span><br />
<br />
<h2 style='display: inline' id='Whatsnext'>What's next</h2><br />
<br />
<span>DTail 5 won't be released any time soon I guess, but some 4.x.y releases will follow this year fore sure. I can think of:</span><br />
<br />
<ul>
<li>New (but backwards compatible) features which don't require a new major version bump (some features have been requested at work internally).</li>
<li>Even more improved documentation.</li>
<li>Dependency updates.</li>
</ul><br />
<span>I use usually DTail at work, but I have recently installed it on my personal OpenBSD machines too. I might write a small tutorial here (and I might also add the rc scripts as examples to one of the next DTail releases).</span><br />
<br />
<span>I am a bit busy at the moment with two other pet projects of mine (one internal work-project, and one personal one, the latter you will read about in the next couple of months). If you have ideas (or even a patch), then please don't hesitate to contact me (either via E-Mail or a request at GitHub).</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-04-22-dtail-the-distributed-log-tail-program.html'>2021-04-22 DTail - The distributed log tail program</a><br />
<a class='textlink' href='./2022-03-06-the-release-of-dtail-4.0.0.html'>2022-03-06 The release of DTail 4.0.0 (You are currently reading this)</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2023-09-25-dtail-usage-examples.html'>2023-09-25 DTail usage examples</a><br />
<br />
<span>Thanks!</span><br />
<br />
<span>Paul</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Computer operating systems I use(d)</title>
<link href="gemini://foo.zone/gemfeed/2022-02-04-computer-operating-systems-i-use.gmi" />
<id>gemini://foo.zone/gemfeed/2022-02-04-computer-operating-systems-i-use.gmi</id>
<updated>2022-02-04T09:58:22+00:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is a list of Operating Systems I currently use. This list is in no particular order and also will be updated over time. The very first operating system I used was MS-DOS (mainly for games) and the very first Unix like operating system I used was SuSE Linux 5.3. My first smartphone OS was Symbian on a clunky Sony Ericsson device.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>Computer operating systems I use(d)</h1><br />
<br />
<span class='quote'>Published at 2022-02-04T09:58:22+00:00; Updated at 2022-02-18</span><br />
<br />
<pre>
/( )`
\ \___ / |
/- _ `-/ '
(/\/ \ \ /\
/ / | ` \
O O ) / |
`-^--'`< '
(_.) _ ) /
`.___/` /
`-----' /
<----. __ / __ \
<----|====O)))==) \) /====
<----' `--' `.__,' \
| |
\ /
______( (_ / \______
(FL) ,' ,-----' | \
`--{__________) \/ "Berkeley Unix Daemon"
</pre>
<br />
<span>This is a list of Operating Systems I currently use. This list is in no particular order and also will be updated over time. The very first operating system I used was MS-DOS (mainly for games) and the very first Unix like operating system I used was SuSE Linux 5.3. My first smartphone OS was Symbian on a clunky Sony Ericsson device.</span><br />
<br />
<h2 style='display: inline'>Fedora Linux</h2><br />
<br />
<span>Fedora Linux is the operating system I use on my primary (personal) laptop. It's a ThinkPad X1 Carbon Gen. 9. Lenovo which comes along with official Lenovo Linux support. I already noticed hardware firmware updates being installed directly through Fedora from Lenovo. Fedora is a real powerhouse, cutting-edge and reasonably stable at the same time. It's baked by Red Hat.</span><br />
<br />
<span>I also use Fedora on my Microsoft Surface Go 2 convertible tablet. Fedora works quite OK (and much better than Windows) on this device. It's also the perfect travel companion.</span><br />
<br />
<span>I use the GNOME Desktop on my Fedora boxes. I have memorized and customized a bunch of keyboard shortcuts. But the fact that I mostly work in the terminal (with tmux) makes the Desktop environment I use only secondary.</span><br />
<br />
<h2 style='display: inline'>EndeavourOS</h2><br />
<br />
<span>I installed EndeavourOS on my (older) ThinkPad X240 to try out an Arch based Linux distribution. I also could have installed plain Arch, but I don't see the point when there is EndeavourOS. EndeavourOS is as close as you can get to the plain Arch experience but with an easy installer. I am not saying that it's difficult to install plain Arch but it's, unless you are new to Linux and want to learn about the installation procedure, just waste of time in my humble opinion. Give Linux From Scratch a shot instead if you really want to learn about Linux.</span><br />
<br />
<a class='textlink' href='https://www.linuxfromscratch.org/'>https://www.linuxfromscratch.org/</a><br />
<br />
<span>On EndeavourOS, I use the Xfce desktop environment which feels very snappy and fast on the X240 (which I purchased back in 2014). Usually, I have my X240 standing right next to my work laptop and use it for playing music (mainly online radio streams), for personal note taking and occasional emailing and instant messaging.</span><br />
<br />
<span>As this is a rolling Linux distribution there are a lot of software updates coming through every day. Sometimes, it only takes a minute until the next version of a package is available. Honestly, I find that a bit annoying to constantly catch up with all the updates. As for now I will live with it and/or automate it a bit more. It'll be OK if it breaks occasionally, as this is not my primary laptop anyway. </span><br />
<br />
<span>Arch Linux and EndeavourOS are community distributions. This means, that there is no big corporation in the backyard lurking around. They won't give you the firmware updates for cutting edge hardware out of the box, though, but they are still a very good choice for hobbyist and also for older hardware where future firmware updates are less likely to happen.</span><br />
<br />
<span>I am very happy with the package availability through the official repository and AUR.</span><br />
<br />
<a class='textlink' href='https://endeavouros.com/'>https://endeavouros.com/</a><br />
<br />
<h2 style='display: inline'>FreeBSD</h2><br />
<br />
<span>I have run FreeBSD in many occasions. Right after SuSE Linux, FreeBSD (around 4.x) was the second open source system I used in my life on regular basis. I didn't even go to university yet then I started using it :-). Also, a former employer of mine even allowed me to install FreeBSD on my main workstation (which I actually did and used it for a couple of years). </span><br />
<br />
<span>I remember it used to be a pain bootstrapping Java for FreeBSD due to the lack of pre-compiled binary packages. You had first to enable the Linux compatibility layer, then install Linux Java, and then compile FreeBSD Java with the bootstrapped Linux Java (yes, Java is mainly programmed in C++, but for some reason compiling Java for FreeBSD also required an installation of Java). Nowadays, there are ready OpenJDK binary packages you could install. So things have improved a lot since.</span><br />
<br />
<span>FreeBSD always had a place somewhere in my life:</span><br />
<br />
<ul>
<li>On a Desktop PC (personal and work)</li>
<li>On a Laptop</li>
<li>On a webserver, FTP server, DNS server, mail server</li>
<li>On a server offering FreeBSD jails to customers for rent</li>
<li>As an experiment running Debian GNU/kFreeBSD inside of jails</li>
</ul><br />
<span>Debian GNU/kFreeBSD is now dead (same is my experiment)...</span><br />
<br />
<a class='textlink' href='https://www.debian.org/ports/kfreebsd-gnu/'>https://www.debian.org/ports/kfreebsd-gnu/</a><br />
<br />
<span>...but I still have saved and old uname output :-):</span><br />
<br />
<pre>
[root@saturn /usr/jail/serv14/etc] # jexec 21 bash
root@rhea:/ # uname -a
GNU/kFreeBSD rhea.buetow.org 8.0-RELEASE-p5 FreeBSD 8.0-RELEASE-p5 #2: Sat Nov 27 13:10:09 CET
2010 root@saturn.buetow.org:/usr/obj/usr/srcs/freebsd.src8/src/sys/SERV10 x86 64 amd64 Intel(R)
Core(TM) i7 CPU 920 @ 2.67GHz GNU/kFreeBSD
</pre>
<br />
<span>Currently, I use FreeBSD on my personal NAS server. The server is a regular PC with a bunch of hard drives and a ZFS RAIDZ (with 4x2TB drives) + a couple of external backup drives.</span><br />
<br />
<a class='textlink' href='https://www.FreeBSD.org'>https://www.FreeBSD.org</a><br />
<br />
<h2 style='display: inline'>CentOS 7</h2><br />
<br />
<span>While CentOS 8 is already out of support, I still use CentOS 7 (which will receive security updates until 2024). CentOS 7 runs in a cloud VM and is the home to my personal NextCloud and Wallabag installations. You probably know already NextCloud. About Wallabag: It is a great free and open source alternative to Pocket (for reading articles from the web offline later). Yes, you can pay for a Wallabag subscription, but you can also host it for free on your own server.</span><br />
<br />
<a class='textlink' href='https://nextcloud.com'>NextCloud</a><br />
<a class='textlink' href='https://www.wallabag.it/en'>Wallabag</a><br />
<br />
<span>The reason I use Linux and not *BSD at the moment for these services is Docker. With Docker, it's so easy-peasy to get these up and running. I will have to switch to another OS before CentOS 7 runs out of support, though. It might be CentOS Stream, Rocky Linux, or, more likely, I will use FreeBSD. On FreeBSD there isn't Docker, but what can be done is to create a self-contained Jail for each of the web-apps. </span><br />
<br />
<span>I have been using FreeBSD Jails for LAMP stacks before I started using CentOS. The reason why I switched to CentOS (it was still CentOS 6 at that time) in the first place was, that I wanted to try out something new.</span><br />
<br />
<a class='textlink' href='https://www.centos.org'>https://www.centos.org</a><br />
<br />
<h2 style='display: inline'>OpenBSD</h2><br />
<br />
<span>I use two small OpenBSD "cloud" boxes for my "public facing internet front-ends". The services I run here are:</span><br />
<br />
<ul>
<li>HTTP server (serving this site via https://foo.zone)</li>
<li>Gemini server (serving this site via gemini://foo.zone)</li>
<li>MTA server (for receiving E-Mails to my hosts)</li>
<li>Authorative DNS server (for all of my "domains")</li>
<li>Some personal/private git repositories (accessible only via SSH)</li>
</ul><br />
<span>OpenBSD is a complete operating system. I love it due to it's "simplicity" and "correctness" and the good documentation (I love the manual pages in particular). OpenBSD is also known for its innovations in security. I must admin, though, that most Unix like operating system would be secure enough for my personal needs and that I don't really need to use OpenBSD here. But nevertheless, I think it's the ideal operating system for what I am using it for.</span><br />
<br />
<span>The only softwares which were not part of the base system and I had to install additionally were the Gemini server (vger) and Git, which both were available as pre-compiled OpenBSD binary packages. So, besides of these two packages, it is indeed a pretty complete operating system for my use case.</span><br />
<br />
<a class='textlink' href='https://www.openbsd.org'>https://www.openbsd.org</a><br />
<br />
<h2 style='display: inline'>macOS (proprietary)</h2><br />
<br />
<span>I have to use a MacBook Pro with macOS for work. What else can I say but that this would have never been my personal choice. At least macOS is a UNIX under the hood and comes with a decent terminal and there are plenty of terminal apps available via Brew. Some of the inner workings of macOS were actually forked from the FreeBSD project. </span><br />
<br />
<a class='textlink' href='https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/BSD/BSD.html'>developer.apple.com: BSD in macOS/Darwin</a><br />
<br />
<span>I find the macOS UI rather confusing.</span><br />
<br />
<h2 style='display: inline'>LineageOS (mobile)</h2><br />
<br />
<span>At some point I got fed up with big tech, like Google and Samsung (or Apple, but personally I don't use Apple), spying on me. So I purchased a Google phone (a midrange Pixel phone) and installed LineageOS, a free and open source distribution of Android, on it. I don't have anything from Google installed on it (not even the play store, I install my apps from F-Droid). It's my daily driver since mid 2021 now. </span><br />
<br />
<span>So far the experience is not great but good. The main culprits are not having Google Maps, Google Gboard and the camera app. The latter lacks some features on LineageOS (e.g. No wide angle lens support). Also, I can't use my banking apps anymore. Sometimes apps crash for no apparent reason(s) but I get around it so far. I shouldn't spend so much time on my smartphone anyway! And the whole point of switching to LineageOS was to get away of big tech and therefore I should not complain :-). What I do like is that 95% the things I used to do on a proprietary mobile phone also can be done with LineageOS.</span><br />
<br />
<a class='textlink' href='./2021-08-01-on-being-pedantic-about-open-source.html'>Read also "The Midle Way" section of this blog post regarding smartphones.</a><br />
<br />
<span>There's also the excellent Termux app in the F-Droid store, which transforms the phone into a small Linux handheld device. I am able to run all of my Linux/Unix terminal apps with it.</span><br />
<br />
<a class='textlink' href='https://lineageos.org/'>https://lineageos.org/</a><br />
<a class='textlink' href='https://termux.com/'>https://termux.com/</a><br />
<br />
<h2 style='display: inline'>Samsung's Stock Android (mobile proprietary)</h2><br />
<br />
<span>Unfortunatley, I still have to keep my proprietary Android phone around. Sometimes, I really need to use some proprietary apps which are only available form the Google play store and also require the Google services installed on the phone. I don't carry this phone around all the time and I only use it intentionally for very specific use cases. I think this is the best compromise I can make.</span><br />
<br />
<h2 style='display: inline'>iOS (mobile proprietary)</h2><br />
<br />
<span>I have to use an iPhone for work. I like the hardware but I hate the OS (you can also call it spyOS), but it's the necessarries evil, unfortunately. Apple is even worse than Google here (despite claiming for themselves to produce the most secure phone(s)). I don't have it with me all the time or switched off when I don't need it. I also find iOS quite unintuitive to use.</span><br />
<br />
<span>Being on-call for work means to to be reachable 24/7. This implies that the phone is carried around all the time (in an switched-on state). 1984 is now.</span><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Nineteen_Eighty-Four'>https://en.wikipedia.org/wiki/Nineteen_Eighty-Four</a><br />
<br />
<h2 style='display: inline'>Other OSes</h2><br />
<br />
<h3 style='display: inline'>InfinyTime (smartwatch)</h3><br />
<br />
<span>I use it on my PineTime smartwatch. Other than checking the time and my step count, I really don't do anything else fancy with it (yet). </span><br />
<br />
<a class='textlink' href='https://www.pine64.org/pinetime/'>https://www.pine64.org/pinetime/</a><br />
<a class='textlink' href='https://infinitime.io/'>https://infinitime.io/</a><br />
<br />
<h3 style='display: inline'>motionEyeOS</h3><br />
<br />
<span>I usually install an army of RaspberryPi 3's in my house before I travel for a prolonged amount of time. All Pi's are equipped with an camera and have motionEyeOS (Linux based video surveillance system) installed. There's a neat Android app in the F-Droid store which let's me keep an eye on everything. I make the Pi's accessible from the internet via reverse SSH tunnels through one of my frontend servers.</span><br />
<br />
<a class='textlink' href='https://github.com/ccrisan/motioneyeos'>https://github.com/ccrisan/motioneyeos</a><br />
<br />
<h3 style='display: inline'>Kobo OS (proprietary)</h3><br />
<br />
<span>I use a Kobo Forma as my e-reader device. I have started to switch off the Wifi and to only sideload DRM free ePubs on it. Even offline, it's a fully capable reader device. I wouldn't like the Kobo to call home to Rakuten. I would love to replace it one day with an open source e-reader alternative like the PineNote. There are also some interesting attempts installing postmarketOS Linux on Kobo devices. The latter boots already, but is far from being usable as a normal e-reader.</span><br />
<br />
<a class='textlink' href='https://www.pine64.org/pinenote/'>The PineNote</a><br />
<a class='textlink' href='https://liliputing.com/2021/07/kobo-clara-hd-becomes-an-e-ink-linux-tablet-with-the-help-of-postmarketos.html'>Kobo Clara HD becomes an e-ink Linux tablet</a><br />
<br />
<span>But as a fall-back, someone could still use the good old dead tree format!</span><br />
<br />
<h3 style='display: inline'>Android TV (proprietary)</h3><br />
<br />
<span>An Android TV box is used for watching movies and series on Netflix and Amazon Prime video (yes, I am human too and rely once in a while on big tech streaming services). The Android TV box is currently in the process of being replaced by OSMC, though. Most services seem to work fine with OSMC, but didn't get around tinkering with Netflix and Amazon there yet.</span><br />
<br />
<a class='textlink' href='https://osmc.tv/'>https://osmc.tv/</a><br />
<br />
<h2 style='display: inline'>Other OSes..</h2><br />
<br />
<span>This section is just for the sake of having a complete list of all OSes I used for some significant amount of time. I might not use all of them any more...</span><br />
<br />
<h3 style='display: inline'>NetBSD</h3><br />
<br />
<span>I have been using NetBSD on an old Sun Sparcstation 10 as a student. I also have run NetBSD on a very old ThinkPad with 96MB!!! of RAM (even with X/evilWM). I also installed (but never really used) NetBSD on an HP Jornada 680. But that's all more than 10 years ago. I haven't looked at NetBSD for long time. I want to revive it on an "old" ThinkPad T450 of mine which I currently don't use.</span><br />
<br />
<a class='textlink' href='https://netbsd.org'>https://netbsd.org</a><br />
<br />
<h3 style='display: inline'>Other OSes in use...</h3><br />
<br />
<a class='textlink' href='https://sailfish.org'>SailfishOS - Nice mobile OS, but unfortunately includes proprietary components</a><br />
<a class='textlink' href='https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux'>Red Hat Enterprise Linux - Only for some work stuff</a><br />
<br />
<h3 style='display: inline'>Other OSes not used any more...</h3><br />
<br />
<a class='textlink' href='https://en.opensuse.org/Archive:S.u.S.E._Linux_5.3'>SuSE Linux 5.3 - The first Linux OS I used</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/IRIX'>SGI's IRIX - On a SGI Onyx 3200</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/MeeGo'>MeeGo - On a Nokia N9</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Microsoft_Windows'>Microsoft Windows</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/MS-DOS'>Microsoft DOS - With and without Windows 3.x</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Symbian'>Symbian - The first smartphone OS I used </a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Wear_OS'>WearOS - On a Google smartwatch</a><br />
<a class='textlink' href='https://www.debian.org'>Debian GNU/Linux - Rock solid, but atm. I prefer Fedora/EndeavourOS</a><br />
<a class='textlink' href='https://www.ubuntu.com'>Ubuntu Linux (based on Debian)</a><br />
<a class='textlink' href='https://www.linuxfromscratch.org/'>Linux from scratch - The best way to learn Linux</a><br />
<a class='textlink' href='https://www.suse.com/products/server/'>SUSE Linux Enterprise - Only for some work stuff</a><br />
<br />
<h3 style='display: inline'>Other OSes I only had a glance at...</h3><br />
<br />
<a class='textlink' href='https://archiveos.org/opensolaris/'>OpenSolaris - Continuation of the open source version of Solaris</a><br />
<a class='textlink' href='https://archlinuxarm.org/'>Arch Linux ARM</a><br />
<a class='textlink' href='https://ecomstation.com/'>eComStation - Continuation of IBM OS/2</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Minix'>MINIX</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/OpenVMS'>OpenVMS</a><br />
<a class='textlink' href='https://en.wikipedia.org/wiki/OS/2'>IBM OS/2 Warp</a><br />
<a class='textlink' href='https://freedos.org'>FreeDOS - Open source alternative to DOS</a><br />
<a class='textlink' href='https://plan9.io/plan9/'>Plan9 </a><br />
<a class='textlink' href='https://reactos.org/'>ReactOS - A Microsoft Windows open source clone</a><br />
<a class='textlink' href='https://www.debian.org/ports/hurd/'>Debian GNU/Hurd - Debian on the GNU kernel</a><br />
<a class='textlink' href='https://www.debian.org/ports/kfreebsd-gnu/'>Debian GNU/kFreeBSD - Debian on the FreeBSD kernel</a><br />
<a class='textlink' href='https://www.gentoo.org'>Gentoo Linux</a><br />
<a class='textlink' href='https://www.haiku-os.org/'>Haiku - A BeOS open source clone</a><br />
<a class='textlink' href='https://www.oracle.com/solaris/solaris11/'>Sun Solaris (now owned by Oracle)</a><br />
<a class='textlink' href='https://www.puredarwin.org/'>OpenDarwin ("now" PureDarwin) - Open source operating system based on the open parts of macOS</a><br />
<br />
<h3 style='display: inline'>Other OSes which seem interesting...</h3><br />
<br />
<a class='textlink' href='https://asteroidos.org/'>Asteroids OS - Open source smartphone OS</a><br />
<a class='textlink' href='https://www.dragonflybsd.org/'>DragonFly BSD - Fork of FreeBSD 4</a><br />
<a class='textlink' href='http://wiki.postmarketos.org/wiki/Phosh'>Phosh (on postmarketOS) - A true Linux shell for the smartphone</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Welcome to the foo.zone</title>
<link href="gemini://foo.zone/gemfeed/2022-01-23-welcome-to-the-foo.zone.gmi" />
<id>gemini://foo.zone/gemfeed/2022-01-23-welcome-to-the-foo.zone.gmi</id>
<updated>2022-01-23T16:42:04+00:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I don't count this as a real blog post, but more of an announcement (I aim to write one real post once monthly). From now on, 'foo.zone' is the new address of this site. All other addresses will still forward to it and eventually (based on the traffic still going through) will be deactivated.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>Welcome to the foo.zone</h1><br />
<br />
<span class='quote'>Published at 2022-01-23T16:42:04+00:00</span><br />
<br />
<pre>
__
/ _| ___ ___ _______ _ __ ___
| |_ / _ \ / _ \ |_ / _ \| '_ \ / _ \
| _| (_) | (_) | / / (_) | | | | __/
|_| \___/ \___(_)___\___/|_| |_|\___|
</pre>
<br />
<span>I don't count this as a real blog post, but more of an announcement (I aim to write one real post once monthly). From now on, "foo.zone" is the new address of this site. All other addresses will still forward to it and eventually (based on the traffic still going through) will be deactivated.</span><br />
<br />
<span>As you can read on Wikipedia, "foo" is, alongside to "bar" and "baz", a metasyntactic variable (you know what I mean if you are a programmer or IT person):</span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Metasyntactic_variable'>https://en.wikipedia.org/wiki/Metasyntactic_variable</a><br />
<br />
<h2 style='display: inline'>What is the foo zone?</h2><br />
<br />
<span>It's my personal internet site and blog. Everything you read on this site is my personal opinion and experience. It's not intended to be anything professional. If you want my professional background, then go to my LinkedIn profile.</span><br />
<br />
<span>Since I re-booted this blog last year, I struggled to find a good host name for it. I started off with "buetow.org", and later I switched halfway to "snonux.de". Buetow is my last name, and snonux relates to some of my internet nicknames and personal IT projects. I also have a "SnonuxBSD" ASCII-art banner in the motd of my FreeBSD based home-NAS.</span><br />
<br />
<span>For a while, I was thinking about a better host name for this site, meeting the following criteria:</span><br />
<br />
<ul>
<li>Isn't directly linked to my name or my internet nicknames.</li>
<li>Reflects the "nature" of this site.</li>
<li>Is still pretty generic.</li>
<li>Is "cool".</li>
<li>Is short and easy to remember. </li>
<li>Doesn't cost millions.</li>
</ul><br />
<span>So I think that foo.zone is the perfect match. It's a bit geeky, but so is this site. The meta-syntactic variable relates to computer science and programming, so does this site. Other than that, staying in this sphere, it's a pretty generic name.</span><br />
<br />
<h2 style='display: inline'>To be in the .zone and not in a .surf club</h2><br />
<br />
<span>I was pretty happy finding out that foo.zone was still available for registration. I stumbled across it just yesterday while I was playing around with my new authoritative DNS servers. I was actually quite surprised as usually such short SLDs (second level domains), especially "foo", are all taken already.</span><br />
<br />
<span>As a funny bit, I almost chose "foo.surf" over "foo.zone" as in "surfing this site", but then decided against it as I would have to tell everyone that I am not into water sports so much. Well, on the other hand, I now may have to explain to non-programmers that I am not a fan of the rock band "Foo Fighters". But that will be acceptable, as I don't expect "normal" people visiting the foo zone as much anyway. If you reached as far, I have to congratulate you. You are not a normal person.</span><br />
<br />
<h2 style='display: inline'>What about my old hosts</h2><br />
<br />
<span>The host buetow.org will stay. However, not as the primary address for this site. I will keep using it for my personal internet infrastructure as well as for most of my E-Mail addresses. I used buetow.org for that over the past 10 years already anyway and that won't change any time soon. I don't know what I am going to do with snonux.de in the long run. A .de SLD (for Germany) is pretty cheap, so I might just keep it for now. </span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Bash Golf Part 2</title>
<link href="gemini://foo.zone/gemfeed/2022-01-01-bash-golf-part-2.gmi" />
<id>gemini://foo.zone/gemfeed/2022-01-01-bash-golf-part-2.gmi</id>
<updated>2022-01-01T23:36:15+00:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='BashGolfPart2'>Bash Golf Part 2</h1><br />
<br />
<span class='quote'>Published at 2022-01-01T23:36:15+00:00; Updated at 2022-01-05</span><br />
<br />
<pre>
'\ '\ . . |>18>>
\ \ . ' . |
O>> O>> . 'o |
\ .\. .. . |
/\ . /\ . . |
/ / . / / .' . |
jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Art by Joan Stark, mod. by Paul Buetow
</pre>
<br />
<span>This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.</span><br />
<br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2 (You are currently reading this)</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br />
<br />
<h2 style='display: inline' id='Redirection'>Redirection</h2><br />
<br />
<span>Let's have a closer look at Bash redirection. As you might already know that there are 3 standard file descriptors:</span><br />
<br />
<ul>
<li>0 aka stdin (standard input)</li>
<li>1 aka stdout (standard output)</li>
<li>2 aka stderr (standard error output)</li>
</ul><br />
<span>These are most certainly the ones you are using on regular basis. "/proc/self/fd" lists all file descriptors which are open by the current process (in this case: the current Bash shell itself):</span><br />
<br />
<pre>
❯ ls -l /proc/self/fd/
total 0
lrwx------. 1 paul paul 64 Nov 23 09:46 0 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 1 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 2 -> /dev/pts/9
lr-x------. 1 paul paul 64 Nov 23 09:46 3 -> /proc/162912/fd
</pre>
<br />
<span>The following examples demonstrate two different ways to accomplish the same thing. The difference is that the first command is directly printing out "Foo" to stdout and the second command is explicitly redirecting stdout to its own stdout file descriptor:</span><br />
<br />
<pre>
❯ echo Foo
Foo
❯ echo Foo > /proc/self/fd/0
Foo
</pre>
<br />
<span class='quote'>Update: A reader pointed out, that the redirection should actually go to <span class='inlinecode'>/proc/self/fd/1</span> and not <span class='inlinecode'>0</span>. But apparently, either way works for this particular example. Do you know why? </span><br />
<br />
<span>Other useful redirections are:</span><br />
<br />
<ul>
<li>Redirect stderr to stdin: "echo foo 2>&1"</li>
<li>Redirect stdin to stderr: "echo foo >&2"</li>
</ul><br />
<span>It is, however, not possible to redirect multiple times within the same command. E.g. the following won't work. You would expect stdin to be redirected to stderr and then stderr to be redirected to /dev/null. But as the example shows, Foo is still printed out:</span><br />
<br />
<pre>
❯ echo Foo 1>&2 2>/dev/null
Foo
</pre>
<br />
<span class='quote'>Update: A reader sent me an email and pointed out that the order of the redirections is important. </span><br />
<br />
<span>As you can see, the following will not print out anything:</span><br />
<br />
<pre>
❯ echo Foo 2>/dev/null 1>&2
❯
</pre>
<br />
<span>A good description (also pointed out by the reader) can be found here:</span><br />
<br />
<a class='textlink' href='https://wiki.bash-hackers.org/howto/redirection_tutorial#order_of_redirection_ie_file_2_1_vs_2_1_file'>Order of redirection</a><br />
<br />
<span>Ok, back to the original blog post. You can also use grouping here (neither of these commands will print out anything to stdout):</span><br />
<br />
<pre>
❯ { echo Foo 1>&2; } 2>/dev/null
❯ ( echo Foo 1>&2; ) 2>/dev/null
❯ { { { echo Foo 1>&2; } 2>&1; } 1>&2; } 2>/dev/null
❯ ( ( ( echo Foo 1>&2; ) 2>&1; ) 1>&2; ) 2>/dev/null
❯
</pre>
<br />
<span>A handy way to list all open file descriptors is to use the "lsof" command (that's not a Bash built-in), whereas $ is the process id (pid) of the current shell process:</span><br />
<br />
<pre>
❯ lsof -a -p $ -d0,1,2
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 62676 paul 0u CHR 136,9 0t0 12 /dev/pts/9
bash 62676 paul 1u CHR 136,9 0t0 12 /dev/pts/9
bash 62676 paul 2u CHR 136,9 0t0 12 /dev/pts/9
</pre>
<br />
<span>Let's create our own descriptor "3" for redirection to a file named "foo":</span><br />
<br />
<pre>
❯ touch foo
❯ exec 3>foo # This opens fd 3 and binds it to file foo.
❯ ls -l /proc/self/fd/3
l-wx------. 1 paul paul 64 Nov 23 10:10 \
/proc/self/fd/3 -> /home/paul/foo
❯ cat foo
❯ echo Bratwurst >&3
❯ cat foo
Bratwurst
❯ exec 3>&- # This closes fd 3.
❯ echo Steak >&3
-bash: 3: Bad file descriptor
</pre>
<br />
<span>You can also override the default file descriptors, as the following example script demonstrates:</span><br />
<br />
<pre>
❯ cat grandmaster.sh
#!/usr/bin/env bash
# Write a file data-file containing two lines
echo Learn You a Haskell > data-file
echo for Great Good >> data-file
# Link fd with fd 6 (saves default stdin)
exec 6<&0
# Overwrite stdin with data-file
exec < data-file
# Read the first two lines from it
declare LINE1 LINE2
read LINE1
read LINE2
# Print them
echo First line: $LINE1
echo Second line: $LINE2
# Restore default stdin and delete fd 6
exec 0<&6 6<&-
</pre>
<br />
<span>Let's execute it:</span><br />
<br />
<pre>
❯ chmod 750 ./grandmaster.sh
❯ ./grandmaster.sh
First line: Learn You a Haskell
Second line: for Great Good
</pre>
<br />
<h2 style='display: inline' id='HERE'>HERE</h2><br />
<br />
<span>I have mentioned HERE-documents and HERE-strings already in this post. Let's do some more examples. The following "cat" receives a multi line string from stdin. In this case, the input multi line string is a HERE-document. As you can see, it also interpolates variables (in this case the output of "date" running in a subshell).</span><br />
<br />
<pre>
❯ cat <<END
> Hello World
> It’s $(date)
> END
Hello World
It's Fri 26 Nov 08:46:52 GMT 2021
</pre>
<br />
<span>You can also write it the following way, but that's less readable (it's good for an obfuscation contest):</span><br />
<br />
<pre>
❯ <<END cat
> Hello Universe
> It’s $(date)
> END
Hello Universe
It's Fri 26 Nov 08:47:32 GMT 2021
</pre>
<br />
<span>Besides of an HERE-document, there is also a so-called HERE-string. Besides of...</span><br />
<br />
<pre>
❯ declare VAR=foo
❯ if echo "$VAR" | grep -q foo; then
> echo '$VAR ontains foo'
> fi
$VAR ontains foo
</pre>
<br />
<span>...you can use a HERE-string like that:</span><br />
<br />
<pre>
❯ if grep -q foo <<< "$VAR"; then
> echo '$VAR contains foo'
> fi
$VAR contains foo
</pre>
<br />
<span>Or even shorter, you can do:</span><br />
<br />
<pre>
❯ grep -q foo <<< "$VAR" && echo '$VAR contains foo'
$VAR contains foo
</pre>
<br />
<span>You can also use a Bash regex to accomplish the same thing, but the points of the examples so far were to demonstrate HERE-{documents,strings} and not Bash regular expressions:</span><br />
<br />
<pre>
❯ if [[ "$VAR" =~ foo ]]; then echo yay; fi
yay
</pre>
<br />
<span>You can also use it with "read":</span><br />
<br />
<pre>
❯ read a <<< ja
❯ echo $a
ja
❯ read b <<< 'NEIN!!!'
❯ echo $b
NEIN!!!
❯ dumdidumstring='Learn you a Golang for Great Good'
❯ read -a words <<< "$dumdidumstring"
❯ echo ${words[0]}
Learn
❯ echo ${words[3]}
Golang
</pre>
<br />
<span>The following is good for an obfuscation contest too:</span><br />
<br />
<pre>
❯ echo 'I like Perl too' > perllove.txt
❯ cat - perllove.txt <<< "$dumdidumstring"
Learn you a Golang for Great Good
I like Perl too
</pre>
<br />
<h2 style='display: inline' id='RANDOM'>RANDOM</h2><br />
<br />
<span>Random is a special built-in variable containing a different pseudo random number each time it's used.</span><br />
<br />
<pre>
❯ echo $RANDOM
11811
❯ echo $RANDOM
14997
❯ echo $RANDOM
9104
</pre>
<br />
<span>That's very useful if you want to randomly delay the execution of your scripts when you run it on many servers concurrently, just to spread the server load (which might be caused by the script run) better.</span><br />
<br />
<span>Let's say you want to introduce a random delay of 1 minute. You can accomplish it with:</span><br />
<br />
<pre>
❯ cat ./calc_answer_to_ultimate_question_in_life.sh
#!/usr/bin/env bash
declare -i MAX_DELAY=60
random_delay () {
local -i sleep_for=$((RANDOM % MAX_DELAY))
echo "Delaying script execution for $sleep_for seconds..."
sleep $sleep_for
echo 'Continuing script execution...'
}
main () {
random_delay
# From here, do the real work. Calculating the answer to
# the ultimate question can take billions of years....
: ....
}
main
❯
❯ ./calc_answer_to_ultimate_question_in_life.sh
Delaying script execution for 42 seconds...
Continuing script execution...
</pre>
<br />
<h2 style='display: inline' id='setxandseteandpipefile'>set -x and set -e and pipefile</h2><br />
<br />
<span>In my opinion, -x and -e and pipefile are the most useful Bash options. Let's have a look at them one after another.</span><br />
<br />
<h3 style='display: inline' id='x'>-x</h3><br />
<br />
<span>-x prints commands and their arguments as they are executed. This helps to develop and debug your Bash code:</span><br />
<br />
<pre>
❯ set -x
❯ square () { local -i num=$1; echo $((num*num)); }
❯ num=11; echo "Square of $num is $(square $num)"
+ num=11
++ square 11
++ local -i num=11
++ echo 121
+ echo 'Square of 11 is 121'
Square of 11 is 121
</pre>
<br />
<span>You can also set it when calling an external script without modifying the script itself:</span><br />
<br />
<pre>
❯ bash -x ./half_broken_script_to_be_debugged.sh
</pre>
<br />
<span>Let's do that on one of the example scripts we covered earlier:</span><br />
<br />
<pre>
❯ bash -x ./grandmaster.sh
+ bash -x ./grandmaster.sh
+ echo Learn You a Haskell
+ echo for Great Good
+ exec
+ exec
+ declare LINE1 LINE2
+ read LINE1
+ read LINE2
+ echo First line: Learn You a Haskell
First line: Learn You a Haskell
+ echo Second line: for Great Good
Second line: for Great Good
+ exec
❯
</pre>
<br />
<h3 style='display: inline' id='e'>-e</h3><br />
<br />
<span>This is a very important option you want to use when you are paranoid. This means, you should always "set -e" in your scripts when you need to make absolutely sure that your script runs successfully (with that I mean that no command should exit with an unexpected status code).</span><br />
<br />
<span>Ok, let's dig deeper:</span><br />
<br />
<pre>
❯ help set | grep -- -e
-e Exit immediately if a command exits with a non-zero status.
</pre>
<br />
<span>As you can see in the following example, the Bash terminates after the execution of "grep" as "foo" is not matching "bar". Therefore, grep exits with 1 (unsuccessfully) and the shell aborts. And therefore, "bar" will not be printed out anymore:</span><br />
<br />
<pre>
❯ bash -c 'set -e; echo hello; grep -q bar <<< foo; echo bar'
hello
❯ echo $?
1
</pre>
<br />
<span>Whereas the outcome changes when the regex matches:</span><br />
<br />
<pre>
❯ bash -c 'set -e; echo hello; grep -q bar <<< barman; echo bar'
hello
bar
❯ echo $?
0
</pre>
<br />
<span>So does it mean that grep will always make the shell terminate whenever its exit code isn't 0? This will render "set -e" quite unusable. Frankly, there are other commands where an exit status other than 0 should not terminate the whole script abruptly. Usually, what you want is to branch your code based on the outcome (exit code) of a command:</span><br />
<br />
<pre>
❯ bash -c 'set -e
> grep -q bar <<< foo
> if [ $? -eq 0 ]; then
> echo "matching"
> else
> echo "not matching"
> fi'
❯ echo $?
1
</pre>
<br />
<span>...but the example above won't reach any of the branches and won't print out anything, as the script terminates right after grep.</span><br />
<br />
<span>The proper solution is to use grep as an expression in a conditional (e.g. in an if-else statement):</span><br />
<br />
<pre>
❯ bash -c 'set -e
> if grep -q bar <<< foo; then
> echo "matching"
> else
> echo "not matching"
> fi'
not matching
❯ echo $?
0
❯ bash -c 'set -e
> if grep -q bar <<< barman; then
> echo "matching"
> else
> echo "not matching"
> fi'
matching
❯ echo $?
0
</pre>
<br />
<span>You can also temporally undo "set -e" if there is no other way:</span><br />
<br />
<pre>
❯ cat ./e.sh
#!/usr/bin/env bash
set -e
foo () {
local arg="$1"; shift
if [ -z "$arg" ]; then
arg='You!'
fi
echo "Hello $arg"
}
bar () {
# Temporally disable e
set +e
local arg="$1"; shift
# Enable e again.
set -e
if [ -z "$arg" ]; then
arg='You!'
fi
echo "Hello $arg"
}
# Will succeed
bar World
foo Universe
bar
# Will terminate the script
foo
❯ ./e.sh
Hello World
Hello Universe
Hello You!
</pre>
<br />
<span>Why does calling "foo" with no arguments make the script terminate? Because as no argument was given, the "shift" won't have anything to do as the argument list $@ is empty, and therefore "shift" fails with a non-zero status.</span><br />
<br />
<span>Why would you want to use "shift" after function-local variable assignments? Have a look at my personal Bash coding style guide for an explanation :-):</span><br />
<br />
<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>./2021-05-16-personal-bash-coding-style-guide.html</a><br />
<br />
<h3 style='display: inline' id='pipefail'>pipefail</h3><br />
<br />
<span>The pipefail option makes it so that not only the exit code of the last command of the pipe counts regards its exit code but any command of the pipe:</span><br />
<br />
<pre>
❯ help set | grep pipefail -A 2
pipefail the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
</pre>
<br />
<span>The following greps for paul in passwd and converts all lowercase letters to uppercase letters. The exit code of the pipe is 0, as the last command of the pipe (converting from lowercase to uppercase) succeeded:</span><br />
<br />
<pre>
❯ grep paul /etc/passwd | tr '[a-z]' '[A-Z]'
PAUL:X:1000:1000:PAUL BUETOW:/HOME/PAUL:/BIN/BASH
❯ echo $?
0
</pre>
<br />
<span>Let's look at another example, where "TheRock" doesn't exist in the passwd file. However, the pipes exit status is still 0 (success). This is so because the last command ("tr" in this case) still succeeded. It is just that it didn't get any input on stdin to process:</span><br />
<br />
<pre>
❯ grep TheRock /etc/passwd
❯ echo $?
1
❯ grep TheRock /etc/passwd | tr '[a-z]' '[A-Z]'
❯ echo $?
0
</pre>
<br />
<span>To change this behaviour, pipefile can be used. Now, the pipes exit status is 1 (fail), because the pipe contains at least one command (in this case grep) which exited with status 1:</span><br />
<br />
<pre>
❯ set -o pipefail
❯ grep TheRock /etc/passwd | tr '[a-z]' '[A-Z]'
❯ echo $?
1
</pre>
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>2021-05-16 Personal Bash coding style guide</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2 (You are currently reading this)</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>How to stay sane as a DevOps person </title>
<link href="gemini://foo.zone/gemfeed/2021-12-26-how-to-stay-sane-as-a-devops-person.gmi" />
<id>gemini://foo.zone/gemfeed/2021-12-26-how-to-stay-sane-as-a-devops-person.gmi</id>
<updated>2021-12-26T12:02:02+00:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>Log4shell (CVE-2021-44228) made it clear, once again, that working in information technology is not an easy job (especially when you are a DevOps person). I thought it would be interesting to summarize a few techniques to help you to relax.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>How to stay sane as a DevOps person </h1><br />
<br />
<span class='quote'>Published at 2021-12-26T12:02:02+00:00; Updated at 2022-01-12</span><br />
<br />
<pre>
)
) (( (
( )) )
) ) // (
_ ( __ ( ~->>
,-----' |__,_~~___<'__`)-~__--__-~->> <
| // : | -__ ~__ o)____)),__ - '> >- >
| // : |- \_ \ -\_\ -\ \ \ ~\_ \ ->> - , >>
| // : |_~_\ -\__\ \~'\ \ \, \__ . -<- >>
`-----._| ` -__`-- - ~~ -- ` --~> >
_/___\_ //)_`// | ||]
_____[_______]_[~~-_ (.L_/ ||
[____________________]' `\_,/'/
||| / ||| ,___,'./
||| \ |||,'______|
||| / /|| I==||
||| \ __/_|| __||__
-----||-/------`-._/||-o--o---o---
~~~~~'
</pre>
<br />
<span>Log4shell (CVE-2021-44228) made it clear, once again, that working in information technology is not an easy job (especially when you are a DevOps person). I thought it would be interesting to summarize a few techniques to help you to relax.</span><br />
<br />
<span>(PS: When I mean DevOps, I also mean Site Reliability Engineers and Sysadmins. I believe SRE, DevOps Engineer and Sysadmin are just synonym titles for the same job).</span><br />
<br />
<a class='textlink' href='https://en.wikipedia.org/wiki/Log4Shell'>https://en.wikipedia.org/wiki/Log4Shell</a><br />
<br />
<h2 style='display: inline'>Set clear expectations</h2><br />
<br />
<span>It's important to set clear expectations. It can be difficult to guess what others expect or don't expect from you. If you know exactly what you are supposed to do, you can work towards a specific goal and don't worry about all the other noise so much.</span><br />
<br />
<span>However, if you are in a more senior position, it is expected from you to plan your tasks by yourself to a large degree and also be flexible, so you can react quickly to new situations (e.g. resolving incidents). Also, to a large degree, you have to prioritise your work by yourself. This can overthrow all of your plans. In extreme cases, it can help to share your plans with your team so that everyone is on the same page. Afterwards, be the execution machine. People are happy when they see that stuff gets done. Communicate clearly all critical work you do. This will capture all the technical debt there might be. It does not help in the long run if things are fixed in the background without any visibility. </span><br />
<br />
<span>Due to politeness, many people are not setting clear expectations. I personally may sound sometimes "too German" when setting expectations, but so far nobody complained, and I have even received positive feedback about it.</span><br />
<br />
<h2 style='display: inline'>Always respond to requests but set expectations and boundaries</h2><br />
<br />
<span>There are many temptations to get side-tracked by other projects and/or issues. It is important to set boundaries here. But always answer to all requests as nothing is more frustrating than asking a person and never getting any answer back. This is especially the case when everyone is working form home where people are using tools such as Slack and E-Mail for most of their communications.</span><br />
<br />
<h3 style='display: inline'>Dealing with requests</h3><br />
<br />
<span>If the request is urgent, and you have the capacity to help, probably you should help. If it's not urgent, maybe ask to pospone the request (e.g. ask to create a ticket, so that someone from your team can work on it later).</span><br />
<br />
<span>If the request is urgent, but you don't have the knowledge or the capacity to help, try to defer to a colleague who might be able to help. You could also provide some quick tips and hints, so that the requester can resolve the issue by himself. Make it transparent why you might not have the time right now, as this can help the person to review his own priorities or to escalate. </span><br />
<br />
<h3 style='display: inline'>Escalation is only a tool</h3><br />
<br />
<span>Never make or take an escalation personally. The only forms of escalation should be due to technical issues or lack of resources. An escalation then becomes like a math equation and does not need human resources involved. So de-facto, an escalation is nothing negative, but just a process people can follow to form decision-making. In a good company escalations tend to be an exception, though. Staff knows how to deal with the things by themselves without bothering management too much. </span><br />
<br />
<h2 style='display: inline'>Think positively</h2><br />
<br />
<span>If times are very stressful, think that it could always be worse:</span><br />
<br />
<ul>
<li>Nobody is dying, we are only doing some IT stuff.</li>
<li>Your time after work is your own time, look forward to time with your family or a nice dinner or your favourite sports class.</li>
<li>You probably will never run out of work in the IT sector. So you will always be able to make a living.</li>
<li>Your IT job and life is actually pretty good (compared to a homeless person for example). You are probably part of the world's top 1% regarding life standard.</li>
</ul><br />
<h2 style='display: inline'>Go slower even if you could go faster</h2><br />
<br />
<span>When working in a team, you may feel that you could get done things faster when you just did everything by yourself. This can be a bit frustrating at times, as you might need to work late hours and also might need to explain things over and over again to others. Also, you could be the one who needs to get things explained over and over again as you are not so familiar with the topic (yet). You will appreciate it if the other person is slowing down for you a bit.</span><br />
<br />
<h3 style='display: inline'>You work in a team</h3><br />
<br />
<span>Security is a team sport. So slow down and make sure that everyone is on track with the goals. You can go full-speed with your very own subtasks, though. Not everyone knows how to use all the tools so well like a full-time DevOps person. As a DevOps person, you are not a security expert, though. Security experts are different people in your company, but DevOps will be the main tribe deploying mitigations (following the security recommendations) and management will be the main tribe coordinating all the efforts. </span><br />
<br />
<span>So even if you think that you can do everything faster by your own, can you really? You probably don't know what you don't know about IT security. The more you know about it, the more you know about what you don't know.</span><br />
<br />
<h3 style='display: inline'>Don't rush</h3><br />
<br />
<span>Slowing down also helps to prevent errors. Don't rush your tasks, even if they are urgent. Try to be quick, but don't rush them. Maybe you are writing a script to mitigate a production issue. You could others peer review that script, for example. Their primary programming language may not be the same (e.g. Golang vs Perl), but they would understand the logic. Or ask another DevOps person from your company with good scripting skills review your mitigation, but he then may lack the domain knowledge of the software you are patching. So in either case, the review will take a bit longer as the reviewer might not be an expert in everything.</span><br />
<br />
<span>So relax, don't always expect immediate results. Set clear and reasonable timelines for the management about the mitigations. You are not a superhero who has to do everything by yourself. Sometimes, you will miss a deadline. But that will have good reasons. Don't rush to complete just to meet a deadline. </span><br />
<br />
<a class='textlink' href='./2021-10-22-defensive-devops.html'>Read also "Defensive DevOps" about deploying mitigation scripts.</a><br />
<br />
<h2 style='display: inline'>You are not a superhero</h2><br />
<br />
<span>Always keep that in mind. You can't solve all problems by your own. Maybe you could, but that would be a lot of additional stress (and this will reflect to your personal life). Also, Superman and Wonder Woman receive much higher salaries than you will ever do ;-).</span><br />
<br />
<span>I have been a superhero multiple times mitigating critical incidents, and I was proud about it in those moments. But actually, I am not proud looking at those retrospectively as for everything there should be other people around who should be able to resolve an incident. No company should rely on a single person, there must always be a substitute. You are not a superhero and as harsh as it sounds, everyone is replaceable. Every superhero can be replaced with another superhero. The only thing it takes to become a superhero is time to get to know the infrastructure and tools very well, paired with work dedication.</span><br />
<br />
<span>This doesn't mean, that you shouldn't try your best. But you don't need to try to be the superhero. Maybe someone else will be the superhero, but that's OK as long as it's not always the same person every time. Everyone can have a good day after all. If I could choose between being a superhero or having a good night sleep, I would probably prefer the sleep. </span><br />
<br />
<h3 style='display: inline'>Give away some of your superpowers</h3><br />
<br />
<span>If you are a superhero, try to give away some of your superpowers, so that you can relax in the evening knowing that others (e.g. the current on-call engineers) know how to tackle things. Every member of the team needs to do DevOps (even the team managers, in my humble opinion). Some may be less experienced than others or have other expertises, but to counteract this you could document the recurring tasks so that they are easy to follow (which then later could be either automated away or, even better, fully fixed).</span><br />
<br />
<span>On the other side, if you are a DevOps person, try to sneak into other people's shoes too. For example, you might not be an expert in Java programming, but a lot of the infrastructure is programmed in Java. This is where usually the Software Developers and Engineers shine. But if you know how to read, debug and even extend Java code too (by learning from the Software Developer superheroes), then your will only benefit from it. </span><br />
<br />
<span>So you are not a superhero. Or, if you are a superhero, then all colleagues should be superheroes too.</span><br />
<br />
<h2 style='display: inline'>Don't jump on all problems immediately</h2><br />
<br />
<span>In a perfect world, every member of a team comes along with the same strengths and skills. But in reality, everyone is different. </span><br />
<br />
<span>In order to distribute the troubleshooting skills across the team, you should not jump on every problem immediately. Leave some space for others to resolve the issue. This is where the best learning happens. Nobody will learn from you when you solve all problems. People might learn something after you explained what you did, but the takeaways will be minimal compared to when people try to resolve issues by themselves. Always be available for questions which will help your colleagues to steer into the right direction and if you think it helps, give them some tips resolving the issue, even if they didn't ask for it. Sometimes, engineers are too proud to ask. </span><br />
<br />
<span>The whole paragraph changes when there is an issue you don't know how to resolve. Jump on it, so you can learn from it. But also ask for advice if you are unsure about it.</span><br />
<br />
<span>If the issue is a very critical one, then you might better off trying to resolve it as fast as possible with your full powers in order to avoid any major damage to the company. This, of course, only works if you know how to resolve it quickly. So, don't leave others with not much experience yet looking at it. If possible, work with the team to resolve the issue. Unfortunately, solving it with the team is not always the fastest way. So in this particular circumstance, the company may be better off being saved by a single superhero. Make sure that the problem will not occur again or, at least, that others can fix it the next time without Superman flying by.</span><br />
<br />
<h2 style='display: inline'>Force breaks; and shutdown now</h2><br />
<br />
<span>Be strict about your time off. Nowadays, tech workers check their messages also out of office hours and are reachable 24/7. This really should only be the case when you are on-call, to be honest (or if you work for a startup). All other out-of-office time is owned by you and not your employer. You have signed an 40 hour/week and not 7 days/week contract. Of course, there will be always some sort of flexibility and exceptions. You might need to work over the weekend to get a migration done or a problem solved. But to balance it out, you should have other days off as substitutes.</span><br />
<br />
<span>It's important to shut down your brain from work during your breaks (be strict with your breaks, leave your desk for lunch or for a walk early afternoon and if you aren't on-call also don't take your work-phone with you). You will be happier and also much more energized and productive in the afternoon. Also, when you are reachable 24/7, your colleagues will start thinking that you don't have anything more important to do than work.</span><br />
<br />
<h2 style='display: inline'>Block time every day for personal advance</h2><br />
<br />
<span>It does not matter how many tasks are in your backlog or how many issues are to be tackled. *Always* find time for personal advance. The most issues aren't critical anyway and can wait a bit. At the end of the day, you will have a nice feeling that you have accomplished something meaningful. This can be an interesting project or learning a new technology you are interested in. Of course, there must be consensus with your manager (unless you do that kind of thing in your personal time of course). </span><br />
<br />
<span>If you are too busy at work and just can't block time, then maybe it's time to think about alternatives. But before you do that, probably there is something else you can do. Perhaps you just think you can't block time, but you would be positively surprised to hear from your manager that he will fully support you. Of course, he won't agree to you working full-time on your pet projects. But a certain portion of your time should be allocated for personal advance. After all, your employer also want's you to stay happy so that you don't look for alternatives. It's of everyone's interest that you like your job and stay motivated. The more you are motivated, the more productive you are. The more productive you are, the more valuable you are for the company.</span><br />
<br />
<h2 style='display: inline'>More</h2><br />
<br />
<span>Another blog post worth reading:</span><br />
<br />
<a class='textlink' href='https://unixsheikh.com/articles/how-to-stay-sane-in-todays-world-of-tech.html'>https://unixsheikh.com/articles/how-to-stay-sane-in-todays-world-of-tech.html</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Bash Golf Part 1</title>
<link href="gemini://foo.zone/gemfeed/2021-11-29-bash-golf-part-1.gmi" />
<id>gemini://foo.zone/gemfeed/2021-11-29-bash-golf-part-1.gmi</id>
<updated>2021-11-29T14:06:14+00:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>This is the first blog post about my Bash Golf series. This series is about random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='BashGolfPart1'>Bash Golf Part 1</h1><br />
<br />
<span class='quote'>Published at 2021-11-29T14:06:14+00:00; Updated at 2022-01-05</span><br />
<br />
<pre>
'\ . . |>18>>
\ . ' . |
O>> . 'o |
\ . |
/\ . |
/ / .' |
jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Art by Joan Stark
</pre>
<br />
<span>This is the first blog post about my Bash Golf series. This series is about random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.</span><br />
<br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1 (You are currently reading this)</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br />
<br />
<h2 style='display: inline' id='TCPIPnetworking'>TCP/IP networking</h2><br />
<br />
<span>You probably know the Netcat tool, which is a swiss army knife for TCP/IP networking on the command line. But did you know that the Bash natively supports TCP/IP networking?</span><br />
<br />
<span>Have a look here how that works:</span><br />
<br />
<pre>
❯ cat < /dev/tcp/time.nist.gov/13
59536 21-11-18 08:09:16 00 0 0 153.6 UTC(NIST) *
</pre>
<br />
<span>The Bash treats /dev/tcp/HOST/PORT in a special way so that it is actually establishing a TCP connection to HOST:PORT. The example above redirects the TCP output of the time-server to cat and cat is printing it on standard output (stdout).</span><br />
<br />
<span>A more sophisticated example is firing up an HTTP request. Let's create a new read-write (rw) file descriptor (fd) 5, redirect the HTTP request string to it, and then read the response back:</span><br />
<br />
<pre>
❯ exec 5<>/dev/tcp/google.de/80
❯ echo -e "GET / HTTP/1.1\nhost: google.de\n\n" >&5
❯ cat <&5 | head
HTTP/1.1 301 Moved Permanently
Location: http://www.google.de/
Content-Type: text/html; charset=UTF-8
Date: Thu, 18 Nov 2021 08:27:18 GMT
Expires: Sat, 18 Dec 2021 08:27:18 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 218
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
</pre>
<br />
<span>You would assume that this also works with the ZSH, but it doesn't. This is one of the few things which don't work with the ZSH but in the Bash. There might be plugins you could use for ZSH to do something similar, though.</span><br />
<br />
<h2 style='display: inline' id='Processsubstitution'>Process substitution</h2><br />
<br />
<span>The idea here is, that you can read the output (stdout) of a command from a file descriptor:</span><br />
<br />
<pre>
❯ uptime # Without process substitution
10:58:03 up 4 days, 22:08, 1 user, load average: 0.16, 0.34, 0.41
❯ cat <(uptime) # With process substitution
10:58:16 up 4 days, 22:08, 1 user, load average: 0.14, 0.33, 0.41
❯ stat <(uptime)
File: /dev/fd/63 -> pipe:[468130]
Size: 64 Blocks: 0 IO Block: 1024 symbolic link
Device: 16h/22d Inode: 468137 Links: 1
Access: (0500/lr-x------) Uid: ( 1001/ paul) Gid: ( 1001/ paul)
Context: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Access: 2021-11-20 10:59:31.482411961 +0000
Modify: 2021-11-20 10:59:31.482411961 +0000
Change: 2021-11-20 10:59:31.482411961 +0000
Birth: -
</pre>
<br />
<span>This example doesn't make any sense practically speaking, but it clearly demonstrates how process substitution works. The standard output pipe of "uptime" is redirected to an anonymous file descriptor. That fd then is opened by the "cat" command as a regular file.</span><br />
<br />
<span>A useful use case is displaying the differences of two sorted files:</span><br />
<br />
<pre>
❯ echo a > /tmp/file-a.txt
❯ echo b >> /tmp/file-a.txt
❯ echo c >> /tmp/file-a.txt
❯ echo b > /tmp/file-b.txt
❯ echo a >> /tmp/file-b.txt
❯ echo c >> /tmp/file-b.txt
❯ echo X >> /tmp/file-b.txt
❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt)
--- /dev/fd/63 2021-11-20 11:05:03.667713554 +0000
+++ /dev/fd/62 2021-11-20 11:05:03.667713554 +0000
@@ -1,3 +1,4 @@
a
b
c
+X
❯ echo X >> /tmp/file-a.txt # Now, both files have the same content again.
❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt)
❯
</pre>
<br />
<span>Another example is displaying the differences of two directories:</span><br />
<br />
<pre>
❯ diff -u <(ls ./dir1/ | sort) <(ls ./dir2/ | sort)
</pre>
<br />
<span>More (Bash golfing) examples:</span><br />
<br />
<pre>
❯ wc -l <(ls /tmp/) /etc/passwd <(env)
24 /dev/fd/63
49 /etc/passwd
24 /dev/fd/62
97 total
❯
❯ while read foo; do
> echo $foo
> done < <(echo foo bar baz)
foo bar baz
❯
</pre>
<br />
<span>So far, we only used process substitution for stdout redirection. But it also works for stdin. The following two commands result into the same outcome, but the second one is writing the tar data stream to an anonymous file descriptor which is substituted by the "bzip2" command reading the data stream from stdin and compressing it to its own stdout, which then gets redirected to a file:</span><br />
<br />
<pre>
❯ tar cjf file.tar.bz2 foo
❯ tar cjf >(bzip2 -c > file.tar.bz2) foo
</pre>
<br />
<span>Just think a while and see whether you understand fully what is happening here.</span><br />
<br />
<h2 style='display: inline' id='Grouping'>Grouping</h2><br />
<br />
<span>Command grouping can be quite useful for combining the output of multiple commands:</span><br />
<br />
<pre>
❯ { ls /tmp; cat /etc/passwd; env; } | wc -l
97
❯ ( ls /tmp; cat /etc/passwd; env; ) | wc -l
97
</pre>
<br />
<span>But wait, what is the difference between curly braces and normal braces? I assumed that the normal braces create a subprocess whereas the curly ones don't, but I was wrong:</span><br />
<br />
<pre>
❯ echo $
62676
❯ { echo $; }
62676
❯ ( echo $; )
62676
</pre>
<br />
<span>One difference is, that the curly braces require you to end the last statement with a semicolon, whereas with the normal braces you can omit the last semicolon:</span><br />
<br />
<pre>
❯ ( env; ls ) | wc -l
27
❯ { env; ls } | wc -l
>
> ^C
</pre>
<br />
<span>In case you know more (subtle) differences, please write me an E-Mail and let me know.</span><br />
<br />
<span class='quote'>Update: A reader sent me an E-Mail and pointed me to the Bash manual page, which explains the difference between () and {} (I should have checked that by myself):</span><br />
<br />
<pre>
(list) list is executed in a subshell environment (see COMMAND EXECUTION ENVIRONMENT
below). Variable assignments and builtin commands that affect the shell's
environment do not remain in effect after the command completes. The return
status is the exit status of list.
{ list; }
list is simply executed in the current shell environment. list must be ter‐
minated with a newline or semicolon. This is known as a group command. The
return status is the exit status of list. Note that unlike the metacharac‐
ters ( and ), { and } are reserved words and must occur where a reserved word
is permitted to be recognized. Since they do not cause a word break, they
must be separated from list by whitespace or another shell metacharacter.
</pre>
<br />
<span>So I was right that () is executed in a subprocess. But why does $ not show a different PID? Also here (as pointed out by the reader) is the answer in the manual page:</span><br />
<br />
<pre>
$ Expands to the process ID of the shell. In a () subshell, it expands to the
process ID of the current shell, not the subshell.
</pre>
<br />
<span>If we want print the subprocess PID, we can use the BASHPID variable:</span><br />
<br />
<pre>
❯ echo $BASHPID; { echo $BASHPID; }; ( echo $BASHPID; )
1028465
1028465
1028739
</pre>
<br />
<h2 style='display: inline' id='Expansions'>Expansions</h2><br />
<br />
<span>Let's start with simple examples:</span><br />
<br />
<pre>
❯ echo {0..5}
0 1 2 3 4 5
❯ for i in {0..5}; do echo $i; done
0
1
2
3
4
5
</pre>
<br />
<span>You can also add leading 0 or expand to any number range:</span><br />
<br />
<pre>
❯ echo {00..05}
00 01 02 03 04 05
❯ echo {000..005}
000 001 002 003 004 005
❯ echo {201..205}
201 202 203 204 205
</pre>
<br />
<span>It also works with letters:</span><br />
<br />
<pre>
❯ echo {a..e}
a b c d e
</pre>
<br />
<span>Now it gets interesting. The following takes a list of words and expands it so that all words are quoted:</span><br />
<br />
<pre>
❯ echo \"{These,words,are,quoted}\"
"These" "words" "are" "quoted"
</pre>
<br />
<span>Let's also expand to the cross product of two given lists:</span><br />
<br />
<pre>
❯ echo {one,two}\:{A,B,C}
one:A one:B one:C two:A two:B two:C
❯ echo \"{one,two}\:{A,B,C}\"
"one:A" "one:B" "one:C" "two:A" "two:B" "two:C"
</pre>
<br />
<span>Just because we can:</span><br />
<br />
<pre>
❯ echo Linux-{one,two,three}\:{A,B,C}-FreeBSD
Linux-one:A-FreeBSD Linux-one:B-FreeBSD Linux-one:C-FreeBSD Linux-two:A-FreeBSD Linux-two:B-FreeBSD Linux-two:C-FreeBSD Linux-three:A-FreeBSD Linux-three:B-FreeBSD Linux-three:C-FreeBSD
</pre>
<br />
<h2 style='display: inline' id='akastdinandstdoutplaceholder'>- aka stdin and stdout placeholder</h2><br />
<br />
<span>Some commands and Bash builtins use "-" as a placeholder for stdin and stdout:</span><br />
<br />
<pre>
❯ echo Hello world
Hello world
❯ echo Hello world | cat -
Hello world
❯ cat - <<ONECHEESEBURGERPLEASE
Hello world
ONECHEESEBURGERPLEASE
Hello world
❯ cat - <<< 'Hello world'
Hello world
</pre>
<br />
<span>Let's walk through all three examples from the above snippet:</span><br />
<br />
<ul>
<li>The first example is obvious (the Bash builtin "echo" prints its arguments to stdout).</li>
<li>The second pipes "Hello world" via stdout to stdin of the "cat" command. As cat's argument is "-" it reads its data from stdin and not from a regular file named "-". So "-" has a special meaning for cat.</li>
<li>The third and fourth examples are interesting as we don't use a pipe as of "|" but a so-called HERE-document and a HERE-string. But the end results are the same.</li>
</ul><br />
<span>The "tar" command understands "-" too. The following example tars up some local directory and sends the data to stdout (this is what "-f -" commands it to do). stdout then is piped via an SSH session to a remote tar process (running on buetow.org) and reads the data from stdin and extracts all the data coming from stdin (as we told tar with "-f -") on the remote machine:</span><br />
<br />
<pre>
❯ tar -czf - /some/dir | ssh hercules@buetow.org tar -xzvf -
</pre>
<br />
<span>This is yet another example of using "-", but this time using the "file" command:</span><br />
<br />
<pre>
$ head -n 1 grandmaster.sh
#!/usr/bin/env bash
$ file - < <(head -n 1 grandmaster.sh)
/dev/stdin: a /usr/bin/env bash script, ASCII text executable
</pre>
<br />
<span>Some more golfing:</span><br />
<br />
<pre>
$ cat -
hello
hello
^C
$ file -
#!/usr/bin/perl
/dev/stdin: Perl script text executable
</pre>
<br />
<h2 style='display: inline' id='Alternativeargumentpassing'>Alternative argument passing</h2><br />
<br />
<span>This is a quite unusual way of passing arguments to a Bash script:</span><br />
<br />
<pre>
❯ cat foo.sh
#/usr/bin/env bash
declare -r USER=${USER:?Missing the username}
declare -r PASS=${PASS:?Missing the secret password for $USER}
echo $USER:$PASS
</pre>
<br />
<span>So what we are doing here is to pass the arguments via environment variables to the script. The script will abort with an error when there's an undefined argument.</span><br />
<br />
<pre>
❯ chmod +x foo.sh
❯ ./foo.sh
./foo.sh: line 3: USER: Missing the username
❯ USER=paul ./foo.sh
./foo.sh: line 4: PASS: Missing the secret password for paul
❯ echo $?
1
❯ USER=paul PASS=secret ./foo.sh
paul:secret
</pre>
<br />
<span>You have probably noticed this *strange* syntax:</span><br />
<br />
<pre>
❯ VARIABLE1=value1 VARIABLE2=value2 ./script.sh
</pre>
<br />
<span>That's just another way to pass environment variables to a script. You can write it as well as like this:</span><br />
<br />
<pre>
❯ export VARIABLE1=value1
❯ export VARIABLE2=value2
❯ ./script.sh
</pre>
<br />
<span>But the downside of it is that the variables will also be defined in your current shell environment and not just in the scripts sub-process.</span><br />
<br />
<h2 style='display: inline' id='akathenullcommand'>: aka the null command</h2><br />
<br />
<span>First, let's use the "help" Bash built-in to see what it says about the null command:</span><br />
<br />
<pre>
❯ help :
:: :
Null command.
No effect; the command does nothing.
Exit Status:
Always succeeds.
</pre>
<br />
<span>PS: IMHO, people should use the Bash help more often. It is a very useful Bash reference. Too many fallbacks to a Google search and then land on Stack Overflow. Sadly, there's no help built-in for the ZSH shell though (so even when I am using the ZSH I make use of the Bash help as most of the built-ins are compatible). </span><br />
<br />
<span>OK, back to the null command. What happens when you try to run it? As you can see, absolutely nothing. And its exit status is 0 (success):</span><br />
<br />
<pre>
❯ :
❯ echo $?
0
</pre>
<br />
<span>Why would that be useful? You can use it as a placeholder in an endless while-loop:</span><br />
<br />
<pre>
❯ while : ; do date; sleep 1; done
Sun 21 Nov 12:08:31 GMT 2021
Sun 21 Nov 12:08:32 GMT 2021
Sun 21 Nov 12:08:33 GMT 2021
^C
❯
</pre>
<br />
<span>You can also use it as a placeholder for a function body not yet fully implemented, as an empty function ill result in a syntax error:</span><br />
<br />
<pre>
❯ foo () { }
-bash: syntax error near unexpected token `}'
❯ foo () { :; }
❯ foo
❯
</pre>
<br />
<span>Or use it as a placeholder for not yet implemented conditional branches:</span><br />
<br />
<pre>
❯ if foo; then :; else echo bar; fi
</pre>
<br />
<span>Or (not recommended) as a fancy way to comment your Bash code:</span><br />
<br />
<pre>
❯ : I am a comment and have no other effect
❯ : I am a comment and result in a syntax error ()
-bash: syntax error near unexpected token `('
❯ : "I am a comment and don't result in a syntax error ()"
❯
</pre>
<br />
<span>As you can see in the previous example, the Bash still tries to interpret some syntax of all text following after ":". This can be exploited (also not recommended) like this:</span><br />
<br />
<pre>
❯ declare i=0
❯ $[ i = i + 1 ]
bash: 1: command not found...
❯ : $[ i = i + 1 ]
❯ : $[ i = i + 1 ]
❯ : $[ i = i + 1 ]
❯ echo $i
4
</pre>
<br />
<span>For these kinds of expressions it's always better to use "let" though. And you should also use $((...expression...)) instead of the old (deprecated) way $[ ...expression... ] like this example demonstrates:</span><br />
<br />
<pre>
❯ declare j=0
❯ let j=$((j + 1))
❯ let j=$((j + 1))
❯ let j=$((j + 1))
❯ let j=$((j + 1))
❯ echo $j
4
</pre>
<br />
<h2 style='display: inline' id='Nofloatingpointsupport'>(No) floating point support</h2><br />
<br />
<span>I have to give a plus-point to the ZSH here. As the ZSH supports floating point calculation, whereas the Bash doesn't:</span><br />
<br />
<pre>
❯ bash -c 'echo $(( 1/10 ))'
0
❯ zsh -c 'echo $(( 1/10 ))'
0
❯ bash -c 'echo $(( 1/10.0 ))'
bash: line 1: 1/10.0 : syntax error: invalid arithmetic operator (error token is ".0 ")
❯ zsh -c 'echo $(( 1/10.0 ))'
0.10000000000000001
❯
</pre>
<br />
<span>It would be nice to have native floating point support for the Bash too, but you don't want to use the shell for complicated calculations anyway. So it's fine that Bash doesn't have that, I guess. </span><br />
<br />
<span>In the Bash you will have to fall back to an external command like "bc" (the arbitrary precision calculator language):</span><br />
<br />
<pre>
❯ bc <<< 'scale=2; 1/10'
.10
</pre>
<br />
<span>See you later for the next post of this series.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>2021-05-16 Personal Bash coding style guide</a><br />
<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br />
<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1 (You are currently reading this)</a><br />
<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2</a><br />
<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Defensive DevOps</title>
<link href="gemini://foo.zone/gemfeed/2021-10-22-defensive-devops.gmi" />
<id>gemini://foo.zone/gemfeed/2021-10-22-defensive-devops.gmi</id>
<updated>2021-10-22T10:02:46+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I have seen many different setups and infrastructures during my carreer. My roles always included front-line ad-hoc fire fighting production issues. This often involves identifying and fixing these under time pressure, without the comfort of 2-week-long SCRUM sprints and without an exhaustive QA process. I also wrote a lot of code (Bash, Ruby, Perl, Go, and a little Java), and I followed the typical software development process, but that did not always apply to critical production issues.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>Defensive DevOps</h1><br />
<br />
<span class='quote'>Published at 2021-10-22T10:02:46+03:00</span><br />
<br />
<pre>
c=====e
H
____________ _,,_H__
(__((__((___() //| |
(__((__((___()()_____________________________________// |ACME |
(__((__((___()()()------------------------------------' |_____|
ASCII Art by Clyde Watson
</pre>
<br />
<span>I have seen many different setups and infrastructures during my carreer. My roles always included front-line ad-hoc fire fighting production issues. This often involves identifying and fixing these under time pressure, without the comfort of 2-week-long SCRUM sprints and without an exhaustive QA process. I also wrote a lot of code (Bash, Ruby, Perl, Go, and a little Java), and I followed the typical software development process, but that did not always apply to critical production issues.</span><br />
<br />
<span>Unfortunately, no system is 100% reliable, and you can never be prepared for a subset of the possible problem-space. IT infrastructures can be complex. Not even mentioning Kubernetes yet, a Microservice-based infrastructure can complicate things even further. You can take care of 99% of all potential problems by following all DevOps best practices. Those best practices are not the subject of this blog post; this post is about the sub 1% of the issues arising from nowhere you can't be prepared for. </span><br />
<br />
<span>Is there a software bug in a production, even though the software passed QA (after all, it is challenging to reproduce production behaviour in an artificial testing environment) and the software didn't show any issues running in production until a special case came up just now after it got deployed to production a week ago? Are there multiple hardware failure happening which causes loss of service redundancy or data inaccessibility? Is the automation of external customers connected to our infrastructure putting unexpectedly extra pressure on your grid, driving higher latencies and putting the SLAs at risk? You bet the solution is: Sysadmins, SREs and DevOps Engineers to the rescue. </span><br />
<br />
<span>You agree that fixing production issues this way is not proactive but rather reactive. I prefer to call it defensive, though, as you "defend" your system against a production issue. But, at the same time, you have to take a cautious (defensive) approach to fix it, as you don't want to make things worse. </span><br />
<br />
<span>Over time, I have compiled a list of fire-fighting automation strategies, which I would like to share here. </span><br />
<br />
<h2 style='display: inline'>Meet Defensive DevOps</h2><br />
<br />
<span>Defensive DevOps is a term I invented by myself. I define it this way:</span><br />
<br />
<ul>
<li>It is the practice of automating production issues away ASAP as they appear. </li>
<li>For rapid development, ignore most of the CI and QA best practices.</li>
<li>Ignore the SCRUM process (if your team does SCRUM), as it will take too long to implement a solution. </li>
<li>Be extremely careful (defensive) executing any fixing code in production, taking all failure scenarios into consideration and always have a rollback plan at hand. </li>
<li>Still deliver a high-quality solution so that no customer will ever notice that there was an issue in the first place.</li>
</ul><br />
<span>That sounds a bit crazy, but this is, unfortunately, in rare occasions the reality. As the question is not whether production issues will happen, the question is WHEN they will happen. Every large provider, such as Google, Netflix, and so on, suffered significant outages before, and I firmly believe that their engineers know what they are doing. But you can prepare for the unexpected only to a certain degree.</span><br />
<br />
<h2 style='display: inline'>Don't fully automate from the beginning</h2><br />
<br />
<span>Do you have to solve problem X? The best solution would be to fully automate it away, correct? No, the best way is to fix problem X manually first. Does the problem appear on one server or on thousand servers? The scale does not matter here. The point is that you should fix the problem at least once manually, so you understand the problem and how to solve it before implementing automation around it.</span><br />
<br />
<span>You should also have a short meeting with your team. Every person may has a different perspective and can give valuable input for determining the best strategy. But, again, keep the session short and efficient. Focus on the facts. After all, you are the domain expert and you probably know what you are doing.</span><br />
<br />
<span>Once you understand the problem, fix it on a different server again. This time maybe write a small program or script. Semi-automate the process, but don't fully automate it yet. Start the semi-automated solution manually on a couple of more servers and observe the result. You want to gain more confidence that this really solved the problem. This can take a couple of hours manually running it over and over again. During that process, you will improve your script iteratively.</span><br />
<br />
<h2 style='display: inline'>Develop code directly on production systems</h2><br />
<br />
<span>You have to develop code directly on a production system. This sounds a bit controversial, but you want to get a working solution ASAP, and there is a very high chance that you can't reproduce problem X in a development or QA environment. Or at least it will consume significant effort and time to reproduce the problem, and by the time your code is ready, it's already too late. So the most practical solution is to directly develop your solution against a production system with the problem at hand. </span><br />
<br />
<span>You might not have your full-featured IDE available on a production system, but a text editor, such as Vim (or Neovim), is sufficient for writing scripts. Some editors allow you to edit files remotely. With Vim you can accomplish it with "vim scp://SERVER///path/to/file.sh". Every time you save the file, it will be automatically uploaded via SCP to the server. From there, you can execute it directly. This comes with the additional benefits of still having access to all the Vim plugins installed locally, which you might not have installed on any production machines. This approach also removes any network delays you might experience when running your editor directly on a remote machine. </span><br />
<br />
<span>Unfortunately, it will be a bit more complicated when you rely on code reviews (e.g. in a FIPS environment). Pair-programming could be the solution here.</span><br />
<br />
<h3 style='display: inline'>Don't make it worse</h3><br />
<br />
<span>You want to triple-check that your script is not damaging your system even further. You might introduce a bug to the code, so there should always be a way to roll back any permanent change it causes. You have to program it in a defensive style:</span><br />
<br />
<ul>
<li>Make sure that all that your script does is logged to a file. Best, when it's a Bash script, use "set -x". This makes the script print all commands as they are executed. Always write the output to a file. This helps to verify that your script is working as intended. The log output should always include timestamps for each significant operation performed.</li>
<li>Make sure that no command executed by your script is failing. You should use "set -e" in your script, which makes the whole script terminate immediately if a command in it exits with a non-zero status. This will save you from apparent errors, e.g. trying to move files to a non-existent directory or trying to operate on a non-existent file. </li>
<li>Your script should never delete any files. If solving problem X involves deleting files, don't delete them but rename or move them to a separate directory so that these can be recovered just in case. </li>
<li>When you rename/move files around, always add a timestamp to a directory or the end of the file name (e.g. with "mv FILE FILE.$(date +%s"). This ensures that a backup never gets overwritten by another backup during a subsequential run of your script. Alternatively, before renaming a file, check whether the destination file already exists or not. </li>
<li>When solving problem X involves manipulating files in place, be ultra-cautious. Best try to avoid in-place file manipulation. But if you really have to, you should, if disk space permits, always create a backup of the file first. Depending on the particular case, you might add a timestamp to the backup or only keep the very first initial backup of a file.</li>
<li>You should implement a "--dry" switch in your script. When you run the script in dry mode, it won't manipulate anything on the system, but it would only print out what it would do. Always run your script in dry mode before running it for real. </li>
</ul><br />
<span>Furthermore, when you write Bash script, always run the tool ShellSheck (https://shellshock.io/) on it. This helps to catch many potential issues before applying it in production. </span><br />
<br />
<h2 style='display: inline'>Test your code</h2><br />
<br />
<span>You probably won't have time for writing unit tests. But what you can do is to pedantically test your code manually. But you have to do the testing on a production machine. So how can you test your code in production without causing more damage? </span><br />
<br />
<span>Your script should be idempotent. This means you can run it infinite times in a row, and you will always get the same result. For example, in the first run of the script, a file A get's renamed to A.backup. The second time you run the script, it attempts to do the same, but it recognises that A has already been renamed to A.backup and then it is skipping that step. This is very helpful for manually testing, as it means that you can re-run the script every time you extended it. You should dry-run the script at least once before running it for real. You can apply the same principle for almost all features you add to the code. </span><br />
<br />
<span>You may also want to inject manual negative testing into your script. For example, you want to run a particular function F in your script but only if a certain pre-condition is met, and you want to ensure that the code branching works as expected. The pre-condition check could be pretty complex (e.g. N log messages containing a specific warning string are found in the applications logs, but only on the cluster leader server). You can flip the switch directly in the code manually (e.g. run F only, when the pre-condition isn't met) and then perform a dry run of the script and study the output. Once done, flip the switch back to its correct configuration. For double insurance, test the same on a different server type (e.g. on a follower and not on a leader system).</span><br />
<br />
<span>By following these principles, you test every line of code while you are developing on it. </span><br />
<br />
<h2 style='display: inline'>Automation</h2><br />
<br />
<span>At one point, you will be tired of manually running your script and also confident enough to automate it. You could deploy it with a configuration management system such as puppet Puppet and schedule a periodic execution via cron, a systemd timer or even a separate background daemon process. You have to be extremely careful here. The more you automate, the more damage you can cause. You don't want to automate it on all servers involved at once, but you want to slowly ramp up the automation. </span><br />
<br />
<span>First, automate it only on one single server and monitor the result closely. At first, only automate running the script in dry mode. Also, don't forget that you still should log everything that the script is doing. Once everything looks fine, you can automate the script on the canary server for real. It shouldn't be a disaster if something goes wrong as usually systems are designed in a HA fashion, where the same data is still at least on another server available. In the worst-case scenario, you could recover data from there or from the local backup files your script created.</span><br />
<br />
<span>Now, you can add a handful more canary servers to the automation. You should keep close attention to what the automation is doing. You could use a tool like DTail for distributed log file following. At this point, you could also think of deploying a monitoring check (e.g. Icinga) to see whether your script is not terminating abnormally or logging warnings or errors.</span><br />
<br />
<a class='textlink' href='./2021-04-22-dtail-the-distributed-log-tail-program.html'>DTail - The distributed log tail program</a><br />
<br />
<span>From there, you could automate the solution on more and more servers. Best, ramp up the automation to a handful of systems, and later to a whole line of servers (e.g. all secondary servers of a given cluster). And afterwards, automate it on all servers.</span><br />
<br />
<span>Remember, whenever something goes wrong, you will have plenty of logs and backup files available. The disaster recovery would involve extending your script to take care of that too or writing a new script for rolling back the backups. </span><br />
<br />
<h2 style='display: inline'>Out of office hours</h2><br />
<br />
<span>If possible, don't deploy any automation shortly before out of office hours, such as in the evening, before holidays or weekends. The only exception would be that you, or someone else, will be available to monitor the automation out of office hours. If it is a critical issue, someone, for example, the on-call person, could take over. Or ask your boss to work now but to take off another day to compensate.</span><br />
<br />
<span>You should add an easy off-switch to your automation so that everyone from your team knows how to pause it if something goes wrong in order to adjust the automation accordingly. Of course, you should still follow all the principles mentioned in this blog post when making any changes. </span><br />
<br />
<h2 style='display: inline'>Retrospective</h2><br />
<br />
<span>For every major incident, you need to follow up with an incident retrospective. A blame-free, detailed description of exactly what went wrong to cause the incident, along with a list of steps to take to prevent a similar incident from occurring again in the future.</span><br />
<br />
<span>This usually means creating one or more tickets, which will be dealt with soon. Once the permanent fix is deployed, you can remove your ad-hoc automation and monitoring around it and focus on your regular work again.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>Keep it simple and stupid</title>
<link href="gemini://foo.zone/gemfeed/2021-09-12-keep-it-simple-and-stupid.gmi" />
<id>gemini://foo.zone/gemfeed/2021-09-12-keep-it-simple-and-stupid.gmi</id>
<updated>2021-09-12T09:39:20+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>A robust computer system must be kept simple and stupid (KISS). The fancier the system is, the more can break. Unfortunately, most systems tend to become complex and challenging to maintain in today's world. In the early days, so I was told, engineers understood every part of the system, but nowadays, we see more of the 'lasagna' stack. One layer or framework is built on top of another layer, and in the end, nobody has got a clue what's going on.</summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline' id='Keepitsimpleandstupid'>Keep it simple and stupid</h1><br />
<br />
<span class='quote'>Published at 2021-09-12T09:39:20+03:00; Updated at 2023-03-23</span><br />
<br />
<pre>
_______________ |*\_/*|_______
| ___________ | .-. .-. ||_/-\_|______ |
| | | | .****. .****. | | | |
| | 0 0 | | .*****.*****. | | 0 0 | |
| | - | | .*********. | | - | |
| | \___/ | | .*******. | | \___/ | |
| |___ ___| | .*****. | |___________| |
|_____|\_/|_____| .***. |_______________|
_|__|/ \|_|_.............*.............._|________|_
/ ********** \ / ********** \
/ ************ \ / ************ \
-------------------- --------------------
</pre>
<br />
<span>A robust computer system must be kept simple and stupid (KISS). The fancier the system is, the more can break. Unfortunately, most systems tend to become complex and challenging to maintain in today's world. In the early days, so I was told, engineers understood every part of the system, but nowadays, we see more of the "lasagna" stack. One layer or framework is built on top of another layer, and in the end, nobody has got a clue what's going on.</span><br />
<br />
<h1 style='display: inline' id='Needfasterhardware'>Need faster hardware</h1><br />
<br />
<span>This not just makes the system much more complex, difficult to maintain and challenging to troubleshoot, but also slow. So more experts are needed to support it. Also, newer and faster hardware is required to make it run smoothly. Often, it's so much easier to buy speedier hardware than rewrite a whole system from scratch from the bottom-up. The latter would require much more resources in the short run, but in the long run, it should pay off. Unfortunately, many project owners scare away from it as they only want to get their project done and then move on.</span><br />
<br />
<h1 style='display: inline' id='Toocomplextobereplaced'>Too complex to be replaced</h1><br />
<br />
<h2 style='display: inline' id='OnCOBOL'>On COBOL</h2><br />
<br />
<span>Have a look at COBOL, a prevalent programming language of the past. No one is learning COBOL in college or university anymore, but many legacy systems still require COBOL experts. Why is this? It's just too scary to write everything from scratch. There's too much COBOL code out there that can't be replaced from today to tomorrow. </span><br />
<br />
<a class='textlink' href='https://nymag.com/intelligencer/2020/04/what-is-cobol-what-does-it-have-to-do-with-the-coronavirus.html'>https://nymag.com/intelligencer/2020/04/what-is-cobol-what-does-it-have-to-do-with-the-coronavirus.html</a><br />
<br />
<h2 style='display: inline' id='OnKubernetes'>On Kubernetes</h2><br />
<br />
<span>Now have a look at Kubernetes (k8s), the current trendy infrastructure thing to use nowadays. Of course, there are many benefits of using k8s (auto-scaling, reproducible deployments, dynamic resource allocation and resource sharing, saving of hardware costs, good commercial for potential employees as it is the current hot sauce of infrastructure). But all of this also comes with costs: You need experts operating the k8s cluster (or you need to pay extra for a managed cluster in the cloud), increased complexity of the system (k8s comes with a steep learning curve). The latter not only applies to the engineers managing the k8s cluster - it also applies to the software engineers, who now have to develop 'cloud native' applications and, therefore, have to change how they developed software how they used to. They all need to be re-educated on what cloud-native means, and they also need to understand the key concepts of k8s for writing optimal software for it.</span><br />
<br />
<h2 style='display: inline' id='TheyoungergenerationofITprofessionals'>The younger generation of IT professionals</h2><br />
<br />
<span>Maybe the younger generation knows all of this already after graduation, but then they are missing other critical parts of the system for sure. I have seen engineers who knew about containers and how to configure resource restrictions for a Docker container managed via k8s but have never heard the terms Linux control groups and Linux namespaces. So obviously, there is some knowledge gap of the underlying architecture. This can be a big problem when you have to troubleshoot such a system during a production incident and k8s adds a lot of abstraction to the mix which doesn't make it easier. </span><br />
<br />
<span>Coming back to COBOL, k8s is on its way to becoming something similar. One day, k8s might not be the hottest tech stuff everyone wants to use. But there will be still many legacy k8s clusters around but not enough experts available to manage those:</span><br />
<br />
<a class='textlink' href='https://www.techrepublic.com/article/why-kubernetes-is-our-modern-day-cobol-says-a-tech-expert/'>https://www.techrepublic.com/article/why-kubernetes-is-our-modern-day-cobol-says-a-tech-expert/</a><br />
<br />
<span>Another article which stroke me is:</span><br />
<br />
<a class='textlink' href='https://it.slashdot.org/story/21/09/23/163212/todays-students-dont-understand-the-basics-of-computer-operations'>Today's Students Don't Understand the Basics of Computer Operations </a><br />
<br />
<span>And here is something to smile about:</span><br />
<br />
<a class='textlink' href='https://christine.website/blog/theres-a-node-2021-10-02'>https://christine.website/blog/theres-a-node-2021-10-02</a><br />
<br />
<h1 style='display: inline' id='Thebloatedweb'>The bloated web</h1><br />
<br />
<span>Another example is the modern web. Have you ever wondered why the internet becomes slower and slower nowadays? The modern web is so much like lasagna that I decided to use Gemini to be the primary protocol of my website. The HTML version of this website is just a fallback as many visitors don't know what Gemini is and don't have any compatible software installed for surfing the Geminispace:</span><br />
<br />
<a class='textlink' href='2021-04-24-welcome-to-the-geminispace.html'>2021-04-24-welcome-to-the-geminispace.html</a><br />
<br />
<span>The Gemtext protocol is KISS. There's no way to do other formattings than headings, links, paragraphs, lists, quotes, and bare text blocks (e.g., ASCII art or code snippets). There's no way to create bloated Gemini sites, and due to its limited capabilities, there's also no way to commercialise it (e.g. there's no good way to track the site visitors as things like cookies don't exist). By design, the Gemini protocol can't be extended, so there is no chance to abuse it even in the future. Gemini sites will stay KISS forever, and there won't be any fancy HTML/JavaScript frameworks like we see on the modern web.</span><br />
<br />
<h1 style='display: inline' id='Fancylogmanagementsolutions'>Fancy log-management solutions</h1><br />
<br />
<span>Yet another example I want to bring up is DTail, the distributed log tail program I wrote. There are many great and fancy log-management solutions available to choose from, and they all seem complex to set up and maintain. The ELK stack, for example, requires you to operate an ElasticSearch cluster (or multiple, if you are geo-redundant), Logstash (different configurations and instances, depending on your infrastructure) and a Kibana web-frontend (which also needs to be highly available). I have operated ElasticSearch clusters on multiple occasions, and I must say that it is not an easy task to optimise it for the particular workload you might encounter. I also have seen many ES clusters operated by other people, and I have seen these clusters failing a lot (so it's not just me). The reduced complexity of DTail also makes it more robust against outages. You won't troubleshoot your distributed application very well if the log management infrastructure isn't working either.</span><br />
<br />
<a class='textlink' href='2021-04-22-dtail-the-distributed-log-tail-program.html'>2021-04-22-dtail-the-distributed-log-tail-program.html</a><br />
<br />
<span>I don't say that the ELK stack doesn't work, but it requires experts and additional hardware resources to support it. But instead, if you keep your infrastructure simple (e.g. only use DTail), it will maintain pretty much by itself. </span><br />
<br />
<h1 style='display: inline' id='MoreKISS'>More KISS</h1><br />
<br />
<h2 style='display: inline' id='TheAdslowbePDFReader'>The Adslowbe PDF Reader</h2><br />
<br />
<span>Another perfect example is the Adobe PDF reader. How can it be that the inventor of the PDF format creates such a terrible user experience with its official reader? The reader is awful bloated, and slow. There are much better alternatives around (especially for Linux and other UNIX like operating systems, look at Zathura for example). I believe the reason Adobe's reader is like this is featuritis, and 90% of the users don't use 90% of all available features. Less is more; keep it simple and stupid. </span><br />
<br />
<h2 style='display: inline' id='Thepowerofplaintextfiles'>The power of plain text files</h2><br />
<br />
<span>Speaking of file formats, never underestimate the power of plain text files. Plain text files don't require any special software to be opened, and they outlive the software which created them in the first place. You will still be able to read a plain text file on a modern computer system ten (or twenty) years from now, but you probably won't be able to read such an old version of an Adobe Photoshop image file if the software required for reading that format isn't supported anymore and doesn't run anymore on modern computers.</span><br />
<br />
<h2 style='display: inline' id='KISSforprogrammers'>KISS for programmers</h2><br />
<br />
<span>Not to mention, keeping things simple and stupid also reduces the potential malicious attack surface. It's not just about the software and services you use and operate. It's also about the software you write. Here is a nice article about the KISS principle in software development:</span><br />
<br />
<a class='textlink' href='https://thevaluable.dev/kiss-principle-explained/'>https://thevaluable.dev/kiss-principle-explained/</a><br />
<br />
<h1 style='display: inline' id='WhenKISSisnotKISSanymore'>When KISS is not KISS anymore</h1><br />
<br />
<span>There is, however, a trap. The more you spend time with things, the more these things feel natural to you and you become an expert. The more you become an expert, the more you introduce more abstractions and other clever ways of doing things. For you, things seem to be KISS still, but another person may not be an expert and might not understand what you do. One of the fundamental challenges is to keep things really KISS. You might add abstraction upon abstraction to a system and don't even notice it until it is too late.</span><br />
<br />
<h2 style='display: inline' id='Otherrelevantreadings'>Other relevant readings</h2><br />
<br />
<a class='textlink' href='https://unixsheikh.com/articles/is-the-madness-ever-going-to-end.html'>Is the madness ever going to end?</a><br />
<a class='textlink' href='https://sive.rs/plaintext'>Write plain text files</a><br />
<br />
<span>Enough ranted for now!</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other KISS-related posts are:</span><br />
<br />
<a class='textlink' href='./2021-09-12-keep-it-simple-and-stupid.html'>2021-09-12 Keep it simple and stupid (You are currently reading this)</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br />
<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<br />
<span class='quote'>Controversially, a lack of features is a feature. Enjoy your peace an quiet. - Michael W Lucas </span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
<entry>
<title>On being Pedantic about Open-Source</title>
<link href="gemini://foo.zone/gemfeed/2021-08-01-on-being-pedantic-about-open-source.gmi" />
<id>gemini://foo.zone/gemfeed/2021-08-01-on-being-pedantic-about-open-source.gmi</id>
<updated>2021-08-01T10:37:58+03:00</updated>
<author>
<name>Paul Buetow aka snonux</name>
<email>paul@dev.buetow.org</email>
</author>
<summary>I believe that it is essential to always have free and open-source alternatives to any kind of closed-source proprietary software available to choose from. But there are a couple of points you need to take into consideration. </summary>
<content type="xhtml">
<div xmlns="http://www.w3.org/1999/xhtml">
<h1 style='display: inline'>On being Pedantic about Open-Source</h1><br />
<br />
<span class='quote'>Published at 2021-08-01T10:37:58+03:00; Updated at 2023-01-23</span><br />
<br />
<pre>
__
_____....--' .'
___...---'._ o -`(
___...---' \ .--. `\
___...---' | \ \ `|
| |o o | | |
| \___'.-`. '.
| | `---'
'^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^' LGB - Art by lgbearrd
</pre>
<br />
<span>I believe that it is essential to always have free and open-source alternatives to any kind of closed-source proprietary software available to choose from. But there are a couple of points you need to take into consideration. </span><br />
<br />
<h2 style='display: inline'>The costs of open-source</h2><br />
<br />
<span>One benefit of using open-source software is that it doesn't cost anything, right? That's correct in many cases. However, in some cases you still need to spend a significant amount of time configuring the software to work for you. It will be more expensive to use open-source software than proprietary commercial one if you aren't careful. </span><br />
<br />
<span>Not to say that I haven't seen the same effect with commercial software where people had to, after buying it, put a bunch of effort to make it work due to the lack of quality or due to high complexity. But that's either bad luck or bad decision-making. Most commercial providers I have worked with try to make it work for you, so you also will buy other products and services from them later on and don't lose you as a happy customer.</span><br />
<br />
<h2 style='display: inline'>Commercial providers</h2><br />
<br />
<span>Producers of commercial software want to earn money after all. This is to grow their businesses and also to be able to pay their employees, who also need to care for their families. Employees build up their careers, build houses, and are proud of their accomplishments in the company.</span><br />
<br />
<span>So per se, commercial software is not a bad thing. Right? At least, commercial closed-source software is not a bad thing in its heart. Unfortunately, some companies have to keep their software closed-source to not lose their competitive edge over other competitors. </span><br />
<br />
<h2 style='display: inline'>Earning on open-source</h2><br />
<br />
<span>There are also companies that earn on open-source software. All the code they write is free for download and use, but you, as a customer, could pay for service and support if you are not an expert and can't manage it by yourself. </span><br />
<br />
<span>I like this approach, as you can balance the effort and costs the way it suits you best, and in doubt, you can audit the source code. Are you already an expert? Perfect, you don't need to buy additional support for the software. Everything can be set up by yourself, given that you have the time and priority.</span><br />
<br />
<span>Also, once an open-source project reached a certain size, it is unlikely to be abandoned one day. As long as at least one person is willing to be the open-source maintainer, the project won't die. Whereas commercial providers can decide from today to tomorrow to retire software or go bankrupt (unless you purchase Microsoft Word, I don't believe it will die anytime soon). </span><br />
<br />
<h2 style='display: inline'>Open-source organizations and individual contributors</h2><br />
<br />
<span>Besides corporations, millions of individual open-source contributors write free and open-source software not for money but for pleasure. Often, they are organized in non-profit organizations, working together to reach a common goal (it is worth mentioning that there are also many professionals, payed by large corporations, working full-time for non-profit open-source projects in order to push the features and reach the goals of the corporations). Sometimes, people don't agree on the project goal, so it gets forked, which can be a good thing. The more diversity, the better, as this is where competition and innovation happens. Also, the end user will end up with more choices. </span><br />
<br />
<span>These open-source projects are of a very high quality standard and are rock-solid, if not better, alternatives to proprietary counterparts. If the project isn't backed by a large corporation already, you should donate to these open-source organizations and/or individual contributors. I have donated to some projects I use personally. Do you learn a foreign language and use Anki flashcards? It's entirely free and open-source, and they happily accept donations ensuring future maintenance and development.</span><br />
<br />
<h2 style='display: inline'>Lesser known projects and the charm of clunkiness</h2><br />
<br />
<span>Looking at the smaller, lesser-known open-source projects (not talking about established open-source projects like FreeBSD and Linux): You can't, however, expect the software to be perfect and bug-free. After all, most of the code is written for pleasure and fun in the developers' free time. Besides the developer himself, you might be the only user of the project. The software may be a bit clunky to use, and probably bugs are lurking around, and it might only work for a very specific use case.</span><br />
<br />
<span>Clunkiness can be charmful, though. And it can also encourage you to contribute code to make it better. There is a lot of such code in personal GitHub and GitLab repositories. The quality of such small open-source projects varies drastically. Many hobbyist programmers see programming as an art and put tons of effort into their projects. Others upload broken crap, which is dangerous to use. So have a look at the code before you use it!</span><br />
<br />
<h2 style='display: inline'>The security aspect</h2><br />
<br />
<span>One of the main conceptions about open-source software is that it is more secure than closed-source software because everybody can read and fix the code. Is that actually true? You can only be sure when you audit the code by yourself. If you are like me, you won't have time to audit all the open-source software you use. It's impossible to audit more than 100 million lines of Linux kernel code. Static code analysis tools come in handy here, but they still require humans to look at the results.</span><br />
<br />
<span>Security bugs in open-source projects are exposed to the public and fixed quickly, while we don't know exactly what happens to security bugs in closed-source ones. Still, hackers and security specialists can find them through reverse engineering and penetration testing. Overall, thinking of security, In my opinion it is still better to prefer open-source software because the more significant the project, the higher the probability that security bugs are found and fixed as more parties are looking into it. Furthermore, provided you have the necessary resources, you could still deduct an audit by yourself. The latter especially happens when companies with its own security and penetration testing departments are evaluating the use of open-source. This is something not every company can afford though.</span><br />
<br />
<h2 style='display: inline'>Always watch out for open-source alternatives</h2><br />
<br />
<span>Do you need Microsoft Word? Why don't you just use the Vim text editor or GNU Emacs to write your letters? If that's too nerdy, you can still use open-source alternatives such as AbiWord or LibreOffice. Larger organizations have the tendency to standardize the software their employees have to use. Unfortunately, as Microsoft Word is the de-facto standard text processing program, most companies prefer Word over LibreOffice. Same with Microsoft Excel vs LibreOffice Calc or other spreadsheet alternatives like Gnumeric. I don't know why that is; please....</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>I only use free and open-source operating systems on my personal Laptops, Desktop PCs and servers (FreeBSD and Linux based ones). Most of the programs and apps I use on them are free and open-source as well, and I am comfortable with it for over twenty years. Exceptions are the BIOSes and some firmwares of my devices. I also use Skype as most of my friends and family are using it. They are, unfortunately, proprietary software still. But I will be looking into Matrix as a Skype alternative when I have time. There are also open BIOS alternatives, but they usually don't work on my devices.</span><br />
<br />
<h2 style='display: inline'>What about mobile?</h2><br />
<br />
<span class='quote'>Update 2023-01-21: Check out my newer post about GrapheneOS, which solves some of my dilemmas</span><br />
<br />
<a class='textlink' href='./2023-01-23-why-grapheneos-rox.html'>Why GrapheneOS Rox</a><br />
<br />
<span>I struggle to go 100% open-source on my Smartphone. I use a Samsung phone with the stock Android as provided by Samsung. I love the device as it is large enough to use as a portable reading and note-taking device, and it can also take decent pictures. As a cloud backup solution, I have my own NextCloud server (open-source). Android is mainly open-source software, but many closed parts are still included. I replaced most of the standard apps with free and open-source variants from the F-Droid store though.</span><br />
<br />
<span>I could get a LineageOS based phone to get rid of the proprietary Android parts (I tried that out a couple of times in the past). But then a couple of convenient apps, such as Google Maps or Banking or Skype or the E-Ticket apps of various Airlines, various review apps when searching for restaurants, Audible (I think Audible offers an excellent service), etc., won't work anymore. The proprietary Google Maps is still the best maps app, even though there are open alternatives available. It's not that I couldn't live without these apps, but they make life a lot more convenient.</span><br />
<br />
<h2 style='display: inline'>Know the alternatives</h2><br />
<br />
<span>Thinking about alternative solutions is always a good idea. My advice is never to be entirely dependant on any proprietary software. Before you decide to use proprietary software, try to find alternatives in the open-source world. You might need to invest some time playing around with the options available. Maybe they are good enough for you, or maybe not.</span><br />
<br />
<span>If you still want to use proprietary software, use it with caution. Have a look at the recent change at Google Photos: For a long time, "high quality" photos could be uploaded there quota-less for free. However, Google recently changed the model so that people exceeding a quota have to start paying for the extra space consumed. I am not against Google's decision, but it shows you that a provider can always change its direction. So you can't entirely rely on these. I repeat myself: Don't fully rely on anything proprietary, but you might still use proprietary software or services for your own convenience.</span><br />
<br />
<h2 style='display: inline'>You can't control it all</h2><br />
<br />
<span>The biggest problem I have with going 100% open-source is actually time. You can't control all the software you use or might be using in the future. You have only a finite amount of time available in your life. So you have to decide what's more important: Investigate and use an open-source alternative of every program and app you have installed, or rather spend quality time with your family and have a nice walk in the park or go to a sports class or cook a nice meal? You can't control it all in today's world of tech, not as a user and even not as a tech worker. There's a great blog post worth reading: </span><br />
<br />
<a class='textlink' href='https://unixsheikh.com/articles/how-to-stay-sane-in-todays-world-of-tech.html'>https://unixsheikh.com/articles/how-to-stay-sane-in-todays-world-of-tech.html</a><br />
<br />
<h2 style='display: inline'>The middle way</h2><br />
<br />
<span>Regarding my personal Smartphone dilemma: I guess the middle way is to use two phones: </span><br />
<br />
<ul>
<li>Have a secondary, proprietary Android phone with Google Play store (or an Apple iPhone if this is more your thing) and all its benefits for occasional use. Use the proprietary phone only with intention. Such a phone implies some risks regarding your privacy. If you aren't careful, app providers will collect your personal data for building a digital profile of you, which gets used for online advertisement and other things. This doesn't only applies to the Smartphone, this also applies to some proprietary software (including cloud services such as Google Photos) you use on your home computer or websites you visit (I am looking at you, Facebook, Twitter and friends). Try to disable all tracking features on such a phone. It's not a guarantee that nobody will be collecting data from you anymore, but you should take at least the chance. Cal Newport once mentioned that you should not use privacy concerning apps as much anyway and instead spend more time on things which matter.</li>
<li>Have a primary phone, entirely based on free and open-source software. There will be probably no app collecting your personal data. Try to use the primary phone for all of your everyday activities and fall back to the proprietary phone only for particular use cases. Once there is decent hardware (with a decent camera) running Linux (such as Mobian, for example) available, I will consider a purchase. The only 3rd party which then will still be able to track you will be your network provider. You could start your own phone network, but that seems overkill. There is already the Pinephone and the Librem 5 running a real Linux (Android is Linux based, but it doesn't count as a real Linux for me). Still, I want to wait a bit longer for better hardware to be available (I want to have a good camera always with me).</li>
<li>You could also add a tertiary phone to the mix, which you only use for work and nothing else. That one will be very likely a proprietary phone too. You only have to keep this one around when you are working or when you are on-call.</li>
</ul><br />
<span>I have been playing with other smartphone OS alternatives, especially with MeeGo (which has died already) and SailfishOS, too. Security and privacy seem to be significantly improved compared to an Android. As a matter of fact, I bought a cheap and used Sony Xperia XA2 last year and installed SailfishOS on it. It's a nice toy, but it's still not the holy open-source grail as there are also proprietary parts in SailfishOS. Platforms such as Mobian, Ubuntu Touch and Plasma Mobile are more compelling to me. People must explore alternatives to Android and Apple here, as otherwise, you won't own any gadgets anymore:</span><br />
<br />
<a class='textlink' href='https://news.slashdot.org/story/21/07/10/0120236/by-2030-you-wont-own-any-gadgets'>https://news.slashdot.org/story/21/07/10/0120236/by-2030-you-wont-own-any-gadgets</a><br />
<br />
<span>Anyhow, any gadgets, including your phone, should be a tool you use. Don't let the phone use you!</span><br />
<br />
<h2 style='display: inline'>The downside of being a nobody</h2><br />
<br />
<span>Be aware that it might be to your disadvantage if you manage to go completely under cover without anyone collecting data from you. Suppose you are a nobody on the web (no social media profiles, no tracking history, etc.). In that case, you aren't behaving like the mass, and therefore you are suspicious. So it might be even a good thing to leave your marks here and there once in a while. You aren't hiding anything anyway, correct? Just be mindful what you are sharing about yourself. I share personal things very rarely on Facebook for example. And I only share a small subset of my personal life on my personal homepage and this blog and on all of my social media accounts. Nobody is interested in what I have for breakfast anyway I guess. Write me an E-Mail if you are interested in what I am having for breakfast.</span><br />
<br />
<h2 style='display: inline'>Mobile open-source OSes are still evolving</h2><br />
<br />
<span>You might have noticed that I wrote a lot about Smartphones in this article. The reason is that free and open-source software for Smartphones is still evolving. In contrast, for Laptops and Desktop PCs, it's already there. There is no reason to use proprietary operating systems such as Windows or macOS on your computers unless your employer forces you to use one of these. Why would they force you? It has to do with standardization again. The IT department only can manage so many platforms. It wouldn't be manageable by IT if every employee would install their own Linux distribution or one of the *BSDs. That might work for small startups but not for larger companies, especially not for a security-focused companies.</span><br />
<br />
<span>I would love a standardized Linux at work, though. Dell and Lenovo also officially support Linux on their notebooks. The culprit may be knowledgeable IT staff maintaining and giving support to the Desktop Linux users. Not all colleagues are Linux geeks like you and me. I am using macOS for work, but I am not an Apple expert. Occasionally I have to contact IT support regarding some issues I have. I don't use the macOS GUI a lot; I mainly live in the terminal so I can run the same tools I also use on Linux.</span><br />
<br />
<h2 style='display: inline'>Conclusion</h2><br />
<br />
<span>Should you be pedantic about open-source software? It depends. It depends on your fundamental values and how much time you are ready to invest. Open-source software is not just free as in money, but also free as in freedom. You will gain back complete control of your personal data. Unfortunately, installing ready proprietary apps from the Play Store is much more convenient than building up a trustworthy open-source-based infrastructure by yourself. As a guideline, use proprietary software and services with caution. Be mindful about your choices and where you leave your digital fingerprints. In doubt, think less is more. Do you really need this new shiny app? What benefit does it provide to you? Probably you don't really need that shiny new app.</span><br />
<br />
<span>You have better chances when you know how to manage your own server and install and manage alternatives to the big cloud providers by yourself. I have the advantage that I have work experience as a Linux Systems Administrator here. I mentioned NextCloud already. I use NextCloud for online photo and file storage, contact and calendar sync and as an RSS news feed server. You could do the same with your own E-Mail server, you can also host your own website and blog. I also mentioned Matrix as a Skype alternative (which could also be an alternative to WhatsApp, Skype, Telegram, Viber, ...). I don't know a lot about Matrix yet, but it seems to be a very neat alternative. I am ready to invest time in it as one of my future personal pet projects. Not only because I think it's better, but also because for fun and as a hobby. But this doesn't mean that I invest *all* of my personal free time in it.</span><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
</div>
</content>
</entry>
</feed>