Sunday, 25. September 2022
[This article has been bi-posted to Gemini and the Web]
I've been to my share of IT conferences, both as a regular attendee and as a speaker. However I've never been to any BSD conference until last weekend when I went to EuroBSDCon in Vienna. Chances are that you've heard that these conferences are different. And oh boy, they are! Do yourself a favor and go see for yourself if you've got any chance of doing so!
This is a long article with which I hope will give readers interested in this past EuroBSDCon a good overview of what that conference was like for a complete newcomer. Be my guest and join me relive some of what I experienced a week ago!
While I don't remember exactly how I found out about the newly started BSDNow! podcast (I had been a Linux user at that time and my exposure to *BSD... minimal), I started watching it basically from day 1. I still believed in the popular "BSD is dying" myth and was sure that it would be over after a couple of episodes, because "nobody uses BSD anymore", right? Well, turns out that I was wrong - and I couldn't be happier about it! Having started in late 2013, close to 500 episodes have aired since then, delivering the latest BSD news every week!
I've long since switched camps and became a FreeBSD user. While there are multiple reasons for this, the podcast has certainly helped me during the transition time (and way beyond it, mind you!). It was also there where I learned that BSD conferences did indeed exist (EuroBSDCon 2013 had been in Malta) and Allan made things sound pretty appealing. Being a BSD guy, he was certainly overselling them though, right? Wrong again! While I have had the feeling that my initial thoughts had not been exactly right for a couple of years now, I finally actually know. And the one thing that I regret is that I didn't get a chance to find out earlier.
Allan and Kris had me covered about EuroBSDCon 2014 (in Bulgaria) and 2015 (in Sweden) among other conferences. In 2016 and after using FreeBSD more and more both in private and at work, I figured that I should probably attend EuroBSDCon. In that year it took place in Serbia and I asked a Serbian co-worker if he'd like to go. He was an Ubuntu guy but enjoyed IT conferences in general, so he said yes. Then, about a week before we wanted to buy the tickets and book a hotel - his mother died... So he had to fly over there anyway, but of course any former plans were moot. It didn't feel right for me to go alone after that.
For 2017 (France) I didn't have time as we were moving houses. In 2018 (Romania), September was when our second daughter was born... So I hadn't been lucky for a while and really wanted to go in 2019, especially since the place was Lillehammer in Norway (I've learned a little Norwegian just for fun in university but never have been there). I planned for it and then... my car needed expensive repairs meaning that I was going to miss again! So when I learned that 2020 was going to be in Austria, I declared: "Now or never!". Things couldn't have been more ideal for me with the location being a neighboring, German speaking country. Then... COVID hit us all and the conference was cancelled. 2021 saw a conference but it was a virtual one. Since 2022 was finally going to be in Austria and with all the pandemic craziness, I decided that my above resolution was still valid.
So I got my leave from work, booked a hotel and got tickets for both the conference and for the (rather long) train ride. I wasn't really sure what to expect - being the new one who didn't know anybody after all. There were some pretty big names among the speakers and while for me being Mr. Nobody there totally was no chance to talk to any of them (spoiler: I've been so wrong once again!), that would at least mean there would probably be some awesome talks (finally an assumption that turned out to be right). So I figured I'd just enjoy the talks and if I'd get to strike a handful of conversations, that's be a nice plus. I'm neither shy not extremely outgoing, but not knowing how things are handled, I didn't want to leave a bad impression.
I've been rather lucky because right before the second talk something happened that ultimately proved that things were entirely different here. On Linux conferences I got used to keep a low profile because in addition to the friendly and open people there have _always_ been those that will let you know rather bluntly that you're not in their league (as if I didn't know!). I was a moment early when I got to the room, so I opened my backpack to drink something. During that moment an older man with a cane took a seat next to me (well, we all spread out due to the situation with COVID, but nobody would sit between us even though there were two or three seats). When he started a conversation like it was a completely natural thing (well, thinking about it, it _is_; but unfortunately there's this strange culture which makes us unlearn such simple things!).
Even if it hadn't been for the masks that made it a bit harder to recognize people that you only know from pictures or maybe videos, I would have checked the name on his badge. Lord almighty, I indeed found myself talking to Kirk McKusick! In case you're new to *BSD, imagine the same situation with the man being Linus Torvalds. Dr. McKusick has been around with BSD basically forever; he's the man who conceived and implemented the UFS filesystem that has been used even by commercial Unix vendors before some of them created newer ones. He was responsible for the original BSD (not *BSD!) releases at UC Berkeley, taking over when Bill Joy (who had done the early ones) left to co-found Sun Microsystems. We talked about a couple of things and being interested in Unix history, I had the opportunity to ask him about the early days of BSD. This was incredible and encouraged me to just step up to some people whose names I recognized.
One such person was Dr. Hiroki Sato. I remembered that he was responsible for creating the original TeX port in FreeBSD and I thought that I'd just thank him for it. TeXLive is _a beast_, I know that too well as I did the port for Ravenports (and it took me almost a year to feel confident enough to decide on a packaging scheme for its many, many components). What could have been a very short "Thanks for porting TeX! - You're welcome!" kind of thing turned out to become a conversation of more than 10 minutes. We discussed both TeX in FreeBSD and the use of our favorite OS in the company that I work for. He wasn't just being polite (everybody is, so this goes without saying!) but showed genuine interest in various things and eventually gave me his card. I'm going to pass on some more info to him (my boss already gave me clearance for that). Oh, did I mention, that Hiroki is one of the directors of the FreeBSD foundation? Now imagine an employee of a small and commercially basically irrelevant company who is not even a developer talking directly to a board member of the Linux foundation. Rather unlikely? Yes... But this is BSD!
There has not been a single exception for me. Whomever I talked to turned out to be perfectly approachable and very down-to-earth. What a contrast to what I had experienced so far!
My interests seem to be rather niche, even in the niche family of operating systems that are *BSD. So I ended up mostly going to talks in track 3. This had the benefit of usually being able to talk to the speakers right afterwards as most people were going to other tracks and those who went to the same would often head for the hallway instead of staying a moment longer. This has also lead to some very nice conversations which I would not want to have missed.
I didn't go to any tutorials, so for me things started on the first conference day. I got up early (at about 6), so I had some time on my hands. After arriving the day before I had already taken public transport to the Technical University once and thus knew how long it would take to get there. So I did some ports work that I had meant to do for quite some time before I left for the actual conference.
At the registration desk I picked up my badge and announced that I was a first-timer. Kristof Provost welcomed me warmly and gave me some info on how things worked in general as well as in regard to the preparations due to the COVID situation. They had FFP2 masks in the rooms where the talks took place and everybody could just grab a couple over the day (we were unfortunately meant to wear masks the whole time when not eating or drinking, but everybody was so happy to be able to meet in person again that at least I didn't hear of a single complaint). There were quick tests available, too, in case somebody didn't feel well.
I had a look around which rooms hosted the talks for all three tracks. Track 1 had the biggest one while track 3 was in a much smaller room. There were sheets of paper with arrows stuck to the walls and everything was easy to find. Then it was time for this day's keynote. I spotted Benedict Reuschling on my way there but couldn't get to him. Frank Karlitschek, founder of Owncloud and Nextcloud talked about decentralized, Open Source cloud infrastructure and why it's important. He pointed out that we're experiencing the culmination of big tech's efforts of centralization which has lead to new monopolies rising. According to him, a lot of people take for granted that messengers and other ubiquitous applications only work with centralized servers, even though that is not true at all. His main example was email - once designed to be completely decentralized and open to anyone who wants to setup another mail server, monopolization has taken its toll on that by design open system: Certain large mail providers make it hard for newcomers and e.g. simply won't accept mail from servers not run by another _big one_. Sometimes with the excuse of fighting spam, sometimes for completely arbitrary reasons. He suggested to invest more in infrastructure that follows a federated model but admitted openly that the future didn't look too bright right now. Unfortunately he's right IMO, but we've got to continue the fight nevertheless.
After the keynote I heard somebody talk to someone else and recognized the voice: I walked up to him and the badge gave away that it was indeed Tom Jones of BSDNow! Since he only joined the podcast after it went audio-only, I hadn't seen his face before. Well, until the conference, that is. We both arrived at the Technical University at the same time, I could have talked to him while we went for the registration desk. But since he obviously didn't have his badge yet, I missed that opportunity. So I went up to him, introduced myself as a regular listener of the podcast and since some of my blog posts have been covered on the show over the past few years, I mentioned the latest one. He remembered talking about it and told me that it was always great to meet listeners and people who provide the material. I didn't know that they don't actually have useful statistics - if nobody got back to them, they wouldn't actually know if anybody was listening at all! Unfortunately Tom was busy with organization and didn't have much time, but it was good to be able talk to him for a moment nevertheless.
The first talk was already giving me a hard time to decide, in the end I chose to stay for the FreeBSD one by Eirik Øverby. I wasn't sure what to think of the title, so I didn't have too high expectations but was very pleasantly surprised. It was about his company's experiences with massive DDOS attacks (Modirum was also a platinum level sponsor, BTW, thanks a lot!). If you're interested in that topics and possible countermeasures, be sure to watch the recording when it becomes available. It was a truly awesome talk (and it even had DooM references). It took me a while to remember why the name sounded somewhat familiar even though I didn't know anything about Eirik's work. I remembered right in time when I talked to him later, so I could ask him if he wasn't the guy who owned a very special certificate by everyone's favorite tech book writer Michael W. Lucas. He laughed because it was indeed him (and making Michael speechless for a moment is probably priceless indeed). If you don't know what I'm talking about but enjoy some BSD community lore, have a look here:
What the f*ck is wrong with Eirik Øverby?
During the questions after Pierre's talk on devsetup I asked about the degree of support that Pkgsrc has for foreign systems as I have had problems with bootstrapping on FreeBSD before. He said that he didn't know that platform too well and proxied the question to somebody else who was sitting in the back of the room. I found myself discussing Pkgsrc with him for a moment before I realized that I was talking to Jörg Sonnenberger, one of the NetBSD developers who is most knowledgeable about its ports system.
During lunch I met Goran who was to give a talk on FreeBSD audio on the second day. The two of us briefly interacted on a GitHub issue a couple of years ago, but hadn't ever met so far. He's definitely one of the people whom I plan on remaining in contact with. At some point I managed to talk to Benedict, too. He proved to be a bit hard to catch since they tried to do some interviews for the podcast. He was able to answer a lot of questions that I had at that point (being able to talk in German together made things easier and a little bit more efficient as I'm mostly used to writing but not actually speaking English). I promised to write a conference report - well, and here it is.
Another slot where I had trouble deciding where to go was the 4th one. It's dead certain that Allan's talk about ZFS would have a lot of good info. But then again, how often do you get the chance to learn a bit more about a less popular ISA like POWER? Also I'm on a bit of a smaller budget and have been buying used hardware for all of my machines for years (hard drives are in fact the component that I prefer to get new because it's not worth the risk, but I don't expect to be using NVMe drives too soon). Also somebody who gives a talk on OpenPOWER would probably not attract the attention of too many attendees, especially competing with two other interesting talks. So in the end I went there and got a lot of information of what the state of POWER currently is and what the future is likely to hold. I talked a while with Toshaan, too, and thus could get some more answers to my questions. I hope to support FreeBSD as a viable option on the Open Source hardware that a group of POWER enthusiasts is building:
When I decided to book my ticket I hesitated to go for the social event, too. It was another 80 euros and I didn't know what it would be. What I had heard about previous social events sounded nice, though, and fortunately I had eventually decided that if I was going to attend the conference, I might as well "go all in". I'm pretty happy that I made this decision as it turned out to be a really nice evening.
This year's social event was in fact something truly special. One of the conference's sponsors was the city of Vienna, and so we were invited into the Rathaus (town hall) for dinner! The building is beautiful and certainly a place to visit if you ever come to the city. And the interior is not too shabby, either! We went to a large room with a nicely decorated ceiling among other things and paintings of Vienna's past mayors. A politician held a short speech and delivered a word of welcome from the current mayor. There was live piano music while we were standing at tall tables and talking as well as eating.
By chance I ended up at a high table with four other German speakers. I met Jörg again and got to talk to Alexander Bluhm of the OpenBSD project as well and two other people whose names slipped my mind when writing this report a week later. We had a great discussion together about the state of various things in the three main BSDs. While the event was meant so that you'd go to other tables to get a chance to meet different people, we had enough topics to discuss (at times passionately) that we mostly stuck together at our table.
While in the main conference I focused mostly on FreeBSD, this proved to be a great counterbalance. I learned that while ZFS is back on its way in NetBSD and the effort behind it is serious, I should not expect too much at this point. On the OpenBSD side it seems that the two or three initiatives to port over HammerFS from DragonFly have stalled and lead to nothing and there isn't any modern filesystem in sight (which is rather unfortunate). We discussed version control systems and their use in the various projects. NetBSD has been working on migrating away from CVS most likely to mercurial. OpenBSD is sticking to CVS for the foreseeable future, even though Alexander thinks it could be possible that at some point _Game Of Trees_ (OpenBSD's own take at a VCS that's compatible with git repositories but has a sane interface) _might_ become an option. I described how the migration from SVN to git had worked out for FreeBSD, we talked about svnlite and the benefits of having a source control tool in base. A couple of other topics were networking, slow adoption of IPv6 and so on.
The food was excellent. There were various things to choose from, including (of course) famous Wiener Schnitzel and a bit later a variety of sweet desserts. At one time I found myself in queue for food right after Allan Jude whom I had not been able to catch during the day. I told him that I was another active listener of the podcast and appreciate their efforts a lot. We talked but for a moment and he hinted that there might be another podcast coming in the near future. I'm not going to spoil anything here, but it sounded interesting. 😉
I got to my hotel room late and slightly tired but not exhausted at all. While I hadn't been the first to leave, most people had still been there when I left. I have no idea how long it went on, but the my impression was that the atmosphere there has been great and people have been enjoying the evening as much as I have.
The keynote for the second day was Ctrl-Alt-Del by Dylan Beattie and it was excellent! He didn't talk about the well-known key combination but about legacy code and the software life cycle. Be sure to watch the recording if that's of any interest for you, he had great examples and pointed out a couple of mistakes that are extremely common in our trade.
When I had told Oleg (known for CBSD, ClonOS and more as well as member of the Advance!BSD team) that I was going to go to Vienna, he had asked me to say hi to Kirill Ponomarev, chairman of the organizing committee for him. I didn't see him on the first day and when I tried asking for him he never was around where I were, so before I'd miss him entirely, I decided to play another round of "where's Kirill?" right in the morning. To make the roles of people easier to recognize, the badges have one of three colors: Green for regular attendees, red for speakers and blue for organizers. After having had no success again, I walked up to one person with a blue badge and asked about the man and the answer was that he had been at the desk less than a minute ago but seemed to have run off again! I decided to wait and see if he'd return. When I was to give up for now and go to the first talk, he finally did. It was then when I realized that I had seen him several times already yesterday - the trouble was that he was wearing a green badge with his name written in pencil! I hadn't been able to read it yesterday but it was only green, right, so why should I care? Obviously something had happened to his blue one... 🤣
So I passed on Oleg's greetings and talked very briefly to Kirill before running off to the next talk. Speaking about which: Chiu presented on FreeBSD containers in production. He briefly compared some of the more popular jails managers, their strong points and why they didn't work out for the particular use case of the company he works for. Eventually they implemented yet another one with a focus on templating and no hard dependency on ZFS. It's written in Rust and he expects that it will be open-sourced at some point. So watch the ports tree if you're into that kind of thing (or have a look at the Fubarnetes project if you want Rust-based container and virtualization stuff right now).
I caught Nia Alarie when leaving the room and thus another person I wanted to talk to. In December I had published a comparison between Pkgsrc and Ravenports - and Nia had commented a bit harshly on that initially and then not answered again. I wasn't resentful but wanted to take the chance to come to good terms which is always much easier in person than on some Reddit thread with a guy that makes seemingly weird claims (me). Indeed I learned more about what lead to the not too friendly reaction and we also talked about audio stuff and BSD vs. Linux before it was time for the next talk.
Due to a slot change I missed Philipp Buehler's talk on Jitsi hosted on OpenBSD that I wanted to go to and thus will have to watch the recording. I went to Luca Pizzamiglio's talk instead for some more jails goodness. He presented a quite different view on the topic, concentrating more on the "container" approach with single process jails that don't run rc. His jails manager, pot, makes heavy use of those and can be used together with Hashi's Nomad container orchestration engine (and Consul for service discovery). The impressive result: You can have Kubernetes-like workloads on FreeBSD! Luca also mentioned that his manager was originally meant as a proof of concept and for that reason he chose to implement it in shell. However this choice turns out to be problematic now and the tool would need to be rewritten in another language to take it further. I met him later in the hallway track and we talked a little. One more person that I'll be sure to keep in contact with!
After lunch I got to talk to Corvin Köhne for a moment after having been to his talk on GPU passthrough with Bhyve the day before. Cool stuff that Beckhoff is working on, and I'm glad that this company decided not to quietly start using FreeBSD but to get involved in the community instead. This is the way (and something that hardcore GPL folks claim doesn't happen in reality).
Goran Mekić's talk on FreeBSD audio was pretty interesting, too. I knew that it's in better shape than audio on Linux, but not much more. He talked about the various sound servers available and about tweaks that you might or might not want to use in professional setups. Indeed FreeBSD's audio capabilities were one of the things that were the reason for him to come to FreeBSD: He had his daily driver Linux machine with fast SSDs and an older one with HDDs that he used for tinkering with things. When FreeBSD on the slow machine beat Linux on the fast one for his first test, he knew that he'd have to take a much closer look! Obviously he found it to be well fit for his use cases and also very enjoyable in general.
For the last talk I was undecided between Albert Dengg's and Michael Dexter's. I don't know Albert but I had seen Michael on video at least once before. While his talk's title was kind of weird it was probably that which won me over in the end. It's hard to compare as I haven't watched the recording of the other one, yet, but "The FreeBSD build option, OpenZFS, bhyve, compat_linux, and jail.conf.d nexus" proved to be a very inspiring talk that I enjoyed a lot and a great one to conclude a fantastic conference. Thanks, Michael, for conveying so much passion, awareness for things that are not quite ideal and at the same time the hope that they could eventually be done right! You provided the booster that I needed on my way home to counter the sadness that it was already over.
On my way to the closing session I met Martin Matuška who is the author of mfsBSD among other things and we had a short conversation. Then we were shown some statistics (number of attendees, countries of origin, ...), Benedict passed on Groff the BSD goat to Tom so that our favorite plush toy can travel to Scotland and there was a charity auction. Then finally we got the info where next year's conference will be. And finally with a sad voice, Henning Brauer had to announce that that was it!
EuroBSDCon 2022 was not just another conference for me, it _was_ different. What did it feel like? Let me put it this way: I've found myself _a second family_. Not, I'm not exaggerating! Now I know for myself what Allan once expressed in his words as: "These are my people". I'm grateful that I was finally being able to attend. It was so worth to do it despite not knowing whether things would work in September due to the COVID situation that could change any time.
There's exactly one downside for me: Now I've got to explain to my wife, why I have to go to _Portugal_ next year in September (if I can afford it)! 😉