💾 Archived View for vierkantor.com › xukut captured on 2024-08-31 at 11:24:43. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-05-10)
-=-=-=-=-=-=-
XukutOS is a hobbyist operating system taking a lot of design inspiration from the Lisp machines. It has a prototype implementation for x86 computers (works well under QEMU/DOSBox) and right now I'm working on an improved version for AArch64 (it can boot on my Raspberry Pi 4).
Well, the codename for this project was Lisp-Ish Machine Attempt, LIMA. Translate "lima" from Spanish to Totonacan, and you get Xukut. Moreover, there's an X in the name, which all operating systems (Unix, Linux, Mac OS X, Windows XP) have got for some reason.
If some of the following things sound cool to you, XukutOS might be for you:
* high level language computer architecture
* customizable operating systems
* spawning tons of processes
* decentral, understandable design
* you'll learn a lot about operating systems development because you will have to fix or implement many things
See the following links:
XukutOS is my personal project at the moment, but anyone who is interested is very welcome to use it or help with development! Contact me:
By email at vierkantor@vierkantor.com
Through mastodon @vierkantor@mastodon.vierkantor.com
This page collects XukutOS news.
As a birthday gift to myself, I took some time to implement continuations in Swail, in order to implement restarts in the condition system.
XukutOS now includes a compiler (aarch64 only, but so is XukutOS itself!) that can take existing `swail:fn's and turn them into (hopefully) more efficient machine code, using an aarch64 assembler that is also new. The compiler isn't tested very well yet. The parts that are currently confirmed to seem to work are functions, variables and apply so it should be Turing complete already!
You can simply call the function `aarch64:compile' on any callable function and it will return a compiled version. (Or the original function in case the compiler doesn't yet support compiling this function.)
Lexical and dynamic variables are now stored in specialized data structures. This should make it possible for the implementation to be made much more efficient than the current linear scan through the entire environment.
More documentation on variables in Swail
The `swail:fn' special form now allows defining functions with dotted parameter lists. If a parameter occurs in the tail position of a list, this will collect a list of all remaining arguments. For example, `((swail:fn (x . y) (list x y)) 1 2 3)' evaluates to `(1 (2 3))', since `x' gets assigned to `1' and `y' to `(2 3)'.
More documentation on functions and argument parsing in Swail
Implemented a hashtable data structure, in the hope that this can speed up symbol lookup in the editor, and perhaps variable lookup in the interpreter.
Made `cast' a generic function, which requires some subtle changes in the metamethod protocol to ensure `if' doesn't cause an infinite loop by trying to cast its element to `bool'.
Implemented a few text operations: `intercalate', `replace' and `remove', plus tests, plus small refactors and fixes in related code.
Implemented functionality to save and load data, by communicating with a ``mothership'' process over a UART link.
The serialization protocol used for communication over the UART link.
Implemented a UART driver, that can succesfully echo incoming bytes back over the UART.
https://mastodon.vierkantor.com/@Vierkantor/107340232965570004
I implemented macros in the interpreter.
I defined a metamethod protocol that allows for an extensible way to implement methods.
Replaced the list of builtin processes with only the garbage collector and an init process which runs in the interpreter. That should make it feasible to define Swail in terms of Swail, by running this code in the init process before the other processes start.
I changed the stack size for a new process from 8KB to 8MB, since apparently I misinterpreted the `8192` output by `ulimit -s' on my system.
Now the editor has enough stack space to produce a list of suggestions for which symbol you want to name. They are stored in a BK tree, and retrieval is not too slow (even though generating the tree is painfully slow).
https://mastodon.vierkantor.com/@Vierkantor/106659954073234682
Some more bugfixing in the new calling convention later, looks like we can run for a while until we get out of memory. And after re-enabling the garbage collector, we can run for 2.3GB of allocation before it can't keep up (on a 1GB virtual machine).
https://mastodon.vierkantor.com/@Vierkantor/106631011731799266
Implemented a debugger process: whenever a process encounters an unhandled condition, it asks the debugger what to do. In this case, the debugger says "pop up a window describing the issue, then stop running."
https://mastodon.vierkantor.com/@Vierkantor/106388112763622922
Hooked up a new window manager, process spawning and evaluation to the editor. When you press the enter key, your expression gets evaluated and a new window pops up with the result!
https://mastodon.vierkantor.com/@Vierkantor/106371706770029306
Refactored the interpreter to use OARCC in the interpreter. This has reduced the amount of stack overflows by quite a bit!
https://mastodon.vierkantor.com/@Vierkantor/106206602390410042
Today I designed the new OARCC calling convention that supports tail recursion a whole lot better.
https://mastodon.vierkantor.com/@Vierkantor/106144625194626454
The garbage collector is now able to recycle freed pages! After fixing a bug (freeing pages was non-atomic), systems can actually run longer than a couple of minutes without crashing!
https://mastodon.vierkantor.com/@Vierkantor/106085986498945475
My code was running out of memory within one minute, so I had to enable the garbage collector. This also meant a whole lot of bugfixing in the garbage collector, which seems to have succeeded now.
https://mastodon.vierkantor.com/@Vierkantor/106065791769941768
Fixed a bug when handling nested interrupts: when the kernel handled an interrupt it saved registers to a fixed location. This meant that nested interrupts would overwrite the previously saved registers. Also a nested interrupt would re-use the object stack.
https://mastodon.vierkantor.com/@Vierkantor/106026061984733646
The expression editor can now display S-expressions!
https://mastodon.vierkantor.com/@Vierkantor/106012377917144133
I built a new scheduler for XukutOS! Now processes can sleep for a specified time, allowing other processes to run in the meantime. This should make it possible to create a USB keyboard/mouse driver that doesn't consume your whole CPU time as it polls for updates.
https://mastodon.vierkantor.com/@Vierkantor/105882558948976808
Here are a couple of new features added to XukutOS since the last update:
* drivers: the `usb' process serves as a very basic interface with the DesignWare Hi-Speed USB 2.0 On-The-Go Controller (as built into (most?) Raspberry Pi models), with the ability to make a list of attached devices (it doesn't yet do anything else than get their device type)
* kernel: a new memory allocator which is actually concurrent (has not been tested with multiple processors, but it does work with multiple processes)
* swail: started implementing generic functions and methods (Generic functions can already select the correct method for a call. This is not tested well yet.)
* everywhere: many small bugfixes
https://mastodon.vierkantor.com/@Vierkantor/105137608389359937
The codename for this project was Lisp-Ish Machine Attempt, LIMA. Translate "lima" from Spanish to Totonacan, and you get Xukut. Moreover, there's an X in the name, which all operating systems (Unix, Linux, Mac OS X, Windows XP) have got for some reason.
It was called Lisp-Ish Machine because it takes design inspiration from Lisp Machines. Namely, the OS should have a high-level language as its main interface and its main building block. At the moment, all the kernel code is written in assembly, and it will probably stay that way, but most processes are in Lisp. (The Lisp is mostly interpreted, with some parts hand-compiled. Once it's well-featured, I plan on bootstrapping a compiler.)
The kernel doesn't do much more than handle syscalls/interrupts, organise process state and twiddle semaphores. In fact, it's designed in such a way that processes don't notice at all when it's running. The trick is called PCLSRing: whenever a kernel operation would block, set up process state so that it retries at the end of the waiting time.
Apart from the kernel, we have many processes. The main inspiration for these is Erlang. The `send' and `receive' primitives could be called directly, but typically you do this via dynamically bound functions. This allows you to better control effects in your dependencies (e.g. the functions that you call).
So you can say:
(let (*send* (append-arglist-and! syscall:send! sent-messages)) (function-calling-send-that-needs-to-be-tested x y))
And if you have a new `pid' type which allows for forwarding, then you can do:
(set! *send* (λ (pid msg)
(if (is-fancy-pid? pid)
(fancy-send pid msg)
(*send* pid msg))))
Messages are Lisp objects and because processes share an address space, they are stored as just a pointer. (In the prototype, they were byte arrays copied at each privilege boundary, sender->kernel->receiver: lots of alloc!)
The kernel does not enforce any structure in messages sent between processes, but it sends, and the built-in processes expect, a list of the format
(,msg-id ,version ,nonce ,sender ,@rest)
msg-id is a number/symbol that identifies this message. e.g. the kernel sends msg-id 1 to notify processes when an interrupt occurs.
`version' is an incrementing number per msg-id. A higher version implies backward compatibility with lower versions. Specifically: `(index msg n)' gives a sensible answer for all versions `w ≥ v' iff it does for version `v'. A higher version number does not imply the structure changes: the sender can also use it to indicate that it can (alternatively) handle a different kind of response.
One of the goals of XukutOS is decentralised design: no decisions like "how to assign meaning to msg-id" should need to be made by committee. Now you should be saying: hang on, what if *I* want to send a message with `msg-id' = 1? Won't the process get confused with the kernel's messages?
This is where the nonce comes in. If the nonce is 0, the receiver should specify what the meaning of msg-id is. Otherwise the nonce is specified by the receiver and the meaning of msg-id by the sender.
It's no use to send a message to a receiver without knowing its protocol, so nonce=0, msg-id defined by receiver is the only option. For replies, the sender can add a unique nonce, and by matching the nonces, it knows the context and thus the meaning.
Some other assorted facts / design notes:
* Technical decisions are made foremost based on "will I enjoy developing this?". It's just a hobby, won't be big and professional like GNU/Linux.
* Code is written in Dutch, because I enjoy coming up with good translations for techincal terms.
* Probably won't directly support Unicode (see "decentralise/no committee").
* A correct `quote' operator needs to consider variable bindings, analogous to the changes to `lambda' since LISP 1.5.
https://mastodon.vierkantor.com/@Vierkantor/105187953910593931
* protect memory allocation with a semaphore, making it somewhat more multiprocess friendly (TODO: separate regions per-process so it doesn't hang when you do certain syscalls while in the protected section)
* add missing `pop' instruction which caused a kernel thread to jump back to boot-time initialization, making it very confused about what was going on
* ensure all compiled-in memory regions are marked as such
https://mastodon.vierkantor.com/@Vierkantor/105277873089387672
This week I built a font and a simple text renderer, and used it to implement "useful" kernel panics.
Next step is to install a simple ring buffer for writing debug info to.
... Writing to the buffer works :D (although I just realized that my hexadecimal numbers are being printed backwards)
