💾 Archived View for her.esy.fun › gem-atom.xml captured on 2023-03-20 at 17:46:17.
⬅️ Previous capture (2023-01-29)
-=-=-=-=-=-=-
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
xmlns:georss="http://www.georss.org/georss"
xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#"
xmlns:media="http://search.yahoo.com/mrss/"><channel>
<title>her.esy.fun</title>
<atom:link href="gemini://her.esy.fun/gem-atom.xml" rel="self" type="application/rss+xml" />
<link>gemini://her.esy.fun</link>
<description><![CDATA[her.esy.fun articles, mostly random personal thoughts]]></description>
<language>en</language>
<pubDate>Thu, 02 Mar 2023 00:00:00 +0100</pubDate>
<lastBuildDate>Thu, 02 Mar 2023 00:00:00 +0100</lastBuildDate>
<generator>mkrss.sh</generator>
<webMaster>yann@esposito.host (Yann Esposito)</webMaster>
<image>
<url>gemini://her.esy.fun/img/FlatAvatar.png</url>
<title>her.esy.fun</title>
<link>gemini://her.esy.fun</link>
</image>
<item>
<title>Replace docker-compose with nix-shell</title>
<guid>gemini://her.esy.fun/posts/0024-replace-docker-compose-with-nix-shell/index.gmi</guid>
<pubDate>Thu, 02 Mar 2023 00:00:00 +0100</pubDate>
<category>blog</category>
<category>static</category>
<description><![CDATA[
# Replace docker-compose with nix-shell
description: This is how I created a docker-compose replacement with nix-shell.
description: Here is a solution to have a composable nix shell representation focused on
description: replacing docker-compose.
keywords: blog static
author: Yann Esposito
email: yann@esposito.host
=> /files/publickey.txt gpg
date: [2023-03-02 Thu]
At work we use =docker-compose= to run integration tests on a big project that
need to connect to multiple different databases as well as a few other services.
This article is about how to replace =docker-compose= by =nix= for a local
development environment.
## Quick tutorial
### =nix-shell-fu= level 1 lesson
Let's start with a basic =shell.nix= example:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
with pkgs: mkShell
{ buildInputs = [ hello ];
shellHook = ''
echo "Using ${hello.name}."
'';
}
And this could be understood in plain English as: ______ In the packages of nix version 22.11, create a new shell into which the package =hello= will be installed. At the end of the install, run a script that will print the package name. (Cf => digression) ______ If you copy/paste this in a =shell.nix= file and run ~nix-shell~ you get:
nix-shell
nix-shell shell.nix
these 53 paths will be fetched (84.69 MiB download, 524.77 MiB unpacked):
/nix/store/08pckaqznwh0s3822cjp5aji6y1lsm27-libcxx-11.1.0
...
/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13
copying path '/nix/store/49wn01k9yikhjlxc1ym5b6civ29zz3gv-bash-5.1-p16' from 'https://cache.nixos.org'...
...
copying path '/nix/store/4w2rv6s96fwsb4qyw8b9w394010gxriz-stdenv-darwin' from 'https://cache.nixos.org'...
Using hello-2.12.1.
[nix-shell:~/tmp/nixplayground]$
If you close the session and run it again, it will be much faster and will only show this:
❯ nix-shell
Using hello-2.12.1.
[nix-shell:~/tmp/nixplayground]$
This is because all dependencies will be cached. OK so, this is level 1 of /nix-shell-fu/. Now, let's start level 2. ### =nix-shell-fu= level 2 lesson; scripting and configuring This time, we want to launch a full service, as a redis docker would do. So here is a basic shell script which is similar to the previous one but will request =redis= as a dependency instead of =hello= and also as a launching script. From there will add a little bit more features.
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
pkgs.mkShell {
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port: ${port}"
redis-server
'';
}
Again if you run ~nix-shell~ here is the result:
❯ nix-shell
these 2 paths will be fetched (2.08 MiB download, 6.99 MiB unpacked):
/nix/store/6w4vnaxdx12ccq172i8j5l830mlp8jlg-redis-7.0.5
/nix/store/b47gmsx9qx0c9vh75wsg8bqq9qd0ad6f-openssl-3.0.7
copying path '/nix/store/b47gmsx9qx0c9vh75wsg8bqq9qd0ad6f-openssl-3.0.7' from 'https://cache.nixos.org'...
copying path '/nix/store/6w4vnaxdx12ccq172i8j5l830mlp8jlg-redis-7.0.5' from 'https://cache.nixos.org'...
Using redis-7.0.5
97814:C 10 Feb 2023 20:44:36.960 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
97814:C 10 Feb 2023 20:44:36.960 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=97814, just started
97814:C 10 Feb 2023 20:44:36.960 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
97814:M 10 Feb 2023 20:44:36.961 * Increased maximum number of open files to 10032 (it was originally set to 256).
97814:M 10 Feb 2023 20:44:36.961 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 7.0.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 97814
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
97814:M 10 Feb 2023 20:44:36.962 # WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.
97814:M 10 Feb 2023 20:44:36.962 # Server initialized
97814:M 10 Feb 2023 20:44:36.963 * Ready to accept connections
Woo! Redis is started and it works! But if you have multiple projects you want to have more control. For example, we will want to run redis on a specific port. Here is how you do it:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/21.05.tar.gz) {} }:
let iport = 16380;
port = toString iport;
in pkgs.mkShell {
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port ${port}"
redis-server --port ${port}
'';
}
And here is the result:
rm dump.rdb
nix-shell
Using redis-6.2.3 on port 16380
1785:C 10 Feb 2023 20:50:00.880 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1785:C 10 Feb 2023 20:50:00.880 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=1785, just started
1785:C 10 Feb 2023 20:50:00.880 # Configuration loaded
1785:M 10 Feb 2023 20:50:00.880 * Increased maximum number of open files to 10032 (it was originally set to 256).
1785:M 10 Feb 2023 20:50:00.880 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.3 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 16380
| `-._ `._ / _.-' | PID: 1785
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1785:M 10 Feb 2023 20:50:00.881 # Server initialized
1785:M 10 Feb 2023 20:50:00.881 * Ready to accept connections
Woo! We control the port from the file. That's nice. But, has you might have noticed, when you quit the session it dumps the DB as the file =dump.rdb=. What we would like is to keep all the state in a local directory that would be easy to delete. To achieve this, instead of passing argument to the redis command line we will use a local config file to use.
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let iport = 16380;
port = toString iport;
in pkgs.mkShell (rec {
# ENV Variables the directory to put all the DATA
REDIS_DATA = "${toString ./.}/.redis";
# the config file, as we use REDIS_DATA variable we just declared in the
# same nix set, we need to use rec
redisConf = pkgs.writeText "redis.conf"
''
port ${port}
dbfilename redis.db
dir ${REDIS_DATA}
'';
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port: ${port}"
[ ! -d $REDIS_DATA ] \
&& mkdir -p $REDIS_DATA
cat "$redisConf" > $REDIS_DATA/redis.conf
alias redisstop="echo 'Stopping Redis'; redis-cli -p ${port} shutdown; rm -rf $REDIS_DATA"
nohup redis-server $REDIS_DATA/redis.conf > /dev/null 2>&1 &
echo "When finished just run redisstop && exit"
trap redisstop EXIT
'';
})
And here is a full session using this =shell.nix=:
nix-shell
Using redis-6.2.3 on port: 16380
When finished just run redisstop && exit
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redis-cli -p 16380
127.0.0.1:16380> help
redis-cli 6.2.3
To get help about Redis commands type:
"help @<group>" to get a list of commands in <group>
"help <command>" for help on <command>
"help <tab>" to get a list of possible help topics
"quit" to exit
To set redis-cli preferences:
":set hints" enable online hints
":set nohints" disable online hints
Set your preferences in ~/.redisclirc
127.0.0.1:16380>
-------------------------------
[nix-shell:~/tmp/nixplayground]$ ls -a
. .. .redis shell.nix
-------------------------------
[nix-shell:~/tmp/nixplayground]$ find .redis
.redis
.redis/redis.conf
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redis-cli -p 16380 shutdown
[1]+ Done nohup redis-server $REDIS_DATA/redis.conf > /dev/null 2>&1
-------------------------------
[nix-shell:~/tmp/nixplayground]$ find .redis
.redis
.redis/redis.db
.redis/redis.conf
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redisstop
Stopping Redis
Could not connect to Redis at 127.0.0.1:16380: Connection refused
-------------------------------
[nix-shell:~/tmp/nixplayground]$ ls -a
. .. shell.nix
So with this version all data related to redis is saved into the local =.redis= directory. And in the nix shell we provide a command =redisstop= that once invoked, shutdown redis, then purge all redis related data (as you would like in a development environment). Also, as compared to previous version, redis is launched in background so you could run commands in your nix shell. Notice I also run ~redisstop~ command on exit of the nix-shell. So when you close the nix-shell redis is stopped and the DB state is cleaned up. ## Composable =nix-shell= As a quick recap you now have a boilerplate to create new =shell.nix=:
{ pkgs ? import ( ... ) {} }:
mkShell { MY_ENV_VAR_1 = ...;
MY_ENV_VAR_2 = ...;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' command_to_run_after_init '';
}
But if I give you two such =shell.nix= files, would you be able to compose them? Unfortunately, not directly. To solve the problem we will replace this boilerplate by another one that do not directly uses =mkShell=. And in order to make it fully composable, we will also need to narrow the environment variables declaration in a sub field:
{ pkgs ? import ( ... ) {} }:
let env = { PGDATA = ...; }
in { inherit env; # equivalent to env = env;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' some_command $PG_DATA '';
}
With this, we can compose two nix set into a single merged one that will be suitable to pass as argument to ~mkShell~. Another minor detail, but important one. In bash, the command ~trap~ do not accumulate but replace the function. For our need, we want to run all stop function on exit. So the ~trap~ directive added in the shell hook does not compose naturally. This is why we add a =stop= value that will contain the name of the bash function to call to stop and cleanup a service. Finally the main structure for each of our service will look like this *nix service boilerplate*:
{ pkgs ? import ( ... ) {} }:
let env = { MY_SERVICE_ENV_VAR = ...; }
in { inherit env; # equivalent to env = env;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' my_command $MY_SERVICE_ENV_VAR '';
stop = "stop_my_service"
}
So let's start easy. To run a single shell script like this with =nix-shell=, you should put your service specific nix file in a =service.nix= file and create a =shell.nix= file that contains something like:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let service = import ./service.nix { inherit pkgs; };
in with service; pkgs.mkShell ( env //
{
buildInputs = buildInputs;
nativeBuildInputs = nativeBuildInputs ;
shellHook = shellHook;
})
Now, if you would like to run nix shell for multiple files, here is a first qui solution:
{ pkgs ? import (...) {}}:
let
# merge all the env sets
mergedEnvs = builtins.foldl' (acc: e: acc // e) {} envs;
# merge all the confs by accumulating the dependencies
# and concatenating the shell hooks.
mergedConfs =
builtins.foldl'
(acc: {buildInputs ? [], nativeBuildInputs ? [], shellHook ? "", ...}:
{ buildInputs = acc.buildInputs ++ buildInputs;
nativeBuildInputs = acc.nativeBuildInputs ++ nativeBuildInputs;
shellHook = acc.shellHook + shellHook;
})
emptyConf
confs;
in mkShell (mergedEnvs // mergedConfs)
And now, here is the full solution that also deal with other minor details like importing the files and dealing with the exit of the shell:
{ mergeShellConfs =
# imports should contain a list of nix files
{ pkgs, imports }:
let confs = map (f: import f { inherit pkgs; }) imports;
envs = map ({env ? {}, ...}: env) confs;
# list the name of a command to stop a service (if none provided just use ':' which mean noop)
stops = map ({stop ? ":", ...}: stop) confs;
# we want to stop all services on exit
stopCmd = builtins.concatStringsSep " && " stops;
# we would like to add a shellHook to cleanup the service that will call
# all cleaning-up function declared in sub-shells
lastConf =
{ shellHook = ''
stopall() { ${stopCmd}; }
echo "You can manually stop all services by calling stopall"
trap stopall EXIT
'';
};
# merge Environment variables needed for other shell environments
mergedEnvs = builtins.foldl' (acc: e: acc // e) {} envs;
# zeroConf is the minimal empty configuration needed
zeroConf = {buildInputs = []; nativeBuildInputs = []; shellHook="";};
# merge all confs by appending buildInputs and nativeBuildInputs
# and by concatenating the shellHooks
mergedConfs =
builtins.foldl'
(acc: {buildInputs ? [], nativeBuildInputs ? [], shellHook ? "", ...}:
{ buildInputs = acc.buildInputs ++ buildInputs;
nativeBuildInputs = acc.nativeBuildInputs ++ nativeBuildInputs;
shellHook = acc.shellHook + shellHook;
})
zeroConf
(confs ++ [lastConf]);
in (mergedEnvs // mergedConfs);
}
So I put this function declaration in a file named =./nix/merge-shell.nix=. And I have a =pg.nix= as well as a =redis.nix= file in the =nix= directory. On the root of the project the main =shell.nix= looks like:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let
# we import the file, and rename the function mergeShellConfs as mergeShells
mergeShells = (import ./nix/merge-shell.nix).mergeShellConfs;
# we call mergeShells
mergedShellConfs =
mergeShells { inherit pkgs;
# imports = [ ./nix/pg.nix ./nix/redis.nix ];
imports = [ ./nix/pg.nix ./nix/redis.nix ];
};
in pkgs.mkShell mergedShellConfs
And, that's it. Now when I run =nix-shell= it launch both Postgresql and Redis, and when I quit the shell, the state is cleaned up. Both postgres and redis are shutdown and the local files are erased. I hope this could be useful to someone else. ## Appendix ### <<digression>> Digression In fact, this is a bit more complex than "just that". The reality is a bit more complex. The nix language is "pure", meaning, if you run the nix evaluation multiple times, it will always evaluate to the exact same value. But here, this block represent a function. The function takes as input a "nix set" (which you can see as an associative array, or a hash-map or also a javascript object depending on your preference), and this set is expected to contain a field named =pkgs=. If =pkgs= is not provided, it will use the set from the stable version 22.11 of nixpkgs by downloading them from github archive. The second part of the function generate "something" that is returned by an internal function of the standard library provided by =nix= which is named =mkShell=. So mainly, =mkShell= is a helper function that will generate what nix calls a / => https://blog.ielliott.io/nix-docs/derivation.html derivation /. Mainly, we don't really care about exactly what is a /derivation/. This is an internal to nix representation that could be finally used by different nix tools for different things. Typically, installing a package, running a local development environment with nix-shell or nix develop, etc… So the important detail to remember is that we can manipulate the parameter we pass to the functions =derivation=, =mkDerivation= and =mkShell=, but we have no mechanism to manipulate directly =derivation=. So in order to make that composable, you need to call the =derivation= internal function at the very end only. The argument of all these functions are /nix sets/ ### The full nix files for postgres For postgres:
{ pkgs }:
let iport = 15432;
port = toString iport;
pguser = "pguser";
pgpass = "pgpass";
pgdb = "iroh";
# env should contain all variable you need to configure correctly mkShell
# so ENV_VAR, but also any other kind of variables.
env = {
postgresConf =
pkgs.writeText "postgresql.conf"
''
# Add Custom Settings
log_min_messages = warning
log_min_error_statement = error
log_min_duration_statement = 100 # ms
log_connections = on
log_disconnections = on
log_duration = on
#log_line_prefix = '[] '
log_timezone = 'UTC'
log_statement = 'all'
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
logging_collector = on
log_min_error_statement = error
'';
postgresInitScript =
pkgs.writeText "init.sql"
''
CREATE DATABASE ${pgdb};
CREATE USER ${pguser} WITH ENCRYPTED PASSWORD '${pgpass}';
GRANT ALL PRIVILEGES ON DATABASE ${pgdb} TO ${pguser};
'';
PGDATA = "${toString ./.}/.pg";
};
in env // {
# Warning if you add an attribute like an ENV VAR you must do it via env.
inherit env;
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.coreutils
pkgs.jdk11
pkgs.lsof
pkgs.plantuml
pkgs.leiningen
];
nativeBuildInputs = [
pkgs.zsh
pkgs.vim
pkgs.nixpkgs-fmt
pkgs.postgresql_11
# postgres-11 with postgis support
# (pkgs.postgresql_11.withPackages (p: [ p.postgis ]))
];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.postgresql_11.name}. port: ${port} user: ${pguser} pass: ${pgpass}"
# Setup: other env variables
export PGHOST="$PGDATA"
# Setup: DB
[ ! -d $PGDATA ] \
&& pg_ctl initdb -o "-U postgres" \
&& cat "$postgresConf" >> $PGDATA/postgresql.conf
pg_ctl -o "-p ${port} -k $PGDATA" start
echo "Creating DB and User"
psql -U postgres -p ${port} -f $postgresInitScript
function pgstop {
echo "Stopping and Cleaning up Postgres";
pg_ctl stop && rm -rf $PGDATA
}
alias pg="psql -p ${port} -U postgres"
echo "Send SQL commands with pg"
trap pgstop EXIT
'';
stop = "pgstop";
}
And to just launch Posgresql, there is also this file =./nix/pgshell.nix=, that simply contains
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let pg = import ./pg.nix { inherit pkgs; };
in with pg; pkgs.mkShell ( env //
{
buildInputs = buildInputs;
nativeBuildInputs = nativeBuildInputs ;
shellHook = shellHook;
})
=> /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> <item> <title>Replace docker-compose with nix-shell</title> <guid>gemini://her.esy.fun/posts/0024-replace-docker-compose-with-nix-shell/flycheck_index.gmi</guid> <pubDate>Thu, 02 Mar 2023 00:00:00 +0100</pubDate> <category>blog</category> <category>static</category> <description><![CDATA[ # Replace docker-compose with nix-shell description: This is how I created a docker-compose replacement with nix-shell. description: Here is a solution to have a composable nix shell representation focused on description: replacing docker-compose. keywords: blog static author: Yann Esposito email: yann@esposito.host => /files/publickey.txt gpg date: [2023-03-02 Thu] At work we use =docker-compose= to run integration tests on a big project that need to connect to multiple different databases as well as a few other services. This article is about how to replace =docker-compose= by =nix= for a local development environment. ## Quick tutorial ### =nix-shell-fu= level 1 lesson Let's start with a basic =shell.nix= example:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
with pkgs: mkShell
{ buildInputs = [ hello ];
shellHook = ''
echo "Using ${hello.name}."
'';
}
And this could be understood in plain English as: ______ In the packages of nix version 22.11, create a new shell into which the package =hello= will be installed. At the end of the install, run a script that will print the package name. (Cf => digression) ______ If you copy/paste this in a =shell.nix= file and run ~nix-shell~ you get:
nix-shell
nix-shell shell.nix
these 53 paths will be fetched (84.69 MiB download, 524.77 MiB unpacked):
/nix/store/08pckaqznwh0s3822cjp5aji6y1lsm27-libcxx-11.1.0
...
/nix/store/zqcs5xahjxij0c8vfw60lnfb6d979rn2-zlib-1.2.13
copying path '/nix/store/49wn01k9yikhjlxc1ym5b6civ29zz3gv-bash-5.1-p16' from 'https://cache.nixos.org'...
...
copying path '/nix/store/4w2rv6s96fwsb4qyw8b9w394010gxriz-stdenv-darwin' from 'https://cache.nixos.org'...
Using hello-2.12.1.
[nix-shell:~/tmp/nixplayground]$
If you close the session and run it again, it will be much faster and will only show this:
❯ nix-shell
Using hello-2.12.1.
[nix-shell:~/tmp/nixplayground]$
This is because all dependencies will be cached. OK so, this is level 1 of /nix-shell-fu/. Now, let's start level 2. ### =nix-shell-fu= level 2 lesson; scripting and configuring This time, we want to launch a full service, as a redis docker would do. So here is a basic shell script which is similar to the previous one but will request =redis= as a dependency instead of =hello= and also as a launching script. From there will add a little bit more features.
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
pkgs.mkShell {
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port: ${port}"
redis-server
'';
}
Again if you run ~nix-shell~ here is the result:
❯ nix-shell
these 2 paths will be fetched (2.08 MiB download, 6.99 MiB unpacked):
/nix/store/6w4vnaxdx12ccq172i8j5l830mlp8jlg-redis-7.0.5
/nix/store/b47gmsx9qx0c9vh75wsg8bqq9qd0ad6f-openssl-3.0.7
copying path '/nix/store/b47gmsx9qx0c9vh75wsg8bqq9qd0ad6f-openssl-3.0.7' from 'https://cache.nixos.org'...
copying path '/nix/store/6w4vnaxdx12ccq172i8j5l830mlp8jlg-redis-7.0.5' from 'https://cache.nixos.org'...
Using redis-7.0.5
97814:C 10 Feb 2023 20:44:36.960 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
97814:C 10 Feb 2023 20:44:36.960 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=97814, just started
97814:C 10 Feb 2023 20:44:36.960 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
97814:M 10 Feb 2023 20:44:36.961 * Increased maximum number of open files to 10032 (it was originally set to 256).
97814:M 10 Feb 2023 20:44:36.961 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 7.0.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 97814
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
97814:M 10 Feb 2023 20:44:36.962 # WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.
97814:M 10 Feb 2023 20:44:36.962 # Server initialized
97814:M 10 Feb 2023 20:44:36.963 * Ready to accept connections
Woo! Redis is started and it works! But if you have multiple projects you want to have more control. For example, we will want to run redis on a specific port. Here is how you do it:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/21.05.tar.gz) {} }:
let iport = 16380;
port = toString iport;
in pkgs.mkShell {
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port ${port}"
redis-server --port ${port}
'';
}
And here is the result:
rm dump.rdb
nix-shell
Using redis-6.2.3 on port 16380
1785:C 10 Feb 2023 20:50:00.880 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1785:C 10 Feb 2023 20:50:00.880 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=1785, just started
1785:C 10 Feb 2023 20:50:00.880 # Configuration loaded
1785:M 10 Feb 2023 20:50:00.880 * Increased maximum number of open files to 10032 (it was originally set to 256).
1785:M 10 Feb 2023 20:50:00.880 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.3 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 16380
| `-._ `._ / _.-' | PID: 1785
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1785:M 10 Feb 2023 20:50:00.881 # Server initialized
1785:M 10 Feb 2023 20:50:00.881 * Ready to accept connections
Woo! We control the port from the file. That's nice. But, has you might have noticed, when you quit the session it dumps the DB as the file =dump.rdb=. What we would like is to keep all the state in a local directory that would be easy to delete. To achieve this, instead of passing argument to the redis command line we will use a local config file to use.
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let iport = 16380;
port = toString iport;
in pkgs.mkShell (rec {
# ENV Variables the directory to put all the DATA
REDIS_DATA = "${toString ./.}/.redis";
# the config file, as we use REDIS_DATA variable we just declared in the
# same nix set, we need to use rec
redisConf = pkgs.writeText "redis.conf"
''
port ${port}
dbfilename redis.db
dir ${REDIS_DATA}
'';
buildInputs = [ pkgs.redis ];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.redis.name} on port: ${port}"
[ ! -d $REDIS_DATA ] \
&& mkdir -p $REDIS_DATA
cat "$redisConf" > $REDIS_DATA/redis.conf
alias redisstop="echo 'Stopping Redis'; redis-cli -p ${port} shutdown; rm -rf $REDIS_DATA"
nohup redis-server $REDIS_DATA/redis.conf > /dev/null 2>&1 &
echo "When finished just run redisstop && exit"
trap redisstop EXIT
'';
})
And here is a full session using this =shell.nix=:
nix-shell
Using redis-6.2.3 on port: 16380
When finished just run redisstop && exit
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redis-cli -p 16380
127.0.0.1:16380> help
redis-cli 6.2.3
To get help about Redis commands type:
"help @<group>" to get a list of commands in <group>
"help <command>" for help on <command>
"help <tab>" to get a list of possible help topics
"quit" to exit
To set redis-cli preferences:
":set hints" enable online hints
":set nohints" disable online hints
Set your preferences in ~/.redisclirc
127.0.0.1:16380>
-------------------------------
[nix-shell:~/tmp/nixplayground]$ ls -a
. .. .redis shell.nix
-------------------------------
[nix-shell:~/tmp/nixplayground]$ find .redis
.redis
.redis/redis.conf
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redis-cli -p 16380 shutdown
[1]+ Done nohup redis-server $REDIS_DATA/redis.conf > /dev/null 2>&1
-------------------------------
[nix-shell:~/tmp/nixplayground]$ find .redis
.redis
.redis/redis.db
.redis/redis.conf
-------------------------------
[nix-shell:~/tmp/nixplayground]$ redisstop
Stopping Redis
Could not connect to Redis at 127.0.0.1:16380: Connection refused
-------------------------------
[nix-shell:~/tmp/nixplayground]$ ls -a
. .. shell.nix
So with this version all data related to redis is saved into the local =.redis= directory. And in the nix shell we provide a command =redisstop= that once invoked, shutdown redis, then purge all redis related data (as you would like in a development environment). Also, as compared to previous version, redis is launched in background so you could run commands in your nix shell. Notice I also run ~redisstop~ command on exit of the nix-shell. So when you close the nix-shell redis is stopped and the DB state is cleaned up. ## Composable =nix-shell= As a quick recap you now have a boilerplate to create new =shell.nix=:
{ pkgs ? import ( ... ) {} }:
mkShell { MY_ENV_VAR_1 = ...;
MY_ENV_VAR_2 = ...;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' command_to_run_after_init '';
}
But if I give you two such =shell.nix= files, would you be able to compose them? Unfortunately, not directly. To solve the problem we will replace this boilerplate by another one that do not directly uses =mkShell=. And in order to make it fully composable, we will also need to narrow the environment variables declaration in a sub field:
{ pkgs ? import ( ... ) {} }:
let env = { PGDATA = ...; }
in { inherit env; # equivalent to env = env;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' some_command $PG_DATA '';
}
With this, we can compose two nix set into a single merged one that will be suitable to pass as argument to ~mkShell~. Another minor detail, but important one. In bash, the command ~trap~ do not accumulate but replace the function. For our need, we want to run all stop function on exit. So the ~trap~ directive added in the shell hook does not compose naturally. This is why we add a =stop= value that will contain the name of the bash function to call to stop and cleanup a service. Finally the main structure for each of our service will look like:
{ pkgs ? import ( ... ) {} }:
let env = { MY_SERVICE_ENV_VAR = ...; }
in { inherit env; # equivalent to env = env;
buildInputs = [ dependency-1 ... dependency-n ];
nativeBuildInputs = [ dependency-1 ... dependency-n ];
shellHook = '' my_command $MY_SERVICE_ENV_VAR '';
stop = "stop_my_service"
}
So let's start easy. To run a single shell script like this with =nix-shell=, you should put your service specific nix file in a =service.nix= file and create a =shell.nix= file that contains something like:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let service = import ./service.nix { inherit pkgs; };
in with service; pkgs.mkShell ( env //
{
buildInputs = buildInputs;
nativeBuildInputs = nativeBuildInputs ;
shellHook = shellHook;
})
Now, if you would like to run nix shell for multiple files, here is a first qui solution:
{ pkgs ? import (...) {}}:
let
# merge all the env sets
mergedEnvs = builtins.foldl' (acc: e: acc // e) {} envs;
# merge all the confs by accumulating the dependencies
# and concatenating the shell hooks.
mergedConfs =
builtins.foldl'
(acc: {buildInputs ? [], nativeBuildInputs ? [], shellHook ? "", ...}:
{ buildInputs = acc.buildInputs ++ buildInputs;
nativeBuildInputs = acc.nativeBuildInputs ++ nativeBuildInputs;
shellHook = acc.shellHook + shellHook;
})
emptyConf
confs;
in mkShell (mergedEnvs // mergedConfs)
And now, here is the full solution that also deal with other minor details like importing the files and dealing with the exit of the shell:
{ mergeShellConfs =
# imports should contain a list of nix files
{ pkgs, imports }:
let confs = map (f: import f { inherit pkgs; }) imports;
envs = map ({env ? {}, ...}: env) confs;
# list the name of a command to stop a service (if none provided just use ':' which mean noop)
stops = map ({stop ? ":", ...}: stop) confs;
# we want to stop all services on exit
stopCmd = builtins.concatStringsSep " && " stops;
# we would like to add a shellHook to cleanup the service that will call
# all cleaning-up function declared in sub-shells
lastConf =
{ shellHook = ''
stopall() { ${stopCmd}; }
echo "You can manually stop all services by calling stopall"
trap stopall EXIT
'';
};
# merge Environment variables needed for other shell environments
mergedEnvs = builtins.foldl' (acc: e: acc // e) {} envs;
# zeroConf is the minimal empty configuration needed
zeroConf = {buildInputs = []; nativeBuildInputs = []; shellHook="";};
# merge all confs by appending buildInputs and nativeBuildInputs
# and by concatenating the shellHooks
mergedConfs =
builtins.foldl'
(acc: {buildInputs ? [], nativeBuildInputs ? [], shellHook ? "", ...}:
{ buildInputs = acc.buildInputs ++ buildInputs;
nativeBuildInputs = acc.nativeBuildInputs ++ nativeBuildInputs;
shellHook = acc.shellHook + shellHook;
})
zeroConf
(confs ++ [lastConf]);
in (mergedEnvs // mergedConfs);
}
So I put this function declaration in a file named =./nix/merge-shell.nix=. And I have a =pg.nix= as well as a =redis.nix= file in the =nix= directory. On the root of the project the main =shell.nix= looks like:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let
# we import the file, and rename the function mergeShellConfs as mergeShells
mergeShells = (import ./nix/merge-shell.nix).mergeShellConfs;
# we call mergeShells
mergedShellConfs =
mergeShells { inherit pkgs;
# imports = [ ./nix/pg.nix ./nix/redis.nix ];
imports = [ ./nix/pg.nix ./nix/redis.nix ];
};
in pkgs.mkShell mergedShellConfs
And, that's it. Now when I run =nix-shell= it ## Appendix ### <<digression>> Digression In fact, this is a bit more complex than "just that". The reality is a bit more complex. The nix language is "pure", meaning, if you run the nix evaluation multiple times, it will always evaluate to the exact same value. But here, this block represent a function. The function takes as input a "nix set" (which you can see as an associative array, or a hash-map or also a javascript object depending on your preference), and this set is expected to contain a field named =pkgs=. If =pkgs= is not provided, it will use the set from the stable version 22.11 of nixpkgs by downloading them from github archive. The second part of the function generate "something" that is returned by an internal function of the standard library provided by =nix= which is named =mkShell=. So mainly, =mkShell= is a helper function that will generate what nix calls a / => https://blog.ielliott.io/nix-docs/derivation.html derivation /. Mainly, we don't really care about exactly what is a /derivation/. This is an internal to nix representation that could be finally used by different nix tools for different things. Typically, installing a package, running a local development environment with nix-shell or nix develop, etc… So the important detail to remember is that we can manipulate the parameter we pass to the functions =derivation=, =mkDerivation= and =mkShell=, but we have no mechanism to manipulate directly =derivation=. So in order to make that composable, you need to call the =derivation= internal function at the very end only. The argument of all these functions are /nix sets/ ### The full nix files for postgres For postgres:
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let iport = 15432;
port = toString iport;
pguser = "pguser";
pgpass = "pgpass";
pgdb = "iroh";
# env should contain all variable you need to configure correctly mkShell
# so ENV_VAR, but also any other kind of variables.
env = {
postgresConf =
pkgs.writeText "postgresql.conf"
''
# Add Custom Settings
log_min_messages = warning
log_min_error_statement = error
log_min_duration_statement = 100 # ms
log_connections = on
log_disconnections = on
log_duration = on
#log_line_prefix = '[] '
log_timezone = 'UTC'
log_statement = 'all'
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
logging_collector = on
log_min_error_statement = error
'';
postgresInitScript =
pkgs.writeText "init.sql"
''
CREATE DATABASE ${pgdb};
CREATE USER ${pguser} WITH ENCRYPTED PASSWORD '${pgpass}';
GRANT ALL PRIVILEGES ON DATABASE ${pgdb} TO ${pguser};
'';
PGDATA = "${toString ./.}/.pg";
};
in env // {
# Warning if you add an attribute like an ENV VAR you must do it via env.
inherit env;
# must contain buildInputs, nativeBuildInputs and shellHook
buildInputs = [ pkgs.coreutils
pkgs.jdk11
pkgs.lsof
pkgs.plantuml
pkgs.leiningen
];
nativeBuildInputs = [
pkgs.zsh
pkgs.vim
pkgs.nixpkgs-fmt
pkgs.postgresql_11
# postgres-11 with postgis support
# (pkgs.postgresql_11.withPackages (p: [ p.postgis ]))
];
# Post Shell Hook
shellHook = ''
echo "Using ${pkgs.postgresql_12.name}. port: ${port} user: ${pguser} pass: ${pgpass}"
# Setup: other env variables
export PGHOST="$PGDATA"
# Setup: DB
[ ! -d $PGDATA ] \
&& pg_ctl initdb -o "-U postgres" \
&& cat "$postgresConf" >> $PGDATA/postgresql.conf
pg_ctl -o "-p ${port} -k $PGDATA" start
echo "Creating DB and User"
psql -U postgres -p ${port} -f $postgresInitScript
function pgstop {
echo "Stopping and Cleaning up Postgres";
pg_ctl stop && rm -rf $PGDATA
}
alias pg="psql -p ${port} -U postgres"
echo "Send SQL commands with pg"
trap pgstop EXIT
'';
stop = "pgstop";
}
And to just launch Posgresql, there is also this file =./nix/pgshell.nix=, that simply contains
{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz) {} }:
let pg = import ./pg.nix { inherit pkgs; };
in with pg; pkgs.mkShell ( env //
{
buildInputs = buildInputs;
nativeBuildInputs = nativeBuildInputs ;
shellHook = shellHook;
})
=> /index.gmi Home
=> /gem-atom.xml Feed
=> /slides.gmi Slides
=> /about-me.gmi About
=> https://gitea.esy.fun code
=> https://espial.esy.fun/u:yogsototh bookmarks
=> https://espial.esy.fun/u:yogsototh/notes notes
]]></description>
</item>
<item>
<title>My first ChatGPT exchange</title>
<guid>gemini://her.esy.fun/posts/0023-my-first-chatgpt-exchange/index.gmi</guid>
<pubDate>Sun, 11 Dec 2022 00:00:00 +0100</pubDate>
<category>machine-learning</category>
<category>ai</category>
<category>gpt</category>
<description><![CDATA[
# My first ChatGPT exchange
description: My first interaction with ChatGPT about Clojure and Monads.
keywords: machine-learning ai gpt
author: Yann Esposito
email: yann@esposito.host
=> /files/publickey.txt gpg
date: [2022-12-11 Sun]
# The first few questions for ChatGPT
I finally opened an account to beta-test ChatGPT.
I must say, I am quite impressed.
I asked it a few questions that I could have asked during an interview.
To my surprise, the answer given was, while wrong, believable enough to kind of
sneak the answer as right up to some imprecision.
Like if you squint enough, it could pass, but if you pay attention it is clearly
false.
So yes, mainly ChatGPT in this interaction was quite good at being convincing.
This is the same kind of impression you have from looking at AI generated images.
At first sight everything seems fine, but if you start looking to the details, many
things are off.
So here is the full interaction, I'll let the attentive reader find the
technical problems:
{{{logo}}} Can you write a clojure function that will reverse any string provided to it.
______
(defn reverse-string [s]
(apply str (reverse (seq s))))
This function takes a strings as input, converts it to a sequence of characters using the ~seq~ function, reverses this sequence using the reverse function, and then converts the resulting sequence back to a string using the ~apply~ and ~str~ functions. Here's an example of how you can use this function:
user=> (reverse-string "hello world")
"dirow olleh"
Note that this function only works
for strings that contain characters in the Unicode Basic Multilingual Plane
(BMP), since the ~seq~ function only works on strings that contain characters in
this range.
If you need to reverse strings that contain characters outside of the BMP, you
can use the ~string-seq~ function instead of ~seq~ to convert the string to a
sequence of characters.
______
{{{logo}}} How would this function react if instead of string you provide an integer?
______
user=> (reverse-string 12345)
ClassCastException clojure.lang. Numbers cannot be cast to
clojure.lang. IFn clojure.core/seq (core.clj:136)
To avoid this error, you can either make sure that the input to the ~reverse-string~ function is always a string, or you can modify the function to handle other types of input, such as integers, by first converting them to strings using the ~str~ function. For example, you could write the ~reverse-string~ function like this to handle both strings and integers as input:
(defn reverse-string [s]
(let [s (if (string? s) s (str s))]
(apply str (reverse (seq s)))))
With this version of the function, if you pass an integer as input, the function will convert it to a string using the ~str~ function before calling the ~seq~ function on it, so the ~seq~ function will be able to work with the input. Here's an example of how you can use this modified version of the ~reverse-string~ function:
user-> (reverse-string "hello world")
"dlrow olleh"
user=> (reverse-string 12345)
"54321"
Note that the modified version of the function will also work for other types of
input that can be converted to strings using the ~str~ function, such as floats
and booleans.
______
{{{logo}}} How would you use the concept of the Maybe monad in Haskell in this clojure function?
______
(defn reverse-string [s]
(if (string? s)
(Some (apply str (reverse (seq s))))
None))
With this version of the ~reverse-string~ function, if the input is a string, the function will return a ~Some~ value containing the reversed string, and if the input is not a string, the function will return a ~None~ value. This allows the caller of the function to handle the case where the input is not a string in a predictable and safe way. Here's an example of how you can use this modified version of the ~reverse-string~ function:
user=> (reverse-string "hello world")
Some "dlrow olleh"
user=> (reverse-string 12345)
None
Note that the Option type in Clojure does not provide the same syntactic sugar
for composing computations as the Maybe monad in Haskell, so you cannot use the
~>>=~ operator to chain calls to the ~reverse-string~ function in the same way you
can in Haskell.
However, you can still use the ~Option~ type to represent computations that may
or may not have a result, and you can use the ~match~ or ~case~ forms to pattern
match on the ~Option~ values and handle the different cases in a predictable and
safe way.
______
{{{logo}}} How would you implement Option in clojure?
______
(defprotocol Option
(map [o f])
(flat-map [o f]))
(deftype Some [x]
Option
(map [o f] (Some (f x)))
(flat-map [o f] (f x)))
(deftype None [])
Option
(map [o f] None)
(flat-map [o f] None))
The ~Option~ protocol defines two methods: ~map~ and ~flat-map~ The ~map~ method takes a function ~f~ as input and applies it to the value contained in the ~Some~ type, ~f~ it exists, or returns a ~None~ value ~f~ the ~Option~ type is ~None~ The ~flat-map~ method is similar to ~map~ but it allows the function ~f~ to return an ~Option~ value, which is then flattened into the outer ~Option~ value. The ~Some~ type implements the ~Option~ protocol by providing concrete implementations ~f~ the map and ~flat-map~ methods. The 'map method applies the function ~f~ to the value ~x~ contained in the ~Some~ type, and returns a new ~Some~ value containing the result ~f~ the function. The ~flat-map~ method applies the function ~f~ to the value ~x~ contained in the Some type, and then flattens the resulting ~Option~ value into the outer ~Option~ value. The ~None~ type also implements the Option protocol by providing concrete implementations of the ~map~ and ~flat-map~ methods. The ~map~ and ~flat-map~ methods simply return a ~None~ value in this case, since there is no value to apply the function to. Here's an example of how you can use the ~Option~ protocol and the ~Some~ and ~None~ types:
user=> (def some-value (Some 12345))
user=> (map some-value inc)
Some 12346
user=> (flat-map some-value (fn [x] (Some (* × 2))))
Some 24690
user=> (map some-value (comp (partial * 2) inc))
Some 24690
user=> (def none-value None)
user=> (map none-value inc)
None
user=> (flat-map none-value (fn [x] (Some (* × 2))))
None
user=> (map none-value (comp (partial * 2) inc))
None
In this example, the ~some-value~ variable is defined as a ~Some~ value containing
the integer 12345, and the ~none-value~ variable is defined as a ~None~ value.
The ~map~ and ~flat-map~ functions are then used to apply different functions to the
values contained in the ~Option~ types, and the...
______
=> /index.gmi Home
=> /gem-atom.xml Feed
=> /slides.gmi Slides
=> /about-me.gmi About
=> https://gitea.esy.fun code
=> https://espial.esy.fun/u:yogsototh bookmarks
=> https://espial.esy.fun/u:yogsototh/notes notes
]]></description>
</item>
<item>
<title>My first ChatGPT exchange</title>
<guid>gemini://her.esy.fun/posts/0023-my-first-chatgpt-exchange/flycheck_index.gmi</guid>
<pubDate>Sun, 11 Dec 2022 00:00:00 +0100</pubDate>
<category>machine-learning</category>
<category>ai</category>
<category>gpt</category>
<description><![CDATA[
# My first ChatGPT exchange
description: My first interaction with ChatGPT about Clojure and Monads.
keywords: machine-learning ai gpt
author: Yann Esposito
email: yann@esposito.host
=> /files/publickey.txt gpg
date: [2022-12-11 Sun]
# The first few questions for ChatGPT
I finally opened an account to beta-test ChatGPT.
I must say, I am quite impressed.
I asked it a few questions that I could have asked during an interview.
To my surprise, the answer given was, while wrong, believable enough to kind
of sneak the answer as right up to some imprecision.
Like if you squint enough, it could pass, but if you pay attention
it is clearly false.
So yes, mainly ChatGPT in this interaction was quite good at being convincing,
the same kind of impression you have from looking at AI generated images.
At first sight everything seems fine, but if you start looking to the details, many
things are off.
So here is the full interaction, I'll let the attentive reader find the
technical problems:
{{{logo}}} Can you write a clojure function that will reverse any string provided to it.
______
(defn reverse-string [s]
(apply str (reverse (seq s))))
This function takes a strings as input, converts it to a sequence of characters using the ~seq~ function, reverses this sequence using the reverse function, and then converts the resulting sequence back to a string using the ~apply~ and ~str~ functions. Here's an example of how you can use this function:
user=> (reverse-string "hello world")
"dirow olleh"
Note that this function only works
for strings that contain characters in the Unicode Basic Multilingual Plane
(BMP), since the ~seq~ function only works on strings that contain characters in
this range.
If you need to reverse strings that contain characters outside of the BMP, you
can use the ~string-seq~ function instead of ~seq~ to convert the string to a
sequence of characters.
______
{{{logo}}} How would this function react if instead of string you provide an integer?
______
user=> (reverse-string 12345)
ClassCastException clojure.lang. Numbers cannot be cast to
clojure.lang. IFn clojure.core/seq (core.clj:136)
To avoid this error, you can either make sure that the input to the ~reverse-string~ function is always a string, or you can modify the function to handle other types of input, such as integers, by first converting them to strings using the ~str~ function. For example, you could write the ~reverse-string~ function like this to handle both strings and integers as input:
(defn reverse-string [s]
(let [s (if (string? s) s (str s))]
(apply str (reverse (seq s)))))
With this version of the function, if you pass an integer as input, the function will convert it to a string using the ~str~ function before calling the ~seq~ function on it, so the ~seq~ function will be able to work with the input. Here's an example of how you can use this modified version of the ~reverse-string~ function:
user-> (reverse-string "hello world")
"dlrow olleh"
user=> (reverse-string 12345)
"54321"
Note that the modified version of the function will also work for other types of
input that can be converted to strings using the ~str~ function, such as floats
and booleans.
______
{{{logo}}} How would you use the concept of the Maybe monad in Haskell in this clojure function?
______
(defn reverse-string [s]
(if (string? s)
(Some (apply str (reverse (seq s))))
None))
With this version of the ~reverse-string~ function, if the input is a string, the function will return a ~Some~ value containing the reversed string, and if the input is not a string, the function will return a ~None~ value. This allows the caller of the function to handle the case where the input is not a string in a predictable and safe way. Here's an example of how you can use this modified version of the ~reverse-string~ function:
user=> (reverse-string "hello world")
Some "dlrow olleh"
user=> (reverse-string 12345)
None
Note that the Option type in Clojure does not provide the same syntactic sugar
for composing computations as the Maybe monad in Haskell, so you cannot use the
~>>=~ operator to chain calls to the ~reverse-string~ function in the same way you
can in Haskell.
However, you can still use the ~Option~ type to represent computations that may
or may not have a result, and you can use the ~match~ or ~case~ forms to pattern
match on the ~Option~ values and handle the different cases in a predictable and
safe way.
______
{{{logo}}} How would you implement Option in clojure?
______
(defprotocol Option
(map [o f])
(flat-map [o f]))
(deftype Some [x]
Option
(map [o f] (Some (f x)))
(flat-map [o f] (f x)))
(deftype None [])
Option
(map [o f] None)
(flat-map [o f] None))
The ~Option~ protocol defines two methods: ~map~ and ~flat-map~ The ~map~ method takes a function ~f~ as input and applies it to the value contained in the ~Some~ type, ~f~ it exists, or returns a ~None~ value ~f~ the ~Option~ type is ~None~ The ~flat-map~ method is similar to ~map~ but it allows the function ~f~ to return an ~Option~ value, which is then flattened into the outer ~Option~ value. The ~Some~ type implements the ~Option~ protocol by providing concrete implementations ~f~ the map and ~flat-map~ methods. The 'map method applies the function ~f~ to the value ~x~ contained in the ~Some~ type, and returns a new ~Some~ value containing the result ~f~ the function. The ~flat-map~ method applies the function ~f~ to the value ~x~ contained in the Some type, and then flattens the resulting ~Option~ value into the outer ~Option~ value. The ~None~ type also implements the Option protocol by providing concrete implementations of the ~map~ and ~flat-map~ methods. The ~map~ and ~flat-map~ methods simply return a ~None~ value in this case, since there is no value to apply the function to. Here's an example of how you can use the ~Option~ protocol and the ~Some~ and ~None~ types:
user=> (def some-value (Some 12345))
user=> (map some-value inc)
Some 12346
user=> (flat-map some-value (fn [x] (Some (* × 2))))
Some 24690
user=> (map some-value (comp (partial * 2) inc))
Some 24690
user=> (def none-value None)
user=> (map none-value inc)
None
user=> (flat-map none-value (fn [x] (Some (* × 2))))
None
user=> (map none-value (comp (partial * 2) inc))
None
In this example, the ~some-value~ variable is defined as a ~Some~ value containing the integer 12345, and the ~none-value~ variable is defined as a ~None~ value. The ~map~ and ~flat-map~ functions are then used to apply different functions to the values contained in the ~Option~ types, and the... ______ => /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> <item> <title>Org agenda on your startup page</title> <guid>gemini://her.esy.fun/posts/0022-org-agenda-on-your-startup-page/index.gmi</guid> <pubDate>Thu, 05 May 2022 00:00:00 +0200</pubDate> <category>blog</category> <category>static</category> <description><![CDATA[ # Org agenda on your startup page description: keywords: blog static author: Yann Esposito email: yann@esposito.host => /files/publickey.txt gpg date: [2022-05-05 Thu] # Your org-agenda at your homepage During most of the day my emacs is open. But I sometime forget to look at my org-agenda. This is why I wanted to find another way to be exposed to it. And one thing I am mostly exposed to is my personal start page. This is just the default page I see when I open my browser. Here is the end result: attr_org: :width 560 attr_html: Generate HTML from the Agenda caption: The result inside my start page name: fig:agenda-html => ./agenda-html.png My start page is named /mothership/. And I just put the org-agenda at the top of the page inside an iframe. I have a service that start a server on localhost and I configured my browser to use it as startup page. That's it. So now, here is how to sync my org-agenda on this start page. In my ~config.el~:
(setq y/mothership "~/dev/mothership/")
(setq org-agenda-custom-commands
`(("g" "Plan Today"
((agenda "" ((org-agenda-span 'day))))
nil
( ,(concat y/mothership "/agenda.html") ))))
This provide a custom org agenda command and link that custom command to the export file =~/dev/mothership/agenda.html=. And a shell script:
emacs --batch \
--load "$HOME/.emacs.d/init.el" \
--eval '(org-batch-store-agenda-views)' \
--kill
And finally in my start-page html I just need to add an iframe like so:
<iframe id="agenda" src="gemini://her.esy.fun/posts/0022-org-agenda-on-your-startup-page/agenda.html"></iframe>
But as I also want to be able to toggle the agenda, and auto-resize the iframe. So I added a bit of js code:
<div id="agendatoggle">[-]</div><br/>
<script>
function resizeIframe(iframe) {
iframe.height = ( iframe.contentWindow.document.body.scrollHeight + 30) + "px";
}
</script>
<iframe id="agenda" src="gemini://her.esy.fun/posts/0022-org-agenda-on-your-startup-page/agenda.html" onload="resizeIframe(this)"></iframe>
</div>
<script>
function hideAgenda () {
let agenda = document.getElementById("agenda");
agenda.style.display = 'none';
let at = document.getElementById("agendatoggle");
at.innerHTML="agenda [+]";
at.onclick = showAgenda;
}
function showAgenda () {
let agenda = document.getElementById("agenda");
agenda.style.display = 'block';
resizeIframe(agenda);
// setInterval(function(){ ; }, 1000);
let at = document.getElementById("agendatoggle");
at.innerHTML="agenda [-]";
at.onclick = hideAgenda;
}
showAgenda();
</script>
And that's it. That's a neat trick, and so I'm glad to put a small post about it. ## Bonuses ### auto-resize iframe In order to auto-resize the iframe you must have a non =file:///= URL in the browser. So you must serve your start page. After a lot of different way to serve my pages, I finally use ~lighttpd~ inside a ~nix-shell~. So I added the following to my startpage code: A => ./lighttpd.conf file
server.bind = "127.0.0.1"
server.port = 31337
server.document-root = var.CWD
index-file.names = ( "index.html" )
mimetype.assign = (
".css" => "text/css",
".gif" => "image/gif",
".htm" => "text/html",
".html" => "text/html",
".jpeg" => "image/jpeg",
".jpg" => "image/jpeg",
".js" => "text/javascript",
".png" => "image/png",
".swf" => "application/x-shockwave-flash",
".txt" => "text/plain",
".gmi" => "text/plain",
".svg" => "image/svg+xml",
".svgz" => "image/svg+xml"
)
$HTTP["useragent"] =~ "^(.*MSIE.*)|(.*AppleWebKit.*)$" {
server.max-keep-alive-requests = 0
}
With a
webdir="_site"
port="$(grep server.port ./lighttpd.conf|sed 's/[^0-9]*//')"
echo "Serving: $webdir on http://localhost:$port" && \
lighttpd -f ./lighttpd.conf -D
I have a frozen nixpkgs dependencies via =niv=. But you could probably simply replace the line by:
And it should work fine. ### Start your server at startup on macOS Last but not least, starting this start page server when I login. So for that you should demonize with launchd. I created a => ./y.mothership.plist file to put in =~/Library/LauchAgents/=:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>mothership</string>
<key>ProgramArguments</key>
<array>
<string>/bin/zsh</string>
<string>-c</string>
<string>$HOME/y/mothership/serve.sh</string>
</array>
<key>StandardOutPath</key>
<string>/var/log/mothership.log</string>
<key>StandardErrorPath</key>
<string>/var/log/mothership.log</string>
</dict>
</plist>
Then to ensure that the executable ~nix-shell~ is present in the PATH for the demon, I ran:
launchctl config user path "$PATH"
This will affect the path of all users, but as I am the only user on my computer this is fine for me. Then:
launchctl load ~/Library/LaunchAgents/y.mothership.plist
launchctl start mothership
And that's about it. => /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> <item> <title>My personal environment sync</title> <guid>gemini://her.esy.fun/posts/0021-my-personal-environment-sync/index.gmi</guid> <pubDate>Sat, 30 Oct 2021 00:00:00 +0200</pubDate> <category>programming</category> <description><![CDATA[ # My personal environment sync description: keywords: programming author: Yann Esposito email: yann@esposito.host => /files/publickey.txt gpg date: [2021-10-30 Sat] I have a quite specific system that I improved along the years to manage my local environment. Think about, binaries I expect to have in my shell, as well as configuration files for various utilities, and a few personal scripts. The notion of what is exactly my local environment is not perfectly defined. I expect every of my computers to behave slightly differently. Some are for work-only, some personal use only. For the things I want everywhere, I have a peculiar personal system. I use a personal script that depends on => https://yadm.io yadm and => https://github.com/nix-community/home-manager home-manager My script try to check if some files where updated and react accordingly: 1. I download the dot-files changes via =yadm=. 2. If my home-manager files changes, it will run ~home-manager switch~ ; if it fails, try to update nix channels then try again. 3. If my doom emacs packages changed, it will run ~doom sync~ 4. If the script itself changed, it re-run the script after updating itself. If the script detect that I changed my emacs configuration, it runs ~doom sync~ or ~doom sync -u~. Here it is:
red=1
green=2
yellow=3
blue=4
highpr() {
printf "$(tput setaf $green)→$(tput sgr0) $(tput bold)%-60s$(tput sgr0)" "$*"
}
ok() {
local txt="OK"
echo -e " [$(tput bold)$(tput setaf $green)${txt}$(tput sgr0)]" >&2
}
info() {
echo -e " [$(tput bold)$(tput setaf $blue)$*$(tput sgr0)]" >&2
}
warn() {
echo -e "$(tput bold)$(tput setaf $yellow)$*$(tput sgr0)" >&2
}
err() {
echo -e "$(tput bold)$(tput setaf $red)$*$(tput sgr0)" >&2
}
fail() {
err -e "\n[ERR] $*"
exit 1
}
highpr "check nix"
if ! [ -x "$(command -v nix)" ]; then
echo
err "nix does not seem to be installed."
err "Install it from: https://nixos.org/nix/"
exit 1
fi
ok
highpr "yadm fetch"
yadm fetch --quiet || fail "yadm fetch failed"
ok
OLD_SYNC_ENV_ID=$(yadm rev-parse HEAD:bin/sync-env.sh)
OLD_HOME_MANAGER_ID=$(yadm rev-parse HEAD:.config/nixpkgs/home.nix)
OLD_DOOM_PACKAGES=$(yadm rev-parse HEAD:.doom.d/packages.el)
OLD_DOOM_INIT=$(yadm rev-parse HEAD:.doom.d/init.el)
highpr "yadm pull"
yadm pull --quiet || fail "yadm pull failed"
ok
NEW_SYNC_ENV_ID=$(yadm rev-parse HEAD:bin/sync-env.sh)
NEW_HOME_MANAGER_ID=$(yadm rev-parse HEAD:.config/nixpkgs/home.nix)
NEW_DOOM_PACKAGES=$(yadm rev-parse HEAD:.doom.d/packages.el)
NEW_DOOM_INIT=$(yadm rev-parse HEAD:.doom.d/init.el)
highpr "check sync-env diff"
if ! [ "$OLD_SYNC_ENV_ID" = "$NEW_SYNC_ENV_ID" ]; then
warn " changed"
warn " Starting ~/bin/sync-env.sh again"
echo
~/bin/sync-env.sh
exit $?
fi
ok
if [ -f "$HOME/.yadm/files.gpg" ]; then
highpr "yadm decrypt"
yadm decrypt || fail "yadm decrypt failed"
ok
fi
highpr "home-manager"
USERNAME_NIX_FILE="$HOME/.config/nixpkgs/username.nix"
if [ ! -f "$USERNAME_NIX_FILE" ]; then
echo "\"$USER\"" >> "$USERNAME_NIX_FILE"
fi
if ! [ "$OLD_HOME_MANAGER_ID" = "$NEW_HOME_MANAGER_ID" ]; then
echo
highpr "home-manager switch"
home-manager switch || \
( nix-channel --update && home-manager switch ) || \
fail "home-manager switch failed"
ok
else
info "skipped"
fi
highpr "doom-emacs"
doompath="$HOME/.emacs.d/bin/doom"
if ! [ "$OLD_DOOM_PACKAGES" = "$NEW_DOOM_PACKAGES" ] || \
! [ "$OLD_DOOM_INIT" = "$NEW_DOOM_INIT" ]; then
if => -x $doompath ; then
echo
highpr "doom sync"
$doompath sync || fail "doom failed to sync"
ok
else
fail "Cannot find doom executable at $doompath";
fi
else
info "skipped"
fi
## Bootstrapping
Bootstrapping this system is always a nice problem to think about.
It is smooth when everything is set but to bootstrap it I need binaries
installed by this system...
So...
How to handle the dependency cycle correctly?
To minimize the pain, I removed more and more bootstrapping dependencies.
Now my almost single dependence for bootstrapping my environment is =nix=.
I haven't initialized any machine for a long time now.
The following should work.
0. Use fish[fn:fish] ~chsh /bin/fish~
1. Install nix
~curl -L https://nixos.org/nix/install | sh~
2. Install home-manager
```bash
nix-channel --add https://github.com/nix-community/home-manager/archive/release-21.05.tar.gz home-manager
nix-channel --update
export NIX_PATH=$HOME/.nix-defexpr/channels${NIX_PATH:+:}$NIX_PATH
nix-shell '<home-manager>' -A install
```
3. Install and use ~yadm~
```bash
nix-shell -p yadm
yadm boostrap
yadm remote set-url origin <url-to-my-dot-files-repo>
yadm pull
```
4. Still in the =nix-shell= with =yadm= run ~~/bin/sync-env.sh~
There is a risk that step 3 fail because I pin most of my packages in
home-manager configuration, and it will try to install =yadm=. This can
conflict with the =yadm= installed in the current =nix-shell=.
So sometime I need to:
1. Remove the line installing =yadm= in my home-manager configuration first
2. run =home-manager sync=
3. get out of the =nix-shell=,
4. add =yadm= back in the =home-manager= config
5. run =home-manager sync= again, but this time out of the =nix-shell=.
6. Finally I can run my =~/bin/sync-env.sh= command.
So this post will probably be useful as a personal note in the future.
Because bootstrapping is generally not trivial.
I will probably update this post if something is missing.
[fn:fish] I use fish for interactive shell. I use ~zsh~ for quick dirty
scripts (a lot better than bash), and I switch to
=> https://hackage.haskell.org/package/turtle turtle
if I need to be
serious about the script.
=> /index.gmi Home
=> /gem-atom.xml Feed
=> /slides.gmi Slides
=> /about-me.gmi About
=> https://gitea.esy.fun code
=> https://espial.esy.fun/u:yogsototh bookmarks
=> https://espial.esy.fun/u:yogsototh/notes notes
]]></description>
</item>
<item>
<title>iA writer clone within doom-emacs</title>
<guid>gemini://her.esy.fun/posts/0021-ia-writer-clone-within-doom-emacs/index.gmi</guid>
<pubDate>Sun, 24 Oct 2021 00:00:00 +0200</pubDate>
<category>emacs</category>
<category>org-mode</category>
<description><![CDATA[
# iA writer clone within doom-emacs
description:
keywords: emacs org-mode
author: Yann Esposito
email: yann@esposito.host
=> /files/publickey.txt gpg
date: [2021-10-24 Sun]
So I played with tweaking my emacs configuration again.
I think I made something worth to be shared.
I wanted to have the same effect than in iA writer within emacs.
And I just reached this.
So the effect I am looking to achieve can be seen in this video.
name: demo video
______html
<video src="gemini://her.esy.fun/posts/0021-ia-writer-clone-within-doom-emacs/zen-writer-demo.mov" width="100%" controls autoplay>
<a href="gemini://her.esy.fun/posts/0021-ia-writer-clone-within-doom-emacs/zen-writer-demo.mov">zen-writer-demo video (7.2MB)</a>
</video>
______
______
=> ./zen-writer-demo.mov zen-writer-demo video (7.2MB)
______
It highlight the current sentence (in black) while the rest of the text is
gray.
The main issue with the =hl-sentence= package alone is that it set a specific
face to the current sentence, but does not affect the other text in the
buffer.
In fact to make it work as I would expect, you need to make the default
color grey, and only set black for the highlighted text.
Fortunately, I have recently created a personal theme close to that.
I just created a new specific one.
Everything is mostly "gray" except the font ~hl-sentence~ which is black.
And that's it.
So to make it all work I need.
(package! hl-sentence
:pin "86ae38d3103bd20da5485cbdd59dfbd396c45ee4")
## Helpers You probably want to be able to put you out of this writing mode. Here is a => zen-writer.el zen-writer.el file that contain my keymaps and useful functions. Put it in you =~/.doom.d/= directory and in you =config.el= put
(load! "~/.doom.d/zen-writer.el")
And with this you should pass to zen mode with ~SPC y z z~. To make the un-zen works. You will need to have a ~y/auto-update-theme~ function that set your current theme. My function change the theme depending on the time of the day or the day of the week. So here it is for inspiration:
(defun y/auto-update-theme ()
"depending on time use different theme"
;; very early => gruvbox-light, solarized-light, nord-light
(let* ((day-of-week (format-time-string "%a"))
(week-end? (or (equal "Sat" day-of-week)
(equal "Sun" day-of-week)))
(hour (nth 2 (decode-time (current-time))))
(time-to-sleep? (or (> hour 22) (< hour 7)))
(theme (cond
(time-to-sleep? 'doom-plain-dark)
(week-end? 'doom-nord-light)
((<= 7 hour 8) 'doom-gruvbox-light)
((= 9 hour) 'doom-solarized-light)
((<= 10 hour 16) 'doom-solarized-white)
((<= 17 hour 18) 'doom-gruvbox-light)
((<= 19 hour 22) 'doom-oceanic-next))))
(when (not (equal doom-theme theme))
(setq doom-theme theme)
(load-theme doom-theme t))
;; run that function again next hour
(run-at-time (format "%02d:%02d" (+ hour 1) 0) nil 'y/auto-update-theme)))
## Bonus I use Nerd Fonts and in particular the font `iMWritingDuoS` which is I think a clone of the iAWriter font.
(setq doom-variable-pitch-font
(font-spec :family "iMWritingDuoS Nerd Font" :size 12))
I hope you find this useful. I really like how it looks now. ## Annex The code source used in this article. ### zen-writer name: zen-writer.el
;;; zen-writer.el -*- lexical-binding: t; -*-
(defun y/zen ()
(interactive)
(setq doom-theme 'doom-zen-writer)
(load-theme doom-theme t)
(hl-sentence-mode +1))
(defun y/unzen ()
(interactive)
(y/auto-update-theme)
(hl-sentence-mode -1))
(defun y/zen-full ()
(interactive)
(y/zen)
(toggle-frame-fullscreen)
(doom-big-font-mode +1))
(defun y/unzen-full ()
(interactive)
(y/unzen)
(toggle-frame-fullscreen)
(doom-big-font-mode -1))
(map! :leader
(:prefix ("y z" . "Zen Writer")
:desc "Full Zen Writer" "z" #'y/zen-full
:desc "un-Full Zen Writer" "u" #'y/unzen-full
:desc "Zen Writer" "t" #'y/zen
:desc "un-Zen Writer" "q" #'y/unzen))
### zen-writer doom theme
;;; doom-zen-writer-theme.el --- -*- lexical-binding: t; no-byte-compile: t; -*-
;;
;; Author: Yann Esposito <https://github.com/yogsototh>
;; Created: October 24, 2021
;; Version: 1.0.0
;; Keywords: custom themes, faces
;; Homepage: https://github.com/hlissner/emacs-doom-themes
;; Package-Requires: ((emacs "25.1") (cl-lib "0.5") (doom-themes "2.2.1"))
;;
;;; Code:
(require 'doom-themes)
;;
;;; Variables
(defgroup doom-plain-theme nil
"Options for the `doom-plain' theme."
:group 'doom-themes)
(defcustom doom-plain-padded-modeline doom-themes-padded-modeline
"If non-nil, adds a 4px padding to the mode-line.
Can be an integer to determine the exact padding."
:group 'doom-plain-theme
:type '(or integer boolean))
;;
;;; Theme definition
(def-doom-theme doom-zen-writer
"Theme inspired by gko's plain."
;; name default/256/16
((bg '("#ffffff"))
(bg-alt '("#eaecea"))
(base0 '("#969896"))
(base1 '("#f1f3f5"))
(base2 '("#606666"))
(base3 '("#cccccc"))
(base4 '("#e7e7e7"))
(base5 '("#a5a8a6"))
(base6 '("#fafafa"))
(base7 '("#dfdfdf"))
(base8 '("#fafafa"))
(fg '("#969896"))
(fg-alt (doom-lighten fg 0.15))
(grey fg)
(red fg)
(blue fg)
(dark-blue fg)
(orange fg)
(green fg)
(teal fg)
(yellow fg)
(magenta fg)
(violet fg)
(cyan fg)
(dark-cyan fg)
;; face categories -- required for all themes
(highlight base2)
(vertical-bar base5)
(selection base1)
(builtin base0)
(comments base5)
(doc-comments base5)
(constants base0)
(functions fg)
(keywords fg)
(methods fg)
(operators fg)
(type fg)
(strings base0)
(variables base0)
(numbers base0)
(region base4)
(error (doom-blend fg "#ff0000" 0.4))
(warning base2)
(success green)
(vc-modified base5)
(vc-added (doom-lighten fg 0.7))
(vc-deleted base2)
;; custom categories
(-modeline-pad
(when doom-plain-padded-modeline
(if (integerp doom-plain-padded-modeline) doom-plain-padded-modeline 4)))
(modeline-bg (doom-darken bg-alt 0.15))
(modeline-bg-alt (doom-darken bg-alt 0.1))
(modeline-bg-inactive (doom-darken bg-alt 0.1))
(modeline-bg-inactive-alt bg-alt)
(modeline-fg fg)
(modeline-fg-alt (doom-darken modeline-bg-inactive 0.35)))
;;;; Base theme face overrides
((error :underline `(:style wave :color ,error))
(warning :underline `(:style wave :color ,warning))
(hl-sentence :foreground "#000000" :background bg)
((font-lock-constant-face &override) :slant 'italic)
((font-lock-comment-face &override) :slant 'italic)
((font-lock-function-name-face &override) :slant 'italic)
((font-lock-type-face &override) :slant 'italic)
(hl-line :background base8)
((line-number &override) :foreground base3)
((line-number-current-line &override) :foreground base2)
(mode-line
:background modeline-bg :foreground modeline-fg
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg)))
(mode-line-inactive
:background modeline-bg-inactive :foreground modeline-fg-alt
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg-inactive)))
(mode-line-emphasis :foreground highlight)
;;;; doom-modeline
(doom-modeline-bar :background modeline-bg)
(doom-modeline-bar-inactive :inherit 'doom-modeline-bar)
(doom-modeline-project-dir :foreground fg)
(doom-modeline-buffer-file :foreground fg)
(doom-modeline-buffer-modified :weight 'bold :foreground "#000000")
(doom-modeline-panel :inherit 'mode-line-highlight :background base3 :foreground fg)
;;;; ivy
(ivy-posframe :background bg-alt)
;;;; magit
((magit-diff-added-highlight &override) :foreground fg :background (doom-blend vc-added bg 0.3))
((magit-diff-removed &override) :foreground (doom-lighten fg 0.4) :background (doom-blend vc-deleted bg 0.1))
((magit-diff-removed-highlight &override) :foreground fg :background (doom-blend vc-deleted bg 0.22))
;;;; lsp-mode
(lsp-headerline-breadcrumb-symbols-face :foreground keywords :weight 'bold)
;;;; outline <built-in>
(outline-1 :slant 'italic :foreground fg-alt)
(outline-2 :inherit 'outline-1 :foreground base2)
(outline-3 :inherit 'outline-2)
(outline-4 :inherit 'outline-3)
(outline-5 :inherit 'outline-4)
(outline-6 :inherit 'outline-5)
(outline-7 :inherit 'outline-6)
(outline-8 :inherit 'outline-7)
;;;; org <built-in>
((org-block &override) :background bg-alt)
((org-block-begin-line &override) :foreground base5)
;;;; solaire-mode
(solaire-mode-line-face
:inherit 'mode-line
:background modeline-bg-alt
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg-alt)))
(solaire-mode-line-inactive-face
:inherit 'mode-line-inactive
:background modeline-bg-inactive-alt
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg-inactive-alt)))))
;;; doom-zen-writer-theme.el ends here
=> /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> <item> <title>Cool looking org-mode</title> <guid>gemini://her.esy.fun/posts/0020-cool-looking-org-mode/index.gmi</guid> <pubDate>Sat, 21 Aug 2021 00:00:00 +0200</pubDate> <category>org-mode</category> <category>emacs</category> <description><![CDATA[ # Cool looking org-mode description: A configuration to make org-mode look even better. keywords: org-mode emacs author: Yann Esposito email: yann@esposito.host => /files/publickey.txt gpg date: [2021-08-21 Sat] ______ TL;DR: My tweaked configuration to make org-mode even more pleasant to use. ______ ## The code At the end of this article there is a long digression about why I ended up here. But instead of bothering you with the why here is a what it looks like, and how to achieve it. First you need to install some dependencies. 1. Install nerdfonts[fn:nerdfonts] 2. Tell org-mode to use =variable-pitch-mode= (variable width font) 3. Use ~(setq org-hide-emphasis-markers t)~ 4. Configure a lot of org-mode specific faces to still use a monospaced font. Here are some images of the result. Notice one important factor of the feel is that I work on a mac with retina display. Often font rendering feel too bold by default. But this is perfect to have a writing environment even if screenshot does not look as slick as other ones, the usage is superior. attr_org: :width 560 attr_html: top caption: org-mode headers name: fig:top => ./top.png attr_org: :width 560 attr_html: img-with-caption caption: org-mode some inline image name: fig:img-with-caption => ./img-with-caption.png attr_org: :width 560 attr_html: code caption: org-mode with some code block name: fig:img-with-caption => ./code.png attr_org: :width 560 attr_html: Org mode with a modified doom-solarized light theme (use a grey background) caption: Org mode with a modified doom-solarized light theme (use a grey background) name: fig:nano-emacs => ./y-org-mode.png The main trick is to change org-mode to use different font depending on the kind of bloc. I use two fonts variant which are an iA Writer clone fonts; iM Writing Nerd Font. First you need to install nerd-fonts[fn:nerdfonts]. You will get that =iMWritingDuoS Nerd Font=. If you look at the code block; I support the case when the font is not installed and fall back to Georgia or PT Serif. One nice little bonus of the config is to make the fixed width fonts smaller. This is often something I like when writing in org-mode. There is a minor dependency on =doom= as I use =doom-color= for the color of the links. But you could easily use any color you like if you do not use doom.
(setq org-ellipsis " [+]")
(add-hook 'org-mode-hook 'variable-pitch-mode)
(let* ((variable-tuple
(cond
((x-list-fonts "iMWritingDuoS Nerd Font") '(:family "iMWritingDuoS Nerd Font"))
((x-list-fonts "Georgia") '(:family "Georgia"))
((x-list-fonts "PT Serif") '(:family "PT Serif"))))
(fixed-tuple
(cond
((x-list-fonts "iMWritingDuoS Nerd Font Mono") '(:family "iMWritingDuoS Nerd Font Mono" :height 160))
((x-list-fonts "Menlo") '(:family "Menlo" :height 120))
((x-list-fonts "PT Mono") '(:family "PT Mono" :height 120))))
(headline `(:inherit default :weight bold)))
(custom-theme-set-faces
'user
`(org-level-1 ((t (,@headline ,@variable-tuple))))
`(org-level-2 ((t (,@headline ,@variable-tuple))))
`(org-level-3 ((t (,@headline ,@variable-tuple))))
`(org-level-4 ((t (,@headline ,@variable-tuple))))
`(org-level-5 ((t (,@headline ,@variable-tuple))))
`(org-level-6 ((t (,@headline ,@variable-tuple))))
`(org-level-7 ((t (,@headline ,@variable-tuple))))
`(org-level-8 ((t (,@headline ,@variable-tuple))))
`(org-document-title ((t (,@headline ,@variable-tuple))))
`(variable-pitch ((t ,@variable-tuple)))
`(fixed-pitch ((t ,@fixed-tuple)))
'(org-ellipsis ((t (:inherit fixed-pitch :foreground "gray40" :underline nil))))
'(org-block ((t (:inherit fixed-pitch))))
'(org-block-begin-line ((t (:inherit fixed-pitch))))
'(org-block-end-line ((t (:inherit fixed-pitch))))
'(org-src ((t (:inherit fixed-pitch))))
'(org-properties ((t (:inherit fixed-pitch))))
'(org-code ((t (:inherit (shadow fixed-pitch)))))
'(org-date ((t (:inherit (shadow fixed-pitch)))))
'(org-document-info ((t (:inherit (shadow fixed-pitch)))))
'(org-document-info-keyword ((t (:inherit (shadow fixed-pitch)))))
'(org-drawer ((t (:inherit (shadow fixed-pitch)))))
'(org-indent ((t (:inherit (org-hide fixed-pitch)))))
`(org-link ((t (:inherit fixed-pitch :foreground ,(doom-color 'blue) :underline t))))
'(org-meta-line ((t (:inherit (font-lock-comment-face fixed-pitch)))))
'(org-property-value ((t (:inherit fixed-pitch))) t)
'(org-special-keyword ((t (:inherit (font-lock-comment-face fixed-pitch)))))
'(org-table ((t (:inherit fixed-pitch))))
'(org-tag ((t (:inherit (shadow fixed-pitch) :weight bold :height 0.8))))
'(org-verbatim ((t (:inherit (shadow fixed-pitch)))))))
[fn:nerdfonts] https://www.nerdfonts.com ## Digression about why I did that; For some reason a went to the rabbit hole of tweaking my emacs. In fact, it first started as; let's try to switch from =doom-emacs=[fn:doom-emacs] to =nano-emacs=[fn:nano-emacs]. But, doing so, I realized I wouldn't be able to reach the quality and optimization provided by doom-emacs myself. So instead of doing this, I first tried to copy the theme of nano. Then I realized one of the biggest factor of nano look & feel was its usage of "Roboto Mono" but with weight light (or Thin). See attr_org: :width 560 attr_html: nano-emacs look (light theme) caption: GNU Emacs / N Λ N O Look (light theme) name: fig:nano-emacs => ./nano-emacs-light.png attr_org: :width 560 attr_html: nano-emacs look (dark theme) caption: GNU Emacs / N Λ N O Look (dark theme) name: fig:nano-emacs => ./nano-emacs-dark.png OK so... I just tried to match the theme colors. It was easy to create a theme with matching colors.
(setq doom-font (font-spec :family "SauceCodePro Nerd Font Mono" :size 12 :weight 'semi-light)
doom-variable-pitch-font (font-spec :family "iMWritingDuoS Nerd Font" :size 14))
### An unfinished nano theme for doom Even though the result is not 100% satisfactory, you could start using my work. Save this file into =~/.doom.d/themes/doom-nano-theme.el=:
;;; doom-nano-theme.el --- inspired by Nicolas Rougier nano-theme -*- lexical-binding: t; no-byte-compile: t; -*-
;;
;; Author: Yann Esposito <https://yannesposito.com>
;; Created: August 16, 2021
;; Version: 1.0.0
;; Keywords: custom themes, faces
;; Homepage: https://github.com/hlissner/emacs-doom-themes
;; Package-Requires: ((emacs "25.1") (cl-lib "0.5") (doom-themes "2.2.1"))
;;
;;; Commentary:
;;
;; Ported from nano-theme: https://github.com/rougier/nano-theme
;;
;;; Code:
(require 'doom-themes)
;;; Variables
(defgroup doom-plain-theme nil
"Options for the `doom-plain' theme."
:group 'doom-themes)
(defcustom doom-plain-padded-modeline doom-themes-padded-modeline
"If non-nil, adds a 4px padding to the mode-line.
Can be an integer to determine the exact padding."
:group 'doom-plain-theme
:type '(or integer boolean))
;;
;;; Theme definition
(def-doom-theme doom-nano
"Theme inspired by Nicolas Rougier nano-theme"
;; name default/256/16
((nano-color-foreground '("#37474F")) ;; Blue Grey / L800
(nano-color-background '("#FFFFFF")) ;; White
(nano-color-highlight '("#FAFAFA")) ;; Very Light Grey
(nano-color-critical '("#FF6F00")) ;; Amber / L900
(nano-color-salient '("#673AB7")) ;; Deep Purple / L500
(nano-color-strong '("#000000")) ;; Black
(nano-color-popout '("#FFAB91")) ;; Deep Orange / L200
(nano-color-subtle '("#ECEFF1")) ;; Blue Grey / L50
(nano-color-faded '("#B0BEC5")) ;; Blue Grey / L200
(bg nano-color-background)
(bg-alt nano-color-highlight)
(base0 '("#18282f"))
(base1 '("#24323a"))
(base2 '("#556066"))
(base3 '("#6f787d"))
(base4 '("#8a9296"))
(base5 '("#a6acaf"))
(base6 '("#e7e8e9"))
(base7 '("#f6f6f6"))
(base8 '("#fafafa"))
(fg nano-color-foreground)
(fg-alt nano-color-faded)
(grey fg)
(red fg)
(blue fg)
(dark-blue fg)
(orange fg)
(green fg)
(teal fg)
(yellow fg)
(magenta fg)
(violet fg)
(cyan fg)
(dark-cyan fg)
;; face categories -- required for all themes
(highlight nano-color-salient)
(vertical-bar base5)
(selection nano-color-highlight)
(builtin nano-color-salient)
(comments nano-color-faded)
(doc-comments nano-color-faded)
(constants nano-color-strong)
(functions nano-color-salient)
(keywords nano-color-strong)
(methods nano-color-salient)
(operators nano-color-strong)
(type nano-color-strong)
(strings base0)
(variables base0)
(numbers base0)
(region base4)
(error nano-color-critical)
(warning nano-color-popout)
(success nano-color-salient)
(vc-modified nano-color-salient)
(vc-added fg-alt)
(vc-deleted nano-color-critical)
;; custom categories
(-modeline-pad
(when doom-plain-padded-modeline
(if (integerp doom-plain-padded-modeline) doom-plain-padded-modeline 4)))
(modeline-bg (doom-darken bg-alt 0.15))
(modeline-bg-alt (doom-darken bg-alt 0.1))
(modeline-bg-inactive (doom-darken bg-alt 0.1))
(modeline-bg-inactive-alt bg-alt)
(modeline-fg fg)
(modeline-fg-alt (doom-darken modeline-bg-inactive 0.35)))
;;;; Base theme face overrides
((error :underline `(:style wave :color ,error))
(warning :underline `(:style wave :color ,warning))
((font-lock-constant-face &override) :slant 'italic)
((font-lock-comment-face &override) :slant 'italic)
((font-lock-function-name-face &override) :slant 'italic)
((font-lock-type-face &override) :slant 'italic)
;;(hl-line :background base8)
((line-number &override) :foreground base3)
((line-number-current-line &override) :foreground base2)
(mode-line
:background modeline-bg :foreground modeline-fg
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg)))
(mode-line-inactive
:background modeline-bg-inactive :foreground modeline-fg-alt
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg-inactive)))
(mode-line-emphasis :foreground highlight)
;;;; doom-modeline
(doom-modeline-bar :background modeline-bg)
(doom-modeline-bar-inactive :inherit 'doom-modeline-bar)
(doom-modeline-project-dir :foreground fg)
(doom-modeline-buffer-file :foreground fg)
(doom-modeline-buffer-modified :weight 'bold :foreground "#000000")
(doom-modeline-panel :inherit 'mode-line-highlight :background base3 :foreground fg)
;;;; ivy
(ivy-posframe :background bg-alt)
;;;; magit
((magit-diff-added-highlight &override) :foreground fg :background (doom-blend vc-added bg 0.3))
((magit-diff-removed &override) :foreground (doom-lighten fg 0.4) :background (doom-blend vc-deleted bg 0.1))
((magit-diff-removed-highlight &override) :foreground fg :background (doom-blend vc-deleted bg 0.22))
;;;; lsp-mode
(lsp-headerline-breadcrumb-symbols-face :foreground keywords :weight 'bold)
;;;; outline <built-in>
(outline-1 :slant 'italic :foreground fg-alt)
(outline-2 :inherit 'outline-1 :foreground base2)
(outline-3 :inherit 'outline-2)
(outline-4 :inherit 'outline-3)
(outline-5 :inherit 'outline-4)
(outline-6 :inherit 'outline-5)
(outline-7 :inherit 'outline-6)
(outline-8 :inherit 'outline-7)
(org-level-1 :inherit 'org-level-1 :foreground nano-color-strong)
(org-level-2 :inherit 'org-level-2 :foreground nano-color-strong)
(org-level-3 :inherit 'org-level-3 :foreground nano-color-strong)
(org-level-4 :inherit 'org-level-4 :foreground nano-color-strong)
(org-level-5 :inherit 'org-level-5 :foreground nano-color-strong)
(org-level-6 :inherit 'org-level-6 :foreground nano-color-strong)
(org-level-7 :inherit 'org-level-7 :foreground nano-color-strong)
(org-level-8 :inherit 'org-level-8 :foreground nano-color-strong)
(org-code :inherit 'org-code
:foreground nano-color-salient
:weight 'bold)
(org-verbatim :inherit 'org-verbatim
:foreground nano-color-salient
:weight 'bold)
(org-upcoming-deadline :inherit 'org-upcoming-deadline
:foreground nano-color-critical
:weight 'bold)
(org-upcoming-distant-deadline :inherit 'org-upcoming-distant-deadline
:foreground nano-color-salient)
(org-habit-overdue-face
:inherit 'org-habit-overdue-face
:background fg-alt)
(org-habit-overdue-future-face
:inherit 'org-habit-overdue-future-face
:background nano-color-subtle)
(org-habit-alert-face
:inherit 'org-habit-alert-face
:background nano-color-critical)
(org-habit-alert-future-face
:inherit 'org-habit-alert-future-face
:background nano-color-subtle)
(org-scheduled-today :inherit 'org-scheduled-today :foreground fg)
(org-scheduled-previously :inherit 'org-scheduled-previously :foreground fg)
;;;; org <built-in>
((org-block &override) :background bg-alt)
((org-block-begin-line &override) :foreground base5)
;;;; solaire-mode
(solaire-mode-line-face
:inherit 'mode-line
:background modeline-bg-alt
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg-alt)))
(solaire-mode-line-inactive-face
:inherit 'mode-line-inactive
:background modeline-bg-inactive-alt
:box (if -modeline-pad `(:line-width ,-modeline-pad :color ,modeline-bg-inactive-alt)))))
;;; doom-plain-theme.el ends here
You will probably need more work to achieve the colors you expect. For that, using ~SPC-u C-x =~ will probably be useful. It will show the font face under the cursor. Best of luck. => /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> <item> <title>Utopia (2013)</title> <guid>gemini://her.esy.fun/posts/0019-utopia-tv-show/index.gmi</guid> <pubDate>Tue, 01 Jun 2021 00:00:00 +0200</pubDate> <category>tv-show</category> <description><![CDATA[ # Utopia (2013) description: The Utopia (2013-2014) British TV Show deserve to be known. description: In the age of COVID19 it is even more relevant. description: The filmography is magistral, as well as the soundtrack, description: acting, and overall atmosphere. description: I really urge you to at least take a look at the opening scene. keywords: tv-show author: Yann Esposito email: yann@esposito.host => /files/publickey.txt gpg date: [2021-06-01 Tue] lightbk: #ff0 darkbk: #880 attr_html: Utopia caption: Utopia => ./utopia-s01.jpg I wanted to write a few articles about great shows lot of people around me do not know about. => https://www.themoviedb.org/tv/46511-utopia Utopia (2013) is one of these TV Shows that deserve more attention. The *filmography* is quite original. I have never seen another film/TV show with a similar atmosphere. The usage of bright colors is magistral. Flashy yellow, green, red. The *soundtrack* is also pretty surprising original and enjoyable. The *acting* is really great. Actors are doing a great job and are quite relatable. These are not the (super)heroes you are used to. We are far away from shows where every girl is a bimbo and every guy is a Chippendale. The *scenario*, in regard to the recent events related to COVID19 is just perfect. I do not want to reveal too much. But let's just say the current real events are close to the events predicted in this show. The *Surrealistic Humor* atmosphere make the viewing experience quite exceptionnal. There is a mix between nonchalance and extreme violence. It really feels surrealist. In this show some people act with extreme violence as if there is no choice. As a conclusion, if you are looking for a very innovative TV show then search no further this one is a great original choice. If you are still unsure, just watch the opening scene, it is quite incredible. ps: Also try to get the original content. Amazon Prime apparently cut some very important scene and also changed the ratio which hurt the very good image work. => /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> <item> <title>Fast Static Site with make</title> <guid>gemini://her.esy.fun/posts/0018-makefile-as-static-site-builder-follow-up/index.gmi</guid> <pubDate>Tue, 25 May 2021 00:00:00 +0200</pubDate> <category>blog</category> <category>static</category> <description><![CDATA[ # Fast Static Site with make description: A deeper view of my static site builder Makefile keywords: blog static author: Yann Esposito email: yann@esposito.host => /files/publickey.txt gpg date: [2021-05-25 Tue] This article will dig a bit deeper about my =Makefile= based static website generator. In a => https://her.esy.fun/posts/0017-static-blog-builder/index.html previous article I just gave the rationale and an overview to do it yourself. Mainly it is very fast and portable. A few goals reached by my current build system are: 1. Be fast and make the minimal amount of work as possible. I don't want to rebuild all the html pages if I only change one file. 2. Source file format agnostic. You can use markdown, org-mode or even directly write html. 3. Support gemini 4. Optimize size: minify HTML, CSS, images 5. Generate an index page listing the posts 6. Generate RSS/atom feed (for both gemini and http) =make= will take care of handling the dependency graph to minimize the amount of effort when a change occurs in the sources. For some features, I built specifics small shell scripts. For example to be absolutely agnostic in the source format for my articles I generate the RSS out of a tree of HTML files. But taking advantage of =make=, I generate an index cache to transform those HTML into XML which will be faster to use to build different indexes. To make those transformations I use very short a shell scripts. # =Makefile= overview A Makefile is made out of rules. The first rule of your Makefile will be the default rule. The first rule of my Makefile is called =all=. A rule as the following format:
target: file1 file2
cmd --input file1 file2 \
--output target
if =target= does not exists, then =make= will look at its dependencies. If any of its dependencies need to be updated, it will run all the rules in the correct order to rebuild them and finally run the script to build =target=. A file needs to be updated if one of its dependency needs to be updated or is newer. The usual use case of =make= is about building a single binary out of many source files. But for a static website, we need to generate a lot of files from a lot of files. So we construct the rules like this:
all: site
DST_FILES := ....
ALL += $(DST_FILES)
DST_FILES_2 := ....
ALL += $(DST_FILES_2)
site: $(ALL)
In my =Makefile= I have many similar block with the same pattern. 1. I retrieve a list of source files 2. I construct the list of destination files (change the directory, the extension) 3. I declare a rule to construct these destination files 4. I add the destination files to the =ALL= variable. I have a block for:
SRC_ASSETS := $(shell find src -type f)
DST_ASSETS := $(patsubst src/%,_site/%,$(SRC_ASSETS))
_site/% : src/%
@mkdir -p "$(dir $@)"
cp "{body}lt;" "$@"
.PHONY: assets
assets: $(DST_ASSETS)
ALL += assets
OK, this looks terrible. But mainly:
_site/%.css: src/%.css
minify "{body}lt;" "$@"
And if the selected file is a =CSS= file, this rule will be selected. ## Prelude I start with variables declarations:
all: site
SRC_DIR ?= src
DST_DIR ?= _site
CACHE_DIR ?= .cache
NO_DRAFT := -not -path '$(SRC_DIR)/drafts/*'
NO_SRC_FILE := ! -name '*.org'
## CSS Here we go; the same simple pattern for CSS files.
SRC_CSS_FILES := $(shell find $(SRC_DIR) -type f -name '*.css')
DST_CSS_FILES := $(patsubst $(SRC_DIR)/%,$(DST_DIR)/%,$(SRC_RAW_FILES))
$(DST_DIR)/%.css : $(SRC_DIR)/%.css
@mkdir -p "$(dir $@)"
minify "{body}lt;" > "$@"
.PHONY: css
css: $(DST_CSS_FILES)
ALL += css
This is very similar to the block for raw assets. The difference is just that instead of using =cp= we use the =minify= command. ## ORG → HTML Now this one is more complex but is still follow the same pattern.
EXT ?= .org
SRC_PANDOC_FILES ?= $(shell find $(SRC_DIR) -type f -name "*$(EXT)" $(NO_DRAFT))
DST_PANDOC_FILES ?= $(patsubst %$(EXT),%.html, \
$(patsubst $(SRC_DIR)/%,$(DST_DIR)/%, \
$(SRC_PANDOC_FILES)))
PANDOC_TEMPLATE ?= templates/post.html
MK_HTML := engine/mk-html.sh
PANDOC := $(MK_HTML) $(PANDOC_TEMPLATE)
$(DST_DIR)/%.html: $(SRC_DIR)/%.org $(PANDOC_TEMPLATE) $(MK_HTML)
@mkdir -p "$(dir $@)"
$(PANDOC) "{body}lt;" "$@.tmp"
minify --mime text/html "$@.tmp" > "$@"
@rm "$@.tmp"
.PHONY: html
html: $(DST_PANDOC_FILES)
ALL += html
So to construct =DST_PANDOC_FILES= this time we also need to change the
extension of the file from =org= to =html=.
We need to provide a template that will be passed to pandoc.
And of course, as if we change the template file we would like to
regenerate all HTML files we put the template as a dependency.
But importantly *not* at the first place. Because we use ={body}lt;= that will be
the first dependency.
I also have a short script instead of directly using =pandoc=.
It is easier to handle =toc= using the metadatas in the file.
And if someday I want to put the template in the metas, this will be the
right place to put that.
The =mk-html.sh= is quite straightforward:
set -eu
cd "$(git rev-parse --show-toplevel)" || exit 1
template="$1"
orgfile="$2"
htmlfile="$3"
tocoption=""
if grep -ie '^#+options:' "$orgfile" | grep 'toc:t'>/dev/null; then
tocoption="--toc"
fi
set -x
pandoc $tocoption \
--template="$template" \
--mathml \
--from org \
--to html5 \
--standalone \
$orgfile \
--output "$htmlfile"
Once generated I also minify the html file. And, that's it. But the important part is that now, if I change my script or the template or the file, it will generate the dependencies. ## Indexes We often need indexes to build a website. Typically to list the latest articles, build the RSS file. So for sake of simplicity, I decided to build my index as a set of XML files. Of course, this could be optimizide, by using SQLite for example. But this will already be really fast. For every generated html file I will generate a clean XML file with =hxclean=. Once cleaned, it will be easy to access a specific node of in these XML files.
SRC_POSTS_DIR ?= $(SRC_DIR)/posts
DST_POSTS_DIR ?= $(DST_DIR)/posts
SRC_POSTS_FILES ?= $(shell find $(SRC_POSTS_DIR) -type f -name "*$(EXT)")
RSS_CACHE_DIR ?= $(CACHE_DIR)/rss
DST_XML_FILES ?= $(patsubst %.org,%.xml, \
$(patsubst $(SRC_POSTS_DIR)/%,$(RSS_CACHE_DIR)/%, \
$(SRC_POSTS_FILES)))
$(RSS_CACHE_DIR)/%.xml: $(DST_POSTS_DIR)/%.html
@mkdir -p "$(dir $@)"
hxclean "{body}lt;" > "$@"
.PHONY: indexcache
indexcache: $(DST_XML_FILES)
ALL += indexcache
This rule will generate for every file in =site/posts/*.html= a corresponding =xml= file (=hxclean= takes an HTML an try its best to make an XML out of it). ## HTML Index Now we just want to generate the main =index.html= page at the root of the site. This page should list all articles by date in reverse order. The first step is to take advantage of the cache index. For every XML file I generated before I should generate the small HTML block I want for every entry. For this I use a script =mk-index-entry.sh=. He will use =hxselect= to retrieve the date and the title from the cached XML files. Then generate a small file just containing the date and the link. Here is the block in the Makefile:
DST_INDEX_FILES ?= $(patsubst %.xml,%.index, $(DST_XML_FILES))
MK_INDEX_ENTRY := ./engine/mk-index-entry.sh
INDEX_CACHE_DIR ?= $(CACHE_DIR)/rss
$(INDEX_CACHE_DIR)/%.index: $(INDEX_CACHE_DIR)/%.xml $(MK_INDEX_ENTRY)
@mkdir -p $(INDEX_CACHE_DIR)
$(MK_INDEX_ENTRY) "{body}lt;" "$@"
It means: for every =.xml= file generate a =.index= file with =mk-index-entry.sh=.
cd "$(git rev-parse --show-toplevel)" || exit 1
xfic="$1"
dst="$2"
indexdir=".cache/rss"
dateaccessor='.yyydate'
titleaccessor='title'
finddate(){ < $1 hxselect -c $dateaccessor | sed 's/\[//g;s/\]//g;s/ .*$//' }
findtitle(){ < $1 hxselect -c $titleaccessor }
autoload -U colors && colors
blogfile="$(echo "$xfic"|sed 's#.xml$#.html#;s#^'$indexdir'/#posts/#')"
printf "%-30s" $blogfile
d=$(finddate $xfic)
echo -n " [$d]"
rssdate=$(formatdate $d)
title=$(findtitle $xfic)
keywords=( $(findkeywords $xfic) )
printf ": %-55s" "$title ($keywords)"
{ printf "\\n<li>"
printf "\\n<span class=\"pubDate\">%s</span>" "$d"
printf "\\n<a href=\"%s\">%s</a>" "${blogfile}" "$title"
printf "\\n</li>\\n\\n"
} >> ${dst}
echo " [${fg[green]}OK${reset_color}]"
Then I use these intermediate files to generate a single bigger index file.
HTML_INDEX := $(DST_DIR)/index.html
MKINDEX := engine/mk-index.sh
INDEX_TEMPLATE ?= templates/index.html
$(HTML_INDEX): $(DST_INDEX_FILES) $(MKINDEX) $(INDEX_TEMPLATE)
@mkdir -p $(DST_DIR)
$(MKINDEX)
.PHONY: index
index: $(HTML_INDEX)
ALL += index
This script is a big one, but it is not that complex. For every file, I generate a new file =DATE-dirname=. I sort them in reverse order and put their content in the middle of an HTML file. Important note: this file updates only if the index change. The first part of the script creates files with the creation date in their metadatas. The created file name will contain the creation date, this will be helpful later.
autoload -U colors && colors
cd "$(git rev-parse --show-toplevel)" || exit 1
webdir="_site"
indexfile="$webdir/index.html"
indexdir=".cache/rss"
tmpdir=$(mktemp -d)
echo "Publishing"
dateaccessor='.pubDate'
finddate(){ < $1 hxselect -c $dateaccessor }
for fic in $indexdir/**/*.index; do
d=$(finddate $fic)
echo "${${fic:h}:t} [$d]"
cp $fic $tmpdir/$d-${${fic:h}:t}.index
done
Then I use these files to generate a file that will contain the =body= of the HTML.
previousyear=""
for fic in $(ls $tmpdir/*.index | sort -r); do
d=$(finddate $fic)
year=$( echo "$d" | perl -pe 's#(\d{4})-.*#$1#')
if (( year != previousyear )); then
if (( previousyear > 0 )); then
echo "</ul>" >> $tmpdir/index
fi
previousyear=$year
echo "<h3 name=\"${year}\" >${year}</h3><ul>" >> $tmpdir/index
fi
cat $fic >> $tmpdir/index
done
echo "</ul>" >> $tmpdir/index
And finally, I render the HTML using a template within a shell script:
title="Y"
description="Most recent articles"
author="Yann Esposito"
body=$(< $tmpdir/index)
date=$(LC_TIME=en_US date +'%Y-%m-%d')
template=$(< templates/index.html | \
sed 's/\$\(header-includes\|table-of-content\)\$//' | \
sed 's/\$if.*\$//' | \
perl -pe 's#(\$[^\$]*)\$#$1#g' )
{
export title
export author
export description
export date
export body
echo ${template} | envsubst
} > "$indexfile"
rm -rf $tmpdir
echo "* HTML INDEX [done]"
## RSS My RSS generation is similar to the system I used to generate the index file. I just slightly improved the rules. The =Makefile= blocks look like:
DST_RSS_FILES ?= $(patsubst %.xml,%.rss, $(DST_XML_FILES))
MK_RSS_ENTRY := ./engine/mk-rss-entry.sh
$(RSS_CACHE_DIR)/%.rss: $(RSS_CACHE_DIR)/%.xml $(MK_RSS_ENTRY)
@mkdir -p $(RSS_CACHE_DIR)
$(MK_RSS_ENTRY) "{body}lt;" "$@"
RSS := $(DST_DIR)/rss.xml
MKRSS := engine/mkrss.sh
$(RSS): $(DST_RSS_FILES) $(MKRSS)
$(MKRSS)
.PHONY: rss
rss: $(RSS)
ALL += rss
## Gemini I wrote a minimal script to transform my org files to gemini files. I also need to generate an index and an atom file for gemini:
EXT := .org
SRC_GMI_FILES ?= $(shell find $(SRC_DIR) -type f -name "*$(EXT)" $(NO_DRAFT))
DST_GMI_FILES ?= $(subst $(EXT),.gmi, \
$(patsubst $(SRC_DIR)/%,$(DST_DIR)/%, \
$(SRC_GMI_FILES)))
GMI := engine/org2gemini.sh
$(DST_DIR)/%.gmi: $(SRC_DIR)/%.org $(GMI) engine/org2gemini_step1.sh
@mkdir -p $(dir $@)
$(GMI) "{body}lt;" "$@"
ALL += $(DST_GMI_FILES)
.PHONY: gmi
gmi: $(DST_GMI_FILES)
GMI_INDEX := $(DST_DIR)/index.gmi
MK_GMI_INDEX := engine/mk-gemini-index.sh
$(GMI_INDEX): $(DST_GMI_FILES) $(MK_GMI_INDEX)
@mkdir -p $(DST_DIR)
$(MK_GMI_INDEX)
ALL += $(GMI_INDEX)
.PHONY: gmi-index
gmi-index: $(GMI_INDEX)
GEM_ATOM := $(DST_DIR)/gem-atom.xml
MK_GEMINI_ATOM := engine/mk-gemini-atom.sh
$(GEM_ATOM): $(DST_GMI_FILES) $(MK_GEMINI_ATOM)
$(MK_GEMINI_ATOM)
ALL += $(GEM_ATOM)
.PHONY: gmi-atom
gmi-atom: $(GMI_ATOM)
.PHONY: gemini
gemini: $(DST_GMI_FILES) $(GMI_INDEX) $(GEM_ATOM)
## Images For images, I try to compress them all with imagemagick.
SRC_IMG_FILES ?= $(shell find $(SRC_DIR) -type f -name "*.jpg" -or -name "*.jpeg" -or -name "*.gif" -or -name "*.png")
DST_IMG_FILES ?= $(patsubst $(SRC_DIR)/%,$(DST_DIR)/%, $(SRC_IMG_FILES))
$(DST_DIR)/%.jpg: $(SRC_DIR)/%.jpg
@mkdir -p $(dir $@)
convert "{body}lt;" -quality 50 -resize 800x800\> "$@"
$(DST_DIR)/%.jpg: $(SRC_DIR)/%.jpeg
@mkdir -p $(dir $@)
convert "{body}lt;" -quality 50 -resize 800x800\> "$@"
$(DST_DIR)/%.gif: $(SRC_DIR)/%.gif
@mkdir -p $(dir $@)
convert "{body}lt;" -quality 50 -resize 800x800\> "$@"
$(DST_DIR)/%.png: $(SRC_DIR)/%.png
@mkdir -p $(dir $@)
convert "{body}lt;" -quality 50 -resize 800x800\> "$@"
.PHONY: img
img: $(DST_IMG_FILES)
ALL += $(DST_IMG_FILES)
## Deploy A nice bonus is that I also deploy my website using make.
.PHONY: site
site: $(ALL)
.PHONY: deploy
deploy: $(ALL)
engine/sync.sh
.PHONY: clean
clean:
-[ ! -z "$(DST_DIR)" ] && rm -rf $(DST_DIR)/*
-[ ! -z "$(CACHE_DIR)" ] && rm -rf $(CACHE_DIR)/*
=> /index.gmi Home => /gem-atom.xml Feed => /slides.gmi Slides => /about-me.gmi About => https://gitea.esy.fun code => https://espial.esy.fun/u:yogsototh bookmarks => https://espial.esy.fun/u:yogsototh/notes notes ]]></description> </item> </channel> </rss>