💾 Archived View for thebird.nl › gn-gemtext-threads › topics › installation.gmi captured on 2023-07-10 at 13:53:41. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
This document is WIP and still a mixture of old and new docs.
Large system deployments can get very complex. In this document we explain the GeneNetwork reproducible deployment system which is based on GNU Guix The Guix system can be used to install GN with all its files and dependencies.
Note that the official deployment works through a Guix VM. This is described in
To run GeneNetwork the following services need to function:
Make sure to install GNU Guix using the binary download instructions on the main website. Follow the instructions on Note the download amounts to several GBs of data. Debian-derived distros may support
apt-get install guix
We run a GNU Guix channel with packages at
https://gitlab.com/genenetwork/guix-bioinformatics
The README has instructions hosting a channel (recommended!), but sometimes we use the GUIX_PACKAGE_PATH instead. First upgrade to a recent guix with
mkdir ~/opt guix pull -p ~/opt/guix-pull
It should upgrade (ignore the locales warnings). You can optionally specify the specific git checkout of guix with
guix pull -p ~/opt/guix-pull --commit=f04883d
which is useful when you need to roll back to an earlier version (sometimes our channel goes out of sync). Next, we install GeneNetwork2 with
source ~/opt/guix-pull/etc/profile git clone https://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics.git ~/guix-bioinformatics
you probably also need guix-past (the upstream channel for older packages):
git clone https://gitlab.inria.fr/guix-hpc/guix-past.git ~/guix-past cd ~/guix-past env GUIX_PACKAGE_PATH=$HOME/guix-bioinformatics:$HOME/guix-past/modules ~/opt/guix-pull/bin/guix package -i genenetwork2 -p ~/opt/genenetwork2
Ignore the warnings. Guix should install the software without trying to build everything. If you system insists on building all packages, try the `--dry-run` switch and fix the [[https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html][substitutes]]. You may add the `--substitute-urls="http://guix.genenetwork.org https://ci.guix.gnu.org https://mirror.hydra.gnu.org"` switch.
The guix.genenetwork.org has most of our packages pre-built(!). To use it on your own machine the public key is
(public-key (ecc (curve Ed25519) (q #9F56EAB5CE37AA15693C31F451140588240F259676C137E31C0CA70EC4D1B534#) ) )
Once we have a GNU Guix profile, a running database (see below) and the file storage,
we should be ready to fire up GeneNetwork:
Check out the source with git:
git clone git@github.com:genenetwork/genenetwork2.git cd genenetwork2
You may want to use the testing branch.
Run GN2 with earlier created Guix profile
export GN2_PROFILE=$HOME/opt/genenetwork2 env TMPDIR=$HOME/tmp WEBSERVER_MODE=DEBUG LOG_LEVEL=DEBUG SERVER_PORT=5012 GENENETWORK_FILES=/export/data/genenetwork/genotype_files SQL_URI=mysql://webqtlout:webqtlout@localhost/db_webqtl ./bin/genenetwork2 etc/default_settings.py -gunicorn-dev
The script comes with debug and logging switches can be particularly useful when
developing GN2. Location and files are examples.
It may be useful to tunnel the web server to your local browser with an ssh tunnel:
If you want to test a service running on the server on a certain port (say 8202) use
ssh -L 8202:127.0.0.1:8202 -f -N myname@penguin2.genenetwork.org
And browse on your local machine to http://localhost:8202/
GeneNetwork requires a separate gn-proxy server which handles
authorisation and access control. For instructions see the
[[https://github.com/genenetwork/gn-proxy][README]]. Note it may already be running on our servers!
Redis part of GN2 deployment and will be started by the ./bin/genenetwork2
startup script.
These are the steps you can take to install a fresh installation of
mariadb (which comes as part of the GNU Guix genenetwork2 install).
As root configure the Guix profile
: . ~/opt/genenetwork2/etc/profile
and run for example
adduser mariadb && addgroup mariadb
mkdir -p /export2/mariadb/database
chown mariadb.mariadb -R /export2/mariadb/
mkdir -p /var/run/mysqld
chown mariadb.mariadb /var/run/mysqld
su mariadb
mysql --version
mysql Ver 15.1 Distrib 10.1.45-MariaDB, for Linux (x86_64) using readline 5.1
mysql_install_db --user=mariadb --datadir=/export2/mariadb/database
mysqld -u mariadb --datadir=/exportdb/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048"
If you want to run as root you may have to set
: /etc/my.cnf
: [mariadbd]
: user=root
You also need to set
: ft_min_word_len = 3
To make sure word text searches (shh) work and rebuild the tables if
required.
To check error output in a file on start-up run with something like
: mariadbd -u mariadb --console --explicit_defaults_for_timestamp --datadir=/gnu/mariadb --log-error=~/test.log
Other tips are that Guix installs mariadbd in your profile, so this may work
: /home/user/.guix-profile/bin/mariadbd -u mariadb --explicit_defaults_for_timestamp --datadir=/gnu/mariadb
When you get errors like:
: qlalchemy.exc.IntegrityError: (_mariadb_exceptions.IntegrityError) (1215, 'Cannot add foreign key constraint')
you may need to set
: set foreign_key_checks=0
At this point we require the underlying distribution to install and
run mysqld (see next section for GNU Guix). Currently we have two databases for deployment,
'db_webqtl_s' is the small testing database containing experiments
from BXD mice and 'db_webqtl_plant' which contains all plant related
material.
Download one database from
http://ipfs.genenetwork.org/ipfs/QmRUmYu6ogxEdzZeE8PuXMGCDa8M3y2uFcfo4zqQRbpxtk
After installation unzip the database binary in the MySQL directory
cd ~/mysql
p7zip -d db_webqtl_s.7z
chown -R mysql:mysql db_webqtl_s/
chmod 700 db_webqtl_s/
chmod 660 db_webqtl_s/*
restart MySQL service (mysqld). Login as root
: mysql_upgrade -u root --force
: myslq -u root
and
: mysql> show databases;
: +--------------------+
: | Database |
: +--------------------+
: | information_schema |
: | db_webqtl_s |
: | mysql |
: | performance_schema |
: +--------------------+
Set permissions and match password in your settings file below:
: mysql> grant all privileges on db_webqtl_s.* to gn2@"localhost" identified by 'webqtl';
You may need to change "localhost" to whatever domain you are
connecting from (mysql will give an error).
Note that if the mysql connection is not working, try connecting to
the IP address and check server firewall, hosts.allow and mysql IP
configuration (see below).
Note for the plant database you can rename it to db_webqtl_s, or
change the settings in etc/default_settings.py to match your path.
The script looks for genotype files. You can find them in
http://ipfs.genenetwork.org/ipfs/QmXQy3DAUWJuYxubLHLkPMNCEVq1oV7844xWG2d1GSPFPL
mkdir -p $HOME/genotype_files
cd $HOME/genotype_files
List of all runtime dependencies for GN2 as installed by GNU Guix.
https://genenetwork.org/environments/
See [[development.org]].
If you want to understand the architecture of GN2 read
[[Architecture.org]]. The rest of this document is mostly on deployment
of GN2.
If you have all the Guix packages installed this error points out that
the environment variables are not set. Copy-paste the paths into your
terminal (mainly so PYTHON_PATH and R_LIBS_SITE are set) from the
information given by guix:
: guix package --search-paths
On one system:
: export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages"
: export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
: export GEM_PATH="$HOME/.guix-profile/lib/ruby/gems/2.2.0"
and perhaps a few more.
The default settings file looks in your $HOME/gn2_data. Since these
files come with a Guix installation you should take a hint from the
values in the installed version of default_settings.py (see above in
this document).
You can use the GENENETWORK_FILES switch to set the datadir, for example
: env GN2_PROFILE=~/opt/gn-latest GENENETWORK_FILES=/gnu/data/gn2_data ./bin/genenetwork2
In rare cases, development modules are not brought in with Guix
because no source code is available. This can lead to missing modules
on a running server. Please check with the authors when a module
is missing.
This error
: __show = rpy2.rinterface.baseenv.get("show")
: LookupError: 'show' not found
means that R was updated in your path, and that Rpy2 needs to be
recompiled against this R - don't you love informative messages?
In our case it means that GN's PYTHONPATH is not in sync with
R_LIBS_SITE. Please check your GNU Guix GN2 installation paths,
you man need to reinstall. Note that this may be the point you
may want to start using profiles (see profile section).
The following error
: sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, 'Can\'t connect to local MySQL server through socket \'/run/mysqld/mysqld.sock\' (2 "No such file or directory")')
means that MySQL is trying to connect locally to a non-existent MySQL
server, something you may see in a container. Typically replicated with something like
: mysql -h localhost
try to connect over the network interface instead, e.g.
: mysql -h 127.0.0.1
if that works run genenetwork after setting SQL_URI to something like
: export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s
Let's see how fast we can deploy a second copy of GN2.
- [ ] Base install
+ [ ] First install a Debian server with GNU Guix on board
+ [ ] Get Guix build going
- [ ] Build the correct version of Guix
- [ ] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics
- [ ] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable
+ [ ] Create a gn2 user and home with space
+ [ ] Install redis
- [ ] add to systemd
- [ ] update redis.cnf
- [ ] update database
+ [ ] Install mariadb (currently debian mariadb-server)
- [ ] add to systemd
- [ ] system stop mysql
- [ ] update mysql.cnf
- [ ] update database (see gn-services/services/mariadb.md)
- [ ] check tables
+ [ ] run gn2
+ [ ] update nginx
+ [ ] install genenetwork3
- [ ] add to systemd