Entries Tagged 'guix' ↓

GuixSD: mastodon. An experiment in theoretic packaging. (very early draft/thoughtcollection)

Today we are going to “package” Mastodon, as requested. Or at least document the theory before being put to practice (I actually lack the resources to run a public test node).


We are going the “No Docker” route here first:

  • NodeJS
  • imagemagick
  • ffmpeg
  • libpq-dev
  • libxml2-dev
  • libxslt1-dev
  • nodejs
  • file
  • git
  • curl
  • curl -sL https://deb.nodesource.com/setup_4.x | sudo bash –
  • sudo apt-get install nodejs
  • sudo npm install -g yarn
  • sudo apt-get install redis-server redis-tools
  • sudo apt-get install postgresql postgresql-contrib
  • pidentd
  • git (to check out the source)
  • It is recommended to install “rbenv”. I’m not a ruby hacker, but I’m pretty sure with Guix if we manage to roll out a package for mastodon, we don’t need this:

    Use rbenv to pick a Ruby version for your application and guarantee that your development environment matches production. Put rbenv to work with Bundler for painless Ruby upgrades and bulletproof deployments.

    as Guix can use all kinds of versions in parallel.
    It is suggested to roll with ruby 2.4.1

  • gem install bundler
  • bundle install –deployment –without development test
  • yarn install

Missing packages in Guix: pidentd, node 4.x. We do have node 6.x though and 4.x (in theory) is only a matter of getting back to it.


And set up the database for the first time, this will create the tables and basic data:
RAILS_ENV=production bundle exec rails db:setup
Finally, pre-compile all CSS and JavaScript files:
RAILS_ENV=production bundle exec rails assets:precompile

Breaking up the language specific requirements

(as this is the Guix way, unbundling where possible)
gem ‘pkg-config’
gem ‘rails’, ‘~> 5.0.2’
gem ‘sass-rails’, ‘~> 5.0’
gem ‘uglifier’, ‘>= 1.3.0’
gem ‘jquery-rails’
gem ‘puma’
gem ‘hamlit-rails’
gem ‘pg’
gem ‘pghero’
gem ‘dotenv-rails’
gem ‘font-awesome-rails’
gem ‘best_in_place’, ‘~> 3.0.1’
gem ‘paperclip’, ‘~> 5.1’
gem ‘paperclip-av-transcoder’
gem ‘aws-sdk’, ‘>= 2.0’
gem ‘addressable’
gem ‘devise’
gem ‘devise-two-factor’
gem ‘doorkeeper’
gem ‘fast_blank’
gem ‘goldfinger’
gem ‘hiredis’
gem ‘htmlentities’
gem ‘http’
gem ‘http_accept_language’
gem ‘httplog’
gem ‘kaminari’
gem ‘link_header’
gem ‘nokogiri’
gem ‘oj’
gem ‘ostatus2’, ‘~> 1.1’
gem ‘ox’
gem ‘rabl’
gem ‘rack-attack’
gem ‘rack-cors’, require: ‘rack/cors’
gem ‘rack-timeout’
gem ‘rails-i18n’
gem ‘rails-settings-cached’
gem ‘redis’, ‘~>3.2’, require: [‘redis’, ‘redis/connection/hiredis’]
gem ‘rqrcode’
gem ‘ruby-oembed’, require: ‘oembed’
gem ‘sidekiq’
gem ‘sidekiq-unique-jobs’
gem ‘simple-navigation’
gem ‘simple_form’
gem ‘sprockets-rails’, :require => ‘sprockets/railtie’
gem ‘statsd-instrument’
gem ‘twitter-text’
gem ‘tzinfo-data’
gem ‘whatlanguage’
gem ‘react-rails’
gem ‘browserify-rails’
gem ‘autoprefixer-rails’
gem ‘rails_12factor’
gem ‘redis-rails’
gem ‘lograge’

Most of these gems and their dependencies are not available. So it’s mostly a matter of spending time to package gems.

For npm / nodejs the current recommendation seems to be: Give up, just embed a large binary blob for each direct dependency.


1. Packaging mastodon is not trivial.
2. Packaging the gems: I stopped setting deadlines in packaging, but my first impression is that it will take long.
3. The other effort will be to get the npm build system merged or get enough info so that we can rebase and use a branch based on jelle’s GSoC branch. Get more info on the state of it, etc.



GuixSD on servers: How to run a tor-relay (draft)

Obligatory Disclaimer: Read the upstream documentation and notices, especially the legal ones, about running tor relays.

Today in GuixSD on servers, we are going to run a tor-relay.

I assume you already went through the joy of installing a basic GuixSD somehow, somewhere and all you need is a system config which gives some insight into how to run this one-service specific server.
With the recent introduction of containers for system services in GuixSD, this adds some safety to running tor relays (not legal one though).

Please keep in mind that this is a theoretic construction, it will be put to practice once I can run a relay again for a short moment.

It also features no notice of adding your OpenSSH keys to the machine + setting a password.
Someone is working on making SSH keys an option in the system config, this post will be updated once it’s possible.

(use-modules (gnu)
	     (gnu system nss)
	     (srfi srfi-1)
	     (srfi srfi-26) ; cut
	     (guix store) ; %default-substitute-urls
	     (gnu services base)) ; %default-authorized-guix-keys
(use-service-modules desktop networking base ssh
		     version-control xorg avahi
		     dbus admin mcron)
(use-package-modules certs gnome vim
		     tor version-control admin
		     ntp suckless xdisorg
		     linux wget web
		     package-management tls ssh)

(define %tordate
  #~(job '(next-hour '(4))
	 (string-append #$tlsdate "/bin/tlsdate -l -t -v -V")))

  (host-name "noisemachine")
  (timezone "UTC")
  (locale "en_US.UTF-8")

  ;; Assuming /dev/sda is the target hard disk, and "my-root"
  ;; is the label of the target root file system.
  (bootloader (grub-configuration (device "/dev/sda")))

  (file-systems (cons (file-system
                        (device "/dev/sda2")
                        (title 'device)
                        (mount-point "/")
                        (type "ext4"))

  ;; Assuming you have created a swap file named "swap.file" in the root of "/".
  (swap-devices '("/swap.file"))

  (users (cons (user-account
                (name "user")
                (comment "")
                (group "users")
                (supplementary-groups '("wheel"))
                (home-directory "/home/user"))

  (packages (cons* nss-certs         ;for HTTPS access
                   nyx tor torsocks

  (services (cons* (service rottlog-service-type (rottlog-configuration))
		   (service mcron-service-type
			      (jobs (list %tordate))))
                    (plain-file "torrc"
                                "#### This section is just for relays ####\n
## See https://www.torproject.org/docs/tor-doc-relay for details.\n
## Required: what port to advertise for incoming Tor connections.\n
ORPort 9001\n
## If you want to listen on a port other than the one advertised in\n
## ORPort (e.g. to advertise 443 but bind to 9090), you can do it as\n
## follows.  You'll need to do ipchains or other port forwarding\n
## yourself to make this work.\n
#ORPort 443 NoListen\n
#ORPort NoAdvertise\n
## The IP address or full DNS name for incoming connections to your\n
## relay. Leave commented out and Tor will guess.\n
#Address noname.example.com\n
## If you have multiple network interfaces, you can specify one for\n
## outgoing traffic to use.\n
## OutboundBindAddressExit will be used for all exit traffic, while\n
## OutboundBindAddressOR will be used for all other connections.\n
## If you do not wish to differentiate, use OutboundBindAddress to\n
## specify the same address for both in a single line.\n
## A handle for your relay, so people don't have to refer to it by key.\n
## Nicknames must be between 1 and 19 characters inclusive, and must\n
## contain only the characters [a-zA-Z0-9].\n
Nickname heckthecistem\n
## Define these to limit how much relayed traffic you will allow. Your\n
## own traffic is still unthrottled. Note that RelayBandwidthRate must\n
## be at least 75 kilobytes per second.\n
## Note that units for these config options are bytes (per second), not\n
## bits (per second), and that prefixes are binary prefixes, i.e. 2^10,\n
## 2^20, etc.\n
#RelayBandwidthRate 100 KBytes  # Throttle traffic to 100KB/s (800Kbps)\n
#RelayBandwidthBurst 200 KBytes # But allow bursts up to 200KB (1600Kb)\n
## Use these to restrict the maximum traffic per day, week, or month.\n
## Note that this threshold applies separately to sent and received bytes,\n
## not to their sum: setting "40 GB" may allow up to 80 GB total before\n
## hibernating.\n
## Set a maximum of 40 gigabytes each way per period.\n
#AccountingMax 40 GBytes\n
## Each period starts daily at midnight (AccountingMax is per day)\n
#AccountingStart day 00:00\n
## Each period starts on the 3rd of the month at 15:00 (AccountingMax\n
## is per month)\n
#AccountingStart month 3 15:00\n
## Administrative contact information for this relay or bridge. This line\n
## can be used to contact you if your relay or bridge is misconfigured or\n
## something else goes wrong. Note that we archive and publish all\n
## descriptors containing these lines and that Google indexes them, so\n
## spammers might also collect them. You may want to obscure the fact that\n
## it's an email address and/or generate a new address for this purpose.\n
ContactInfo Random Person \n
## You might also include your PGP or GPG fingerprint if you have one:\n
#ContactInfo 0xFFFFFFFF Random Person \n
## Uncomment this to mirror directory information for others. Please do\n
## if you have enough bandwidth.\n
DirPort 9030 # what port to advertise for directory connections\n
## If you want to listen on a port other than the one advertised in\n
## DirPort (e.g. to advertise 80 but bind to 9091), you can do it as\n
## follows.  below too. You'll need to do ipchains or other port\n
## forwarding yourself to make this work.\n
#DirPort 80 NoListen\n
#DirPort NoAdvertise\n
## Uncomment to return an arbitrary blob of html on your DirPort. Now you\n
## can explain what Tor is if anybody wonders why your IP address is\n
## contacting them. See contrib/tor-exit-notice.html in Tor's source\n
## distribution for a sample.\n
DirPortFrontPage /var/www/tor-exit-notice.html\n
## Uncomment this if you run more than one Tor relay, and add the identity\n
## key fingerprint of each Tor relay you control, even if they're on\n
## different networks. You declare it here so Tor clients can avoid\n
## using more than one of your relays in a single circuit. See\n
## https://www.torproject.org/docs/faq#MultipleRelays\n
## However, you should never include a bridge's fingerprint here, as it would\n
## break its concealability and potentially reveal its IP/TCP address.\n
#MyFamily $keyid,$keyid,...\n
## A comma-separated list of exit policies. They're considered first\n
## to last, and the first match wins.\n
## If you want to allow the same ports on IPv4 and IPv6, write your rules\n
## using accept/reject *. If you want to allow different ports on IPv4 and\n
## IPv6, write your IPv6 rules using accept6/reject6 *6, and your IPv4 rules\n
## using accept/reject *4.\n
## If you want to _replace_ the default exit policy, end this with either a\n
## reject *:* or an accept *:*. Otherwise, you're _augmenting_ (prepending to)\n
## the default exit policy. Leave commented to just use the default, which is\n
## described in the man page or at\n
## https://www.torproject.org/documentation.html\n
## Look at https://www.torproject.org/faq-abuse.html#TypicalAbuses\n
## for issues you might encounter if you use the default exit policy.\n
## If certain IPs and ports are blocked externally, e.g. by your firewall,\n
## you should update your exit policy to reflect this -- otherwise Tor\n
## users will be told that those destinations are down.\n
## For security, by default Tor rejects connections to private (local)\n
## networks, including to the configured primary public IPv4 and IPv6 addresses,\n
## and any public IPv4 and IPv6 addresses on any interface on the relay.\n
## See the man page entry for ExitPolicyRejectPrivate if you want to allow\n
## "exit enclaving".\n
#ExitPolicy accept *:6660-6667,reject *:* # allow irc ports on IPv4 and IPv6 but no more\n
#ExitPolicy accept *:119 # accept nntp ports on IPv4 and IPv6 as well as default exit policy\n
#ExitPolicy accept *4:119 # accept nntp ports on IPv4 only as well as default exit policy\n
#ExitPolicy accept6 *6:119 # accept nntp ports on IPv6 only as well as default exit policy\n
ExitPolicy reject *:* # no exits allowed\n"))

GuixSD: further thoughts about servers

Multiple people are documenting their efforts and fights with clouds, kvm, xen, and other virtual systems to run GuixSD on.
Most of these platforms don’t differ from each other very much, so this serves also as a test to improve GuixSD to make the individual tweaks unnecessary.

However in the meantime, a centralized place to gather all the documentation is important. I’ll write down my own attempts, and I will collect emails from the list about initial server setups. For now I will collect them here on this web log.

Currently I’m working on:

Limitations are:

  • time … I’m bootstraping/starting pragmatique for the next months
  • money … some platforms, companies, and other entities are located in regions where running servers can’t be done for the (usual) unfair price of $5/month. Speaking of money: Rejoice hosters of virtual servers, we will throw cash at you for testing a system!

Eventually this should end up in a series of finalized posts which are easy to read (easy, non-technical language) and extend or work together with the pretty good documentation of Guix. You wonder how to install GuixSD at your hoster/cloudcastle XY? No problem, read this. Ideally it will be like my current experience with GuixSD where the basic setup is done in less than 30 minutes (depending on the status of hydra).
Eventually these guides will be moved over to some subdomain at pragmatique.xyz

Installing GuixSD… the Borg way

We are the Guix. Your biological and technological distinctiveness will
be added to our own. Resistance is futile.

This post briefly describes how to install GuixSD on Debian 8, the Borg way.
Please be aware that this post is a draft as the system I am using is still building. Currently this means what exactly you need to remove from /etc/ must be documented otherwise you boot into a non-functional system with mounting issues.

  • Captain obvious has a message before we start: make sure that your hardware is supported by linux-libre. Read this for more info. If not, other procedures might apply for getting specific parts to work, which will not be covered here.
  • Get the net-boot image for your architecture (obviously one that is supported by GuixSD: i686, x86_64, armv8 is currently work in progress and not done) from cdimage.debian.org
    The obvious steps apply, verify the signature etc. lessons learned: Due to e2fsprogs 1.42 in Guix, you will encounter problems when you use the guided setup of Debian 9 (testing), where they already use 1.43. We are working on rolling out the update.
  • Fast-forward through the install process: select non-graphical setup, timezone, etc etc setup root account, setup user account, deselect everything except ssh service and the basic applications needed, do the grub thing, reboot. Nothing which should be documented in detail.
  • ssh into the machine with the user account, su.
  • Follow the Guix Binary Installation. Use https://alpha.gnu.org instead of ftp:// if you want to.
  • Start the guix-daemon if not already running, authorize hydra.gnu.org for binary substitutes. These steps are still covered by the installation document.
  • guix pull (update guix and all things attached and included)
  • Write your config.scm for your system. This assumes you read the Guix documentation. Obviously you want to include the (openssh-service). There are example system files included, refer to the documentation of Guix for their location.
  • Remove the leftovers from the Debian installation. Read the bottom part of what wingo wrote in this email.
    Basically you do:

    • # mv /etc /etc-old; mkdir /etc; cp /etc-old/{passwd,group,shadow,gshadow,mtab} /etc/; rm /etc/mtab; cp -R /etc-old/guix /etc/; cd /etc; ln -s /proc/self/mounts mtab

    and then you run guix system reconfigure config.scm

  • make sure if this is a server somewhere remote, that you copy over your pubkey at least into /root/.ssh/authorized_keys. Reason: root has no password set by default in GuixSD, created users have no password set by default (you need to set them with “passwd”). This is with the assumption that it works like this in theory. With this first run I was able to fix things using the keyboard connected to the debian 8 -> GuixSD machine.
  • Reboot.
  • reconnect

I take packaging requests (for Guix packages)

If you lack the time to work on a package and you have no specific deadline for it,
you can message me with your requests.
Please mark the email with accordingly so that I can filter them later on, I suggest starting the subject line with “[packaging request]”.
If you do this, please support me on either flattr or patreon (more platforms will eventually be added).

Currently I work on an 1:1 port of torbrowser for Guix (longterm fun project), and completion of the mailman3 packages. Other work in progress packages can be found here.