15  Docker Swarm

(ns docker-swarm)

We use Docker Swarm, as opposed to Kubernetes. Docker Swarm has been chosen due to our smaller needs, whereas Kubernetes has been deemed too big and complex.

15.1 Re-deployment

A small manual script for re-deploying the Docker swarm.

• Enable maintenance page
  • Edit /var/www/html/index.html on oiiku-proddb and update the maintenence window
  • Edit ansible/roles/haproxy-configure/templates/haproxy.production.cfg.j2 and set maintenence to be on
  • ansible-playbook -i inventory-production.yml configure-haproxy.yml -e "@secrets-production.yml" --extra-vars "hosts=all" --limit=oiiku_proddb
• Verify that wireguard is up and running
• Drain all Swarm nodes: docker node ls --format "{{.ID}}" | xargs docker node update --availability drain
• Remove stack: docker stack rm production
• docker node ls for a list of all nodes
• Demote and remove non-leaders from swarm:
  • On leader: docker node demote <node-id1> <node-id2> ...
  • On non-leaders: docker swarm leave
  • On leader: docker node rm <node-id1> <node-id2> ...
• Remove leader from swarm:
  • On server with leader: docker swarm leave --force
• apt update && apt upgrade on all servers
• Reboot all servers, oiiku-proddb first.
• Recreate Docker Swarm:
  • ansible-playbook -i inventory-production.yml docker-swarm-create.yml -e "@secrets-production.yml" --extra-vars "hosts=all" --limit=oiiku_proddb
  • Note the token and manager node ip, and join all servers: ansible-playbook -i inventory-production.yml docker-swarm-join.yml -e "@secrets-production.yml" --extra-vars "hosts=all swarm_token=<swarm-token> swarm_node_ip=<ip-for-oiiku-proddb> --limit=oiiku_prod1,oiiku_prod2
• Promote all nodes to manager: docker node promote <node-1> <node-2> ...
• Deploy stack:
  • On oiiku-proddb: echo <GH_PAT> | docker login ghcr.io -u <username> --password-stdin
  • On oiiku-proddb: docker stack deploy --compose-file /usr/local/etc/docker-compose.production.yml --with-registry-auth
• Finish:
  • Remove maintenance page
  • Update status.oiiku.com
  • Notify #team

source: docs/docker_swarm.clj