16  NFS (Network File System)

(ns nfs)

We use NFS for certain configuration files that are in all of our containers, spread out over the nodes in the Docker Swarm setup.

16.1 Setup

There are ansible scripts for setting this up. They are available in the ansible directory. configure-nfs-server.yml for setting up the server and configure-nfs-client.yml for the clients. Limit the hosts to the server you wish to be the NFS server when setting up the server.

16.2 Moving the NFS server

Moving the NFS server requires tearing the NFS setup down and then rebuilding it. Steps to take:

Manual script

  • Make sure ansible.posix is installed: ansible-galaxy collection install ansible.posix
  • Drain all Docker Swarm nodes (drain means no services are allowed to run on the node)
  • oiiku-staging1 is src and oiiku-staging2 is host. Change as needed
    • ssh in to oiiku-staging1
    • Run docker node ls --format "{{.ID}}" | xargs -I {} docker node update --availability drain {}
  • Remove all NFS setups
    • ansible-playbook -i inventory-staging.yml remove-nfs.yml -e "@secrets-staging.yml" --extra-vars "hosts=all"
  • Copy over /container-data from the src host to the dest host
    • ssh in to oiiku-staging1 and run
      • cp -R /container-data /container-data.tmp
      • sudo chown deploy:deploy -R /container-data
    • ansible-playbook -i inventory-staging.yml copy-directory.yml -e "@secrets-staging.yml" --extra-vars "hosts=all src=/container-data/ dest=deploy@oiiku-staging2:/home/deploy/container-data/ src_host=oiiku_staging1 dest_host=oiiku_staging2" --limit=oiiku_staging1
    • ssh in to oiiku-staging2 and run
      • sudo rmdir /container-data
      • sudo mv /home/deploy/container-data /container-data
    • in oiiku-staging1 run rm -rf /container-data
  • Setup NFS again
    • change nfs.server in the inventory file to the new server name
      • if this is not done, the fstab will be wrong
    • ansible-playbook -i inventory-staging.yml configure-nfs-server.yml -e "@secrets-staging.yml" --extra-vars "hosts=all" --limit=oiiku_staging2
    • ansible-playbook -i inventory-staging.yml configure-nfs-client.yml -e "@secrets-staging.yml" --extra-vars "hosts=all" --limit=oiiku_staging1
    • Verify that that /container-data shows up in both oiiku-staging1 and oiiku-staging2
    • Double check /etc/fstab on all machines and make sure that the nfs mount point is correct. Check both the nfs server name and its presense in clients and absense in the nfs server
  • Start all Docker Swarm services
    • docker node ls --format "{{.ID}}" | xargs -I {} docker node update --availability active {}
  • Verify that everything works as expected
  • ssh in to oiiku-staging1 and run:
    • sudo rm -rf /container-data.tmp

16.3 Documentation

16.4 Weaknesses

The main weakness is the single point of failure for the server. Choose which node where things physically reside carefully. Any node that is a single point of failure is a good candidate.

source: docs/nfs.clj