Go fsnotify and Kubernetes ConfigMaps

Go fsnotify and Kubernetes ConfigMaps

Since we use k8s more and more at work and some of my small micro-services are using ConfigMaps I needed a simple way to reload the config on-the-fly without the need to restart the whole service. At the time of solving the problem there was nothing my Google-Fu skills could find about it so I thought I would share my solution with the world.

The way ConfigMaps work in k8s is that files are created inside the running container using a couple of symlinks. Example of running ls inside the container with the added configmap:

/etc/healthmonit # ls -al
total 32
drwxrwxrwx    3 root     root          4096 Jul 11 13:24 .
drwxr-xr-x    1 root     root          4096 Jul 11 12:52 ..
drwxr-xr-x    2 root     root          4096 Jul 11 13:24 ..2018_07_11_11_24_39.108325508
lrwxrwxrwx    1 root     root            31 Jul 11 13:24 ..data -> ..2018_07_11_11_24_39.108325508
lrwxrwxrwx    1 root     root            23 Jul 11 12:52 healthmonit.conf -> ..data/healthmonit.conf

So what we need is to use the fsnotify library for Go and to create a watcher for our config-file. Since its actually a symlink we are watching its not Write events we need to react on but Remove. I personally did the mistake to only watch Writes, but later found out about the whole symlink part. In the snippet below I have simple working example that I personally use:

func configWatcher() {
	watcher, err := fsnotify.NewWatcher()
	if err != nil {
        //handle err
	}
	defer watcher.Close()
	err = watcher.Add(configfile)
	if err != nil {
        //handle err
	}
	for {
		select {
		case event := <-watcher.Events:
			// k8s configmaps uses symlinks, we need this workaround.
            // original configmap file is removed
			if event.Op == fsnotify.Remove {
                // remove the watcher since the file is removed
				watcher.Remove(event.Name)
                // add a new watcher pointing to the new symlink/file
				watcher.Add(configfile)
				reloadConfig()
			}
			// also allow normal files to be modified and reloaded.
			if event.Op&fsnotify.Write == fsnotify.Write {
				reloadConfig()
			}
		case err := <-watcher.Errors:
			// handle error
		}
	}
}