Cleaning up a full Docker partition
Docker is great when it comes to isolating applications and services but like many other things, it has some serious drawbacks.
One of the issues that repeatedly bites me is the massive amount of disk space it may use. In rare occasions it can even make your system run out of disk space. On top of this Docker lacks a decent system to clean I'ts mess up, so fixing it isn't as easy as it should be. Fortunately, solving the problem isn't rocket science and the solution is rather simple (scroll down for copy+paste).
During some maintenance on svenv.nl I noticed that Docker did stole all my disk space. A docker's run script (the script I launch at runtime) crashed with the following message:
* /etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full!
I returned to the command line of the host and did a quick check on the disk usage, this confirmed the issue:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 49G 46G 0 100% /
Whenever this happens, I usually run ncdu (apt-get install ncdu) to find what’s using so much space. Because I already suspected Docker from wasting disk space (and ncdu'ing may take up some time) I checked /var/lib/docker first.
ncdu /var/lib/docker
The results were clear, in my case, the container diffs took up all the space:
--- /var/lib/docker ------------------------------------------------------------
41.2GiB [##########] /aufs
17.4MiB [ ] /graph
14.3MiB [ ] /init
400.0KiB [ ] /containers
148.0KiB [ ] /volumes
60.0KiB [ ] /execdriver
48.0KiB [ ] /vfs
40.0KiB [ ] linkgraph.db
8.0KiB [ ] /trust
8.0KiB [ ] /tmp
4.0KiB [ ] repositories-aufs
The solution is to remove old containers and images. All the instructions to re-build them is defined in Dockerfiles so removing old container shouldn't be a problem.
docker ps -a -notrunc | grep 'Exit' | awk '{print $1}' | xargs -r docker rm
docker images -notrunc | grep none | awk '{print $3}' | xargs -r docker rmi
After running these commands disk usage returned back to normal and the application started as usual.
Keep it clean!