Overlay2 consuming way more disk space than reported by docker system df

#1

Our docker storage is mounted on /mnt/docker_storage. It is quickly filled up, but as you can see only a fraction of the total space used is accounted in docker system df. Even after doing a complete prune by deleting all containers, images, volumes, networks, build cache etc. There will be a huge amount left over in the overlay2 directory presumably from artifacts that weren’t cleaned up by an unknown culprit. Is there anyway to identify which images “own” which overlay2 directories? Ideally docker system prune would remove “unowned” overlay2 layers. Currently I have to stop docker. Prune everything, move overlay2 to overlay2bak, start deleting it in the background then resume docker. I have to do this about once a month. If this is the wrong forum I apologize, any help in redirecting me would be greatly appreciated.

ubuntu@ops:~$ sudo su
root@ops:/home/ubuntu# df -H
Filesystem      Size  Used Avail Use% Mounted on
udev             34G     0   34G   0% /dev
tmpfs           6.7G  1.3M  6.7G   1% /run
/dev/nvme3n1p1   90G   62G   25G  72% /
tmpfs            34G  8.2k   34G   1% /dev/shm
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs            34G     0   34G   0% /sys/fs/cgroup
/dev/nvme0n1     79G   16G   60G  22% /mnt/elastic
/dev/nvme2n1    265G   58G  196G  23% /mnt/jenkins_workspace
/dev/nvme1n1    553G  222G  303G  43% /mnt/docker_storage
tmpfs           6.7G     0  6.7G   0% /run/user/113
/dev/loop2       94M   94M     0 100% /snap/core/8268
tmpfs           6.7G     0  6.7G   0% /run/user/1000
/dev/loop1       96M   96M     0 100% /snap/core/8592
overlay         553G  222G  303G  43% /mnt/docker_storage/overlay2/u5fzewgaok0j5jj7n8enbj12j/merged
overlay         553G  222G  303G  43% /mnt/docker_storage/overlay2/ucvj9iv4sf0709zqx4z26ri37/merged
overlay         553G  222G  303G  43% /mnt/docker_storage/overlay2/33nvb1g1ji6uovzea3rdls7ix/merged
root@ops:/home/ubuntu# DOCKER_BUILDKIT=1 docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              82                  0                   54.27GB             54.27GB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         408                 48                  622MB               0B
#2

Overlay does not take space itself.

What you are seeing in the mount table is overlay mounts which just display the full usage/space of everything on the mount where they are rooted.

#3

This is just because we are currently building, but when absolutely nothing is being used. In fact when everything is completely deleted, overlay2 directory will be using like 80% of the drive with zero mounts. Please notice the discrepancy in the the df -h amounts used and what is reported by docker system df

#4 
/dev/nvme1n1    553G  222G  303G  43% /mnt/docker_storage

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              82                  0                   54.27GB             54.27GB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         408                 48                  622MB
#5

df -h reports all space not just what’s in the overlay mount.

This is how overlays work. The mounts do not take space, they are just different views on top of the existing filesystem.

What docker system df reports it the actual space used by the containers/images. It does this by walking the tree and calculating the space and is more like du than df.

#6

I’m talking about when there are no mounts. No images, no containers, docker isn’t running. I go into /mnt/docker_storage/overlay2 and du -sh and there are tons of directories using tons of space. Nothing is mounted.

#7

Please don’t fixate on just the bash output I posted. I assure you I have run docker system prune -a -f --volumes docker builder prune -f etc. unmounted any overaly2 mounts. I’ve made sure there is nothing left in docker, but /mnt/docker_storage/overlay2 still takes up a huge amount of space.

#8

I just wrote a script to find all overlay2//diff which are used by any image, and compared that to the overlay2//diff directories that exist, and there are 436 diff directories unused by any image (there are no containers)

#9

What version of Docker?

How old is the node?

There have been versions that leaked data in the past for various reasons.

#10

Docker is 19.03.5
The node is from Thu Jan 14 19:52:39 2016, but I’ve completely blown away the overlay2 directory a couple of times within the last 4 months

#11

Would you say that diff directories that are not referenced by any images are safe to delete? For me, if I delete something being used it’s not a big deal as this is just a build server that isn’t running any containers or hosting any permanent images.