# Overlay2 consuming way more disk space than reported by docker system df **URL:** \ **Category:** Support\ **Created:** [February 13, 2020, 5:37pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564 "2020-02-13T17:37:00Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 5:37pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/1 "2020-02-13T17:37:00Z")
Our docker storage is mounted on /mnt/docker\_storage. It is quickly filled up, but as you can see only a fraction of the total space used is accounted in `docker system df`. Even after doing a complete prune by deleting all containers, images, volumes, networks, build cache etc. There will be a huge amount left over in the overlay2 directory presumably from artifacts that weren’t cleaned up by an unknown culprit. Is there anyway to identify which images “own” which overlay2 directories? Ideally `docker system prune` would remove “unowned” overlay2 layers. Currently I have to stop docker. Prune everything, move overlay2 to overlay2bak, start deleting it in the background then resume docker. I have to do this about once a month. If this is the wrong forum I apologize, any help in redirecting me would be greatly appreciated. ``` ubuntu@ops:~$ sudo su root@ops:/home/ubuntu# df -H Filesystem Size Used Avail Use% Mounted on udev 34G 0 34G 0% /dev tmpfs 6.7G 1.3M 6.7G 1% /run /dev/nvme3n1p1 90G 62G 25G 72% / tmpfs 34G 8.2k 34G 1% /dev/shm tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs 34G 0 34G 0% /sys/fs/cgroup /dev/nvme0n1 79G 16G 60G 22% /mnt/elastic /dev/nvme2n1 265G 58G 196G 23% /mnt/jenkins_workspace /dev/nvme1n1 553G 222G 303G 43% /mnt/docker_storage tmpfs 6.7G 0 6.7G 0% /run/user/113 /dev/loop2 94M 94M 0 100% /snap/core/8268 tmpfs 6.7G 0 6.7G 0% /run/user/1000 /dev/loop1 96M 96M 0 100% /snap/core/8592 overlay 553G 222G 303G 43% /mnt/docker_storage/overlay2/u5fzewgaok0j5jj7n8enbj12j/merged overlay 553G 222G 303G 43% /mnt/docker_storage/overlay2/ucvj9iv4sf0709zqx4z26ri37/merged overlay 553G 222G 303G 43% /mnt/docker_storage/overlay2/33nvb1g1ji6uovzea3rdls7ix/merged root@ops:/home/ubuntu# DOCKER_BUILDKIT=1 docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 82 0 54.27GB 54.27GB (100%) Containers 0 0 0B 0B Local Volumes 0 0 0B 0B Build Cache 408 48 622MB 0B ``` ---
**Author:** ![cpuguy83](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.mobyproject.org/cpuguy83/32/22_2.png) [@cpuguy83](https://forums.mobyproject.org/u/cpuguy83)\ **Post date:** [February 13, 2020, 5:53pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/2 "2020-02-13T17:53:13Z")
Overlay does not take space itself. What you are seeing in the mount table is overlay mounts which just display the full usage/space of everything on the mount where they are rooted. ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:17pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/3 "2020-02-13T18:17:39Z")
This is just because we are currently building, but when absolutely nothing is being used. In fact when everything is completely deleted, overlay2 directory will be using like 80% of the drive with zero mounts. Please notice the discrepancy in the the df -h amounts used and what is reported by docker system df ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:18pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/4 "2020-02-13T18:18:35Z")
``` /dev/nvme1n1 553G 222G 303G 43% /mnt/docker_storage TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 82 0 54.27GB 54.27GB (100%) Containers 0 0 0B 0B Local Volumes 0 0 0B 0B Build Cache 408 48 622MB ``` ---
**Author:** ![cpuguy83](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.mobyproject.org/cpuguy83/32/22_2.png) [@cpuguy83](https://forums.mobyproject.org/u/cpuguy83)\ **Post date:** [February 13, 2020, 6:23pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/5 "2020-02-13T18:23:17Z")
`df -h` reports _all_ space not just what’s in the overlay mount. This is how overlays work. The mounts do not take space, they are just different views on top of the existing filesystem. What `docker system df` reports it the actual space used by the containers/images. It does this by walking the tree and calculating the space and is more like `du` than `df`. ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:25pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/6 "2020-02-13T18:25:28Z")
I’m talking about when there are no mounts. No images, no containers, docker isn’t running. I go into /mnt/docker\_storage/overlay2 and du -sh and there are tons of directories using tons of space. Nothing is mounted. ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:29pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/7 "2020-02-13T18:29:01Z")
Please don’t fixate on just the bash output I posted. I assure you I have run `docker system prune -a -f --volumes` ` docker builder prune -f` etc. unmounted any overaly2 mounts. I’ve made sure there is nothing left in docker, but /mnt/docker\_storage/overlay2 still takes up a huge amount of space. ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:37pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/8 "2020-02-13T18:37:34Z")
I just wrote a script to find all overlay2/_/diff which are used by any image, and compared that to the overlay2/_/diff directories that exist, and there are 436 diff directories unused by any image (there are no containers) ---
**Author:** ![cpuguy83](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.mobyproject.org/cpuguy83/32/22_2.png) [@cpuguy83](https://forums.mobyproject.org/u/cpuguy83)\ **Post date:** [February 13, 2020, 6:41pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/9 "2020-02-13T18:41:10Z")
What version of Docker? How old is the node? There have been versions that leaked data in the past for various reasons. ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:49pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/10 "2020-02-13T18:49:21Z")
Docker is 19.03.5 The node is from Thu Jan 14 19:52:39 2016, but I’ve completely blown away the overlay2 directory a couple of times within the last 4 months ---
**Author:** ![charlesritchea](https://avatars.discourse-cdn.com/v4/letter/c/48db29/32.png) [@charlesritchea](https://forums.mobyproject.org/u/charlesritchea)\ **Post date:** [February 13, 2020, 6:56pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/11 "2020-02-13T18:56:48Z")
Would you say that diff directories that are not referenced by any images are safe to delete? For me, if I delete something being used it’s not a big deal as this is just a build server that isn’t running any containers or hosting any permanent images. ---
**Author:** ![jdschmidt](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.mobyproject.org/jdschmidt/32/242_2.png) [@jdschmidt](https://forums.mobyproject.org/u/jdschmidt)\ **Post date:** [April 15, 2021, 10:35pm UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/12 "2021-04-15T22:35:57Z")
I know I’m late to the party but I have this issue as well and it’s on a gitlab runner. `docker system df` shows no data yet in the` /var/lib/overlay2` directory there is ~250GB of data. I just stopped docker and wiped that directory and restarted docker and seems to not have caused any issues. But how is all this data in that directory when a `docker ps -a` shows no images? Docker version 19.03.12, build 48a66213fe ---
**Author:** ![zhlhahaha](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.mobyproject.org/zhlhahaha/32/194_2.png) [@zhlhahaha](https://forums.mobyproject.org/u/zhlhahaha)\ **Post date:** [April 16, 2021, 1:55am UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/13 "2021-04-16T01:55:50Z")
Maybe it caused by docker volume. Can you try `docker volume list`? ---
**Author:** ![gablank](https://yyz1.discourse-cdn.com/flex031/user_avatar/forums.mobyproject.org/gablank/32/328_2.png) [@gablank](https://forums.mobyproject.org/u/gablank)\ **Post date:** [October 24, 2023, 8:55am UTC](https://forums.mobyproject.org/t/overlay2-consuming-way-more-disk-space-than-reported-by-docker-system-df/564/14 "2023-10-24T08:55:14Z")
We are seeing the same. After pruning we see a huge amount of disk space still in use. We use XFS for the partition where we store docker data, I wonder if that has anything to do with this (slightly unconventional filesystem choice I guess).