Fixing Bloated Container Logs in /var/lib/docker/containers

* This page contains promotional content

An LDAP server running on Docker had reached 100% disk usage, so I looked into it.
This LDAP server is close to read-only, so the data should not grow that much, should it?

# du -sh /var/lib/docker/* |sort -n
1.2G	/var/lib/docker/overlay2
2.1M	/var/lib/docker/image
4.0K	/var/lib/docker/runtimes
4.0K	/var/lib/docker/swarm
4.0K	/var/lib/docker/tmp
4.0K	/var/lib/docker/trust
4.4M	/var/lib/docker/volumes
7.8G	/var/lib/docker/containers
20K	/var/lib/docker/builder
24K	/var/lib/docker/plugins
60K	/var/lib/docker/network
108K	/var/lib/docker/buildkit

/var/lib/docker/containers is clearly the one using a lot of space

The fix

It is exactly as this article says ↓

Find the container ID

$ docker ps -a
CONTAINER ID   IMAGE                        COMMAND                 CREATED       STATUS       PORTS                                            NAMES
9de577ac08a6   osixia/phpldapadmin:latest   "/container/tool/run"   2 years ago   Up 7 weeks   443/tcp, 0.0.0.0:6001->80/tcp, :::6001->80/tcp   ldap-admin
87f44c8f00af   osixia/openldap:latest       "/container/tool/run"   2 years ago   Up 7 weeks   0.0.0.0:389->389/tcp, :::389->389/tcp, 636/tcp   ldap-host

The ID starts with 87f, so let’s search for the location of the log

$ docker inspect 87f |grep -i log
        "LogPath": "/var/lib/docker/containers/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0-json.log",
            "LogConfig": {

Now that the location of the log is known, let’s check its size

# ls -lh /var/lib/docker/containers/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0-json.log
-rw-r----- 1 root root 7.8G 10月  5 16:15 /var/lib/docker/containers/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0-json.log

It had grown to 7.8GB

Delete the log file (empty it)

$ sudo truncate -s 0 /var/lib/docker/containers/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0-json.log

# ls -lh /var/lib/docker/containers/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0-json.log
-rw-r----- 1 root root 0 10月  5 16:18 /var/lib/docker/containers/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0/87f44c8f00af3c043762fcac9dca218c3bcf7f3111594b56210b18574c0317c0-json.log

Leaving it like this only solves the problem temporarily, and the same symptom will come back sooner or later, so fix the log settings

On this server only the LDAP Docker containers are running, so I configure docker globally

Create /etc/docker/daemon.log anew with the following contents

$ cat /etc/docker/daemon.log
{
   "log-driver": "json-file",
   "log-opts": {"max-size": "10m", "max-file": "3"}
}

For an individual container, write it into docker-compose.yml as below

logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Restart docker and you are done

docker

debian

Fixing Bloated Container Logs in /var/lib/docker/containers

Dealing with bloated Docker logs

The fix

See also