Linux

GlusterFS Client Log Files Are Huge

This is an issue I recently fixed on our GlusterFS installation, but I didn’t see anything directly on Google for lazy people like myself.  The issue is that the log files in /var/log/glusterfs are HUGE.  Ours ended up being > 30 GB after about a day and ended up causing issues since the filesystem of our container filled up.

Our log files were lots of:

[2016-08-24 13:51:20.776255] I [dict.c:370:dict_get] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/xlator/performance/md-cache.so(mdc_lookup+0x2ff) [0x7fd05c6e4c6f] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x114) [0x7fd05c4ce6d4] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e6) [0x7fd05c2b9926]))) 0-dict: !this || key=system.posix_acl_access
[2016-08-24 13:51:20.776281] I [dict.c:370:dict_get] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/xlator/performance/md-cache.so(mdc_lookup+0x2ff) [0x7fd05c6e4c6f] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x114) [0x7fd05c4ce6d4] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x232) [0x7fd05c2b9972]))) 0-dict: !this || key=system.posix_acl_default

Our /etc/fstab line looked like:

atlantia.sthse.co:/gv0    /mnt/gv0    glusterfs    _netdev,acl,defaults    0    0

It’s a simple fix, the default log level (INFO) is much too verbose, and outputs much more than it needs to into the logs.  Fixing this is a simple update to /etc/fstab:

atlantia.sthse.co:/gv0    /mnt/gv0    glusterfs    _netdev,acl,log-level=WARNING,defaults    0    0

Notice the addition of the log-level=WARNING.  There are different log levels, depending on how much data you would like that are listed in the mount.glusterfs man page.

Platform is Debian 8.5, glusterfs-client version 3.5.2-2+deb8u2, glusterfs-common version 3.5.2-2+deb8u2

Leave a Reply

Your email address will not be published. Required fields are marked *