Today I encountered an issue related to CheckPoint Managment server: it started to report incorrect file system space utilization on the logs’ mount point. The output of dfcommand reports the mount point almost full:
|
1 2 3 4 5 6 |
[Expert@cpmgmtsrv:0]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_splat-lv_current 46G 20G 25G 45% / /dev/sda1 289M 24M 251M 9% /boot tmpfs 3.9G 0 3.9G 0% /dev/shm/dev/mapper/vg_splat-lv_log 97G 85G 7.0G 93% /var/log |
Instead, the sum of all files and dir present under /var/log is around 37 Gb.
|
1 2 |
[Expert@cpmgmtsrv:0]# du -hs /var/log 37G /var/log |
This is a strange behavior, especially because the mount point is used for logging. Comparing both commands, it’s easy to understand that more than half of the total disk space is wrongly marked as used, so I would like to figure out where the problem lay. I think the unlinked open files may be the problem and I’m checking for this using lsof:
|
1 2 3 4 5 6 7 |
[Expert@cpmgmtsrv:0]# lsof -n | grep deleted httpd2 3182 admin 12w REG 253,0 10878977 /usr/local/apache2/logs/ssl_mutex (deleted) cplmd 24066 admin 67r REG 253,1 18415633 /var/log/backup_log/2015-09-22_142548_87.log (deleted) cplmd 24066 admin 68r REG 253,1 18415621 /var/log/backup_log/2015-09-25_141903_96.log (deleted) cplmd 24066 admin 69r REG 253,1 18415643 /var/log/backup_log/2015-09-24_143308_93.log (deleted) cplmd 24066 admin 70r REG 253,1 18415643 /var/log/backup_log/2015-09-24_143308_93.log (deleted) [...] |
As I expected, the lsof output reports lots of open file entries, all associated to cplmd daemon. Based on CheckPoint knowledge base (here):
“In order to get the data that should be presented in SmartView Tracker, FWM spawns a child process CPLMD, which reads the information from the log file and performs unification (if necessary). Upon receiving an answer from CPLMD, FWM transfers it to SmartView Tracker.”
The easy way to fix this is to restart the daemon in order to allow it to “flush” all deleted files still hanging:
|
1 2 3 4 |
[Expert@cpmgmtsrv:0]# cpstop [...] [Expert@cpmgmtsrv:0]# cpstart [...] |
This solution fix the problem. Now, df command reports the right space utilizzation.
|
1 2 3 4 5 6 |
[Expert@cpmgmtsrv:0]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_splat-lv_current 46G 20G 25G 45% / /dev/sda1 289M 24M 251M 9% /boot tmpfs 3.9G 0 3.9G 0% /dev/shm/dev/mapper/vg_splat-lv_log 97G 38G 55G 41% /var/log |
and also all orphan files are disappeared.