Question: The filesystem is almost full. There is a big difference between “df” and “du” commands. The lsof command is also getting hung due to a large number of files. How can we find the space-consuming file?
Normally the difference between “du” and “df” is caused by deleted files that are still in use. Files that have been deleted but are still in use by a process, are not listed but are consuming spaces. This can be verified using the “lsof” command.
If the “lsof” command is not working, a list of files marked as “deleted” can be obtained using:
# ls -l /proc/*/fd/* | grep deleted
The PID of the process holding the file can be checked from the output of the command:
lrwx------. 1 [USER] [GROUP] 64 [DATE] /proc/PID/fd/363 -> [ORIGINAL FILE NAME] (deleted)
where PID is the process ID.
The Correct way to clean the space is to stop the process holding the file, so the process from that PID. Alternatively the space used by the process can be freed by truncating the file, example:
# > /proc/PID/fd/363
But this option must be used with carefully since it can cause unexpected problems to the process. Make sure you check what process is not releasing the space in ‘ps -ef’. For example:
# ps -ef | grep 363
You may consult with app/DB team to check what the process is and if it can be killed.