I ran into some interesting space issues on Nutanix cluster recently, I was doing a manual download of NOS to run the upgrade and my /home folder on one of the CVM’s was hitting 95% and giving some warnings.
After a quick call with support my suspicions were confirmed and some stuff could be deleted form this directory , but what to delete as their are lots of folders under the home folder.
Lets take a look at how we got to the correct folder to delete.
First connect to each CVM to check which one is causing the issue, a hint it’s most likely in the alert that been generated in Prism.
Run command –> df-h
This will list the contents on all file systems, In my case the /home folder was showing 95%
Running command –> du – xsh*
Lists the size of each directory so you can see which folders have the most space
Clearly the software_downloads and software_uncompressed folders were the culprits.
You can go directly to these directories and when in directory
Running command –> du-sh
Gives you the total size of the directory.
Running command –> ls
Lists the contents of the folder, in my case I has three versions of NOS downloaded which until installed the binaries are not removed.
So it was safe to delete these and free up the space.
Running Command –> rm -rf /home/nutanix/software_downloads/nos/*
Removes all for files in the folder /NOS
Repeat these steps to clean up the software_uncompressed folder and you will be back to normal.
Update 02/03/2020
Nutanix have now released KB What to do when /home partition or /home/nutanix directory is full
If you use WINSCP to copy the tool to the CVM, make sure you change the permissions to execute and then run it, it will prompt you to install on all CVMs then run the cleanup tool form each CVM and it will do a proper cleanup for you.
Log onto CVM run the tool this extracts it to all CVM’s
Then in the same folder you will see nutanix_home_clean.sh file in same location
Connect to each CVM in the cluster and run the script, this will run through and delete and unneeded files.
I am getting alerts for / (root) filesystem usage. Any idea what to remove? /var/log/journal has 1.1G of files, maybe remove some of them?
LikeLike