9800 - Services Director VA is in a degraded state due to /data partition being full.

This article describes the symptoms that are present when the /data partition is full in the Services Director VA and provides both short and long-term solutions to resolve the issue.  

Symptoms:

• The Services Director (SD) Virtual Appliance (VA) is indicating that it is in a degraded state.  
• Get a sysdump and check the disk usage to identify that the /data partition is 100% full.  
• From the CLI, run the following command to check the disk space.  

> show alarms triggered
Alarm ID: fs_mnt
Alarm Description: Disk Full
Status: error
-------------------------------------

 

Mysql binary logs have filled up /data.

This issue is partially fixed in SD 2.4.  As of 2.4, mysql binlogs are deleted after 10 days, however for busy systems it's still possible for the disk to fill up within that time span.

Follow these steps to resolve the issue:

1.  Obtain shell access.  As of 2.6 this is available by entering the admin password in System > Security, and skip to step 4.  Otherwise, use the following procedure.  Access the command line interface generate a challenge using the following command:
   
   > enable
   # cli challenge generate ​
    
2. Call Pulse Support to obtain the response code.
3. Return to the CLI and run the following command:
   
   # cli challenge response <code from support>
    
4. Enter the following command to get shell access. If you are running 2.6 or above make sure Shell Access is enabled in the UI > System > Security Settings. 
   
   # _shell
    
5. Find which logs are not currently in use, in order to delete them.  In order to do this we need to find which are currently in use first.
    

   # mysql 
   mysql> show master status;

   
   ​This will show e.g.:
    

   | local-bin.000010 | 88255313 |              |                  | ce7417e5-8400-11e6-afa0-bda0e7a997f7:1-2776689 |

   
   which says local-bin.000010 is in use. Next exit out of mysql, and delete an unused files, i.e. one not listed above:
    

   mysql> exit
   # cd /data/ssc/mysql
   # ls -ltrh local-bin.*

   
   This shows e.g.:
    

   -rw-rw---- 1 admin root 1.1G May 14 09:42 local-bin.000003
   -rw-rw---- 1 admin root 1.1G May 15 23:48 local-bin.000004
   -rw-rw---- 1 admin root 1.1G May 17 14:01 local-bin.000005
   -rw-rw---- 1 admin root 1.1G May 19 04:18 local-bin.000006
   -rw-rw---- 1 admin root 1.1G May 20 18:41 local-bin.000007
   -rw-rw---- 1 admin root 1.1G May 22 09:10 local-bin.000008
   -rw-rw---- 1 admin root 1.1G May 23 23:41 local-bin.000009
   -rw-rw---- 1 admin root 573M Jun 30 10:12 local-bin.000010

   
   Since local-bin.000010 is in use, any of the others can be deleted, e.g.
    

   # rm local-bin.000003

   
   Next purge the logs, to delete the rest of the unused files in the official way:
    

   # mysql 
   mysql> PURGE BINARY LOGS TO 'local-bin.000010';

   
   Note that PURGE will hang if the disk is completely full so some space must be freed before running it.
    
6. From the Web UI go to System > Service Status and verify that the Services Director Service shows a status of "running".
    
7. If necessary start the Services Director Service.
   
   Note: Once the logs have been cleared from the /data partition directory it is normal to still see errors in the GUI that the disk is full due to the queued transactions that were not able to be written to the database.  These errors will clear up in time as they are processed from the queue.
    
8. Lastly, ​The default log expire time is set to 3 days in version 18.1.  For versions <18.1 (2.4 onwards) this is set to 10 days and should be manually changed using the following:

          > enable # configure terminal # ssc database replication expire-bin-log 3

It is highly recommended to upgrade to version 18.1 or higher as there are newly added UI features that make this process easier.