This issue is partially fixed in SD 2.4. As of 2.4, mysql binlogs are deleted after 10 days, however for busy systems it's still possible for the disk to fill up within that time span.
Follow these steps to resolve the issue:
- Obtain shell access. As of 2.6 this is available by entering the admin password in System > Security, and skip to step 4. Otherwise, use the following procedure. Access the command line interface generate a challenge using the following command:
# cli challenge generate
- Call Pulse Support to obtain the response code.
- Return to the CLI and run the following command:
# cli challenge response <code from support>
- Enter the following command to get shell access. If you are running 2.6 or above make sure Shell Access is enabled in the UI > System > Security Settings.
- Find which logs are not currently in use, in order to delete them. In order to do this we need to find which are currently in use first.
mysql> show master status;
This will show e.g.:
| local-bin.000010 | 88255313 | | | ce7417e5-8400-11e6-afa0-bda0e7a997f7:1-2776689 |
which says local-bin.000010 is in use. Next exit out of mysql, and delete an unused files, i.e. one not listed above:
# cd /data/ssc/mysql
# ls -ltrh local-bin.*
This shows e.g.:
-rw-rw---- 1 admin root 1.1G May 14 09:42 local-bin.000003
-rw-rw---- 1 admin root 1.1G May 15 23:48 local-bin.000004
-rw-rw---- 1 admin root 1.1G May 17 14:01 local-bin.000005
-rw-rw---- 1 admin root 1.1G May 19 04:18 local-bin.000006
-rw-rw---- 1 admin root 1.1G May 20 18:41 local-bin.000007
-rw-rw---- 1 admin root 1.1G May 22 09:10 local-bin.000008
-rw-rw---- 1 admin root 1.1G May 23 23:41 local-bin.000009
-rw-rw---- 1 admin root 573M Jun 30 10:12 local-bin.000010
Since local-bin.000010 is in use, any of the others can be deleted, e.g.
# rm local-bin.000003
Next purge the logs, to delete the rest of the unused files in the official way:
mysql> PURGE BINARY LOGS TO 'local-bin.000010';
Note that PURGE will hang if the disk is completely full so some space must be freed before running it.
- From the Web UI go to System > Service Status and verify that the Services Director Service shows a status of "running".
- If necessary start the Services Director Service.
Note: Once the logs have been cleared from the /data partition directory it is normal to still see errors in the GUI that the disk is full due to the queued transactions that were not able to be written to the database. These errors will clear up in time as they are processed from the queue.
- Lastly, The default log expire time is set to 3 days in version 18.1. For versions <18.1 (2.4 onwards) this is set to 10 days and should be manually changed using the following:
> enable # configure terminal # ssc database replication expire-bin-log 3
It is highly recommended to upgrade to version 18.1 or higher as there are newly added UI features that make this process easier.