Reset Search
 

 

Article

KB44691 - How to collect Kernel Trace (Console Logs) when PCS/PPS devices go into Freeze/Stall state

« Go Back

Information

 
Last Modified Date5/23/2022 1:48 PM
Synopsis

How to collect Kernel Trace (Console Logs) when PCS/PPS devices go into Freeze/Stall state

This article describes the steps to collect Kernel Trace (Console Logs) when PCS/PPS devices go into Freeze/Stall state.
Problem or Goal
Kernel Trace is a very important log to understand & debug system freezes/stall issues (different from Hardware Issues)

When PCS/PPS hardware devices go into a Freeze/Stall state, the Kernel Trace isn't enabled by default.

Please follow the steps below to Enable and gather the Kernel Trace to help the Pulse Secure Engineering team debug system issues.

 
Note:

a.    This is applicable only to hardware devices & Azure Deployments. We will update the KB for Virtual/other Cloud deployments in the future.
b.    If devices are found to be not responding, please rule out any hardware issues by checking the LED status indicators before proceeding further.




 
Cause
Solution

Prepping the device to be ready to Log Kernel Log/Trace:

  1. Download Putty : https://www.putty.org/.
  2. Configure the Putty Session logging as needed.
  3. Connect to the Serial Console of the affected PCS/PPS device as shown below.
User-added image
.
 .    4. Click on the left tip of the Putty session Window as shown below.

User-added image

      5. Select “Special Command” and Select “Break”.
      6. As soon as “Break” is selected, hit the 9 (Number Key).
      7. This will enable the Kernel Tracing with output “LogLevel set to 9” (As shown below).

User-added image

8.    If this output isn’t obtained, please try it a few times.
9.    Leave the serial console in this state while capturing the contents into a local file as from time to time the device may log important Kernel Messages important to Root Cause system issues.
10.    When the system Freezes/Stalls or when Web/UI goes inaccessible, click on the left tip of the Putty session as shown below and select “Special Command” & select “Break”.

User-added image

11.    As soon as “Break” is selected, hit the T key (Upper or Lower Case).

 
Disclaimer: Do not execute this command in a fully operational system as this could disconnect user sessions.

12.    The console should output a list of entries similar to the screenshot below,
13.    Save the output completely or take a screenshot.
14.    If the console still responds, navigate to Option # 7. System Maintenance on the Serial Console and generate a System snapshot.
15.    Reboot the device & login to the Admin UI.
16.    Save the PCS logs (User Access/Admin/Events) by navigating to System -> Log/Monitoring -> Select “Save All Logs”.
17.    Navigate to “Maintenance” > “Troubleshooting” > “System Snapshot”. Ensure that “Include Debug log” & “Include System config” options are selected.
18.    Generate a system snapshot, if you could not generate via Serial Console.
  • If the device is already in the Stall/Hung state, Execute Step # 6 and wait for 10-15 minutes without fail, before proceeding to step # 11.
  • During these 10-15 minutes, the device may print useful Kernel logs. Therefore, this waiting period becomes very important.
19.    Gather the following details without fail and upload them to the Support case,
  • Complete Serial Console Output.
  • PCS Logs.
  • System snapshots along with any Process Snapshots or Watchdogs that get generated under the System snapshot tab.
  • Dashboard graphs for 1 day/week/month.
  • Date/time of the occurrence along with the timezone.
  • Response to Ping.

Steps to collect Kernel Trace (Console Logs) for Azure Deployments:

  1. Navigate to the Serial Console of the affected Azure Instance.
  2. Click on the “Keyboard” icon on the Command Bar and select “Send SysRq Command” as shown below,
User-added image

Note - A SysRq (System Request) is a sequence of keys understood by the Linux operation system kernel, which can trigger a set of pre-defined actions. These commands are often used when virtual machine troubleshooting or recovery can't be performed through traditional administration.

       3. In the resulting Dialog box, type “9” upon selecting “Enter Key or Key sequence to send below:” and click on “Send SysRq” button (As shown below). This should set the log level to verbose.
       
User-added image 

        4. ONLY when the VM crashes/Freezes, please send lower case t as a SysRq command as shown below (Please DO NOT execute this command on a LIVE/PRODUCTION system).

User-added image


This will generate the stack trace (Console output) on the Serial Console (Wait up to 10-15 minutes if the Console Output takes time).

Gather these console logs along with the other logs described for Hardware as well and upload them to the case for Engineering analysis. 

 
Related Links
Attachment 1 
Created ByRaghu Kumar

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255