Core dumps are essential tools for diagnosing application crashes. They capture a snapshot of an application's memory at the time of failure, enabling developers to analyze and identify the root cause of the issue. This article provides a step-by-step guide to analyzing core dumps on both Linux and UNIX systems using GDB (GNU Debugger).

Applicability

This procedure is applicable to:

• Linux systems (e.g., Ubuntu, CentOS, Debian).
• UNIX variants (e.g., AIX, Solaris, FreeBSD, HP-UX) where GDB is supported.

For UNIX systems without GDB, other tools like dbx (AIX) or adb (Solaris) may be required.

Prerequisites

Before starting, ensure the following:

1. Core dump file: Typically named core.xxx or similar.
2. Binary file: The exact application binary that generated the core dump.
3. GDB: Installed on your system. On Linux, use your package manager:

   <code><span class="hljs-comment">#On Debian/Ubuntu</span>
   sudo apt-get install gdb
   
   <span class="hljs-comment"># On RHEL/CentOS</span>
   sudo yum install gdb</code>

1. Core Dump Analysis Command

To analyze a core dump and get stack traces for all threads, use the following command. Replace <binary> with your application binary and <core.xxx> with the core dump file:

gdb <binary> <core.xxx> -ex "thread apply all bt" -ex "quit" > output.log 2>&1

Explanation

• gdb <binary> <core.xxx>: Launches GDB with the binary and core dump file.
• -ex "thread apply all bt": Runs the bt (backtrace) command for all threads.
• -ex "quit": Exits GDB after completing the backtrace.
• > output.log 2>&1: Redirects the output (both stdout and stderr) to a file named output.log.

Considerations for UNIX

• Core Dump Compatibility: Ensure that GDB can read the core dump file. Some UNIX systems may have different core dump formats. If GDB fails, consider using native debugging tools (e.g., dbx on AIX or adb on Solaris).
• Permissions: Verify you have read access to the core dump file.
• Environment Settings: For systems where core dumps are not generated by default, you may need to enable them. On Linux and many UNIX systems, use:

  <code><span class="hljs-built_in">ulimit</span> -c unlimited</code><code></code>

• Tool Availability: Ensure GDB is installed or compatible with your UNIX version.

Output

The command generates a file, output.log, containing:

• A detailed stack trace of all threads.
• Insight into the state of the application at the time of the crash.

This file is essential for debugging and root cause analysis.

2. Analyzing a Running Process During a Freeze

In cases where an application is frozen but still running, you can attach GDB to the process to analyze its current state without killing it.

Command for Live Process Analysis

Replace <pid> with the process ID of the application (e.g., obtained via ps or top):

gdb -p <pid> -ex="set confirm off" -ex "where" -ex "thread apply all bt" -ex "bt full" -ex "info locals" -ex "info threads" -ex "info registers" -ex "quit" > ../log/pserver_analyse.log 2>&1

Explanation

• gdb -p <pid>: Attaches GDB to the running process identified by <pid>.
• set confirm off: Disables interactive confirmations in GDB for smoother execution.
• where: Provides the current stack trace of the thread where GDB is attached.
• thread apply all bt: Captures a backtrace for all threads in the application.
• bt full: Displays a detailed backtrace, including function arguments and local variables.
• info locals: Displays local variables for the current frame.
• info threads: Lists all active threads and their states.
• info registers: Shows the current CPU register values.
• quit: Detaches GDB from the process and exits.
• > ../log/pserver_analyse.log 2>&1: Redirects output to a log file (pserver_analyse.log).

Output

The output log contains a detailed analysis of the process's current state, including:

• Stack traces for all threads.
• Local variables and function arguments.
• Active thread information.
• CPU register states.