Multi-threaded applications can encounter problems that are rare or hard to reproduce, such as:

• Race conditions — two threads accessing shared data without synchronization.

• Deadlocks — threads blocked forever waiting on locks.

• Livelocks — threads constantly yielding to each other without progress.

• Starvation — some threads never get CPU time.

• Heisenbugs — bugs that change or disappear when observed.

Linux’s scheduler, processors with different cores, interrupts, and I/O all add complexity. Fortunately, Red Hat Linux offers powerful tools and techniques to help.

---

🧰 Essential Tools for Debugging on Red Hat Linux

Before diving into techniques, ensure you are comfortable with these tools:

Tool	Purpose
gdb	GNU Debugger for source-level debugging
strace / ltrace	System call / library call tracing
perf	Performance profiling & hardware event tracing
valgrind	Dynamic analysis (memory/thread errors)
systemtap	Scriptable kernel / userspace tracing
rr	Deterministic debugging/replay
top / htop / pidstat	System/CPU/thread usage
pthread debug APIs	Thread error checks

---

🔁 Step-by-Step Debugging Workflow

1. Reproduce the Issue Reliably

Before debugging, you need a reliable repro:

• Use controlled input or test harnesses.

• Shell loops (while true; ./app; done) can help catch intermittent issues.

• Reduce variability (disable network, fix timers).

If the bug is timing related, consider artificial delays.

---

🧠 Strategy 1: Visibility — Logging & Tracing

📌 Use Logging Effectively

• Log thread IDs pthread_self() and function entry/exits.

• Log lock acquisition/release times.

• Timestamps and sequence numbers help order events.

Example:

fprintf(logfile, "Thread %lu acquired lock A at %ld\n",
        pthread_self(), time(NULL));

Tip: Use log levels and log-rotation to avoid huge output files.

---

📌 strace & ltrace for Quick Insight

• strace -f -p <pid> — trace system calls from all threads (-f for threads).

• ltrace -p <pid> — capture library calls (like pthread_*()).

These can help see if the app is blocked on I/O or waiting for locks.

---

🔧 Strategy 2: Analyze Threads Using GDB

Attach to a Running Process

gdb -p <PID>

Once attached:

• info threads — list all threads

• thread <n> — switch to thread n

• bt — backtrace current thread

Example:

(gdb) info threads
(gdb) thread 3
(gdb) bt

---

Automatic Thread-Awareness

GDB can show where each thread is blocked:

(gdb) thread apply all bt

This outputs all call stacks — invaluable for tracking deadlocks or blocked threads.

---

Conditional Breakpoints

Set breakpoints only when certain conditions occur:

 break myfunc if x == 42

This reduces noise and focuses debugging effort.

---

🧪 Strategy 3: Memory & Thread Analysis with Valgrind

Valgrind’s Helgrind can detect:

• Race conditions

• Incorrect lock use

Run with:

valgrind --tool=helgrind ./app

Output highlights problematic races and lock patterns.

Note: Valgrind slows execution, but it’s incredibly powerful for correctness checks.

---

📊 Strategy 4: Performance Profiling with perf

Use perf to collect performance data:

perf record -g ./app
perf report

This shows CPU hotspots, lock contention, and waiting patterns.

For thread-specific profiles:

perf top -p <PID> --threads

---

🧠 Strategy 5: Deterministic Debugging with rr

The rr tool allows record-and-replay debugging:

rr record ./app
rr replay

Once recorded, execution can be replayed deterministically in GDB—even for multithreaded timing bugs.

This is especially useful for Heisenbugs.

---

🧪 Strategy 6: Kernel & LWP Tracing

systemtap

For deep tracing, use systemtap to instrument kernel events or user functions:

stap -e 'probe process("/usr/bin/app").function("foo") { println("entered foo") }'

This helps when GDB isn’t sufficient.

---

⚙️ Strategy 7: Profiling at the Kernel Level

Commands like:

• top -H — show thread view

• pidstat -t 1 — per-thread CPU usage

• perf sched — scheduler behavior

These help inspect how threads interact system-wide.

---

🛠 Common Thread Issues & How to Spot Them

🧵 Race Conditions

Symptoms: Random failures, data inconsistency.

Detection:

• Helgrind / ThreadSanitizer.

• Instrument code with atomic or lock checks.

---

🔒 Deadlocks

Symptoms: App halts with threads waiting on each other.

Detection:

• GDB backtrace (thread apply all bt)

• Logging lock acquisition

Fix: Reorder lock acquisition or consolidate locks.

---

🚫 Starvation & Priority Inversion

Symptoms: Lower-priority threads block important work.

Detection:

• Scheduler profiling (perf sched)

• top -H

---

📈 Best Practices to Prevent Future Issues

✅ Minimize shared state
✅ Prefer message passing (e.g., queues) over locks
✅ Use thread-safe libraries
✅ Apply static analysis before runtime
✅ Write unit tests for thread interactions

---

📌 Summary

Goal	Recommended Tool/Method
Find race conditions	Valgrind, Helgrind
Inspect thread state	GDB info threads, bt
Trace system activity	strace, systemtap
Profile performance	perf
Reproduce Heisenbugs	rr
Monitor thread CPU usage	top, pidstat

---

📎 Final Thoughts

Debugging complex multi-threaded issues is a mixture of science and art. It requires:

✔ Patience
✔ Methodical logging
✔ Understanding thread interactions
✔ Using the right tools

Red Hat Linux provides a strong ecosystem of debugging support — leveraging these tools effectively can turn frustrating bugs into solved puzzles.

---

✅ Multi-Threaded Application Debugging Checklist (RHEL)

Use this top-down during outages or strange behavior.

---

1️⃣ Initial Triage (5–10 minutes)

Goal: Is it CPU, memory, I/O, or locking?

• Is the process running or hung?

  ps -eLf | grep <process_name>
  

• Any recent crashes?

  coredumpctl list
  

• Check system pressure:

  uptime
  vmstat 1 5
  dmesg -T | tail
  

---

2️⃣ Thread-Level Visibility

Goal: Identify blocked or runaway threads.

• View threads in real time

  top -H -p <PID>
  

• Per-thread CPU usage

  pidstat -t -p <PID> 1
  

• Count threads

  ls /proc/<PID>/task | wc -l
  

🚩 Red flags:

• Threads stuck at 0% CPU

• One thread consuming 100%

• Thread count continuously increasing

---

3️⃣ Attach GDB (Live Process)

Goal: Find deadlocks, infinite waits, blocked syscalls.

gdb -p <PID>

Inside GDB:

info threads
thread apply all bt

🔍 Look for:

• pthread_mutex_lock

• futex_wait

• poll, epoll_wait

• sleep, nanosleep

---

4️⃣ System Call & Lock Inspection

Goal: Is the app blocked on kernel or I/O?

strace -f -p <PID>

Common patterns:

• futex() → lock contention

• read() / write() → I/O blocking

• accept() → network stall

---

5️⃣ Performance & Contention Analysis

Goal: CPU hotspots or scheduler issues.

perf top -p <PID> --threads

Or record:

perf record -g -p <PID> sleep 30
perf report

---

6️⃣ Race Condition & Lock Validation (Offline)

Goal: Find hidden concurrency bugs.

valgrind --tool=helgrind ./app

✔ Detects:

• Data races

• Lock order violations

• Incorrect mutex usage

---

7️⃣ Heisenbugs / Non-Deterministic Failures

Goal: Replay the bug deterministically.

rr record ./app
rr replay

Perfect for:

• Rare crashes

• Timing-related failures

• Interview-grade debugging scenarios

---

8️⃣ Kernel-Level Deep Dive (Advanced)

Goal: Scheduler or kernel contention.

perf sched record
perf sched latency

Or SystemTap (RHEL-heavy environments).

---

🛠️ Ready-to-Use Diagnostic Script (Production Safe)

📌 What it does

• Captures thread state

• GDB backtraces

• CPU usage

• strace snapshot

• Logs everything for RCA

---

🔧 thread_debug_collect.sh

#!/bin/bash

PID=$1
OUTDIR="/tmp/thread_debug_$(date +%F_%H-%M-%S)"

if [ -z "$PID" ]; then
  echo "Usage: $0 <PID>"
  exit 1
fi

mkdir -p $OUTDIR

echo "[+] Collecting process info"
ps -p \(PID -Lf > \)OUTDIR/ps_threads.txt

echo "[+] Collecting top thread snapshot"
top -H -b -n 1 -p \(PID > \)OUTDIR/top_threads.txt

echo "[+] Collecting per-thread CPU usage"
pidstat -t -p \(PID 1 5 > \)OUTDIR/pidstat.txt

echo "[+] Capturing GDB thread backtraces"
gdb -batch -p $PID \
  -ex "set pagination off" \
  -ex "info threads" \
  -ex "thread apply all bt" \
  > $OUTDIR/gdb_backtrace.txt

echo "[+] Capturing short strace snapshot (10s)"
timeout 10 strace -f -p \(PID -o \)OUTDIR/strace.txt

echo "[+] Collecting kernel messages"
dmesg -T | tail -200 > $OUTDIR/dmesg_tail.txt

echo "[+] Collection complete"
echo "Logs stored in: $OUTDIR"

---

▶️ How to Run

chmod +x thread_debug_collect.sh
sudo ./thread_debug_collect.sh <PID>

📁 Output folder example:

/tmp/thread_debug_2026-01-31_12-30-10/
├── ps_threads.txt
├── top_threads.txt
├── pidstat.txt
├── gdb_backtrace.txt
├── strace.txt
└── dmesg_tail.txt

Perfect for:

• Incident RCA

• Sharing with developers

• Interview explanations

• Postmortem documents

---

🧠 Interview-Ready Explanation

If asked “How do you debug multi-threaded issues on Linux?”

“I start with thread-level visibility using top -H and pidstat, then attach GDB to analyze blocked threads using thread apply all bt. For lock contention I rely on strace and perf. For race conditions, I use Helgrind, and for non-deterministic bugs I use rr for record-and-replay debugging.”

That answer alone scores big points 💯

---