NAME

       offcputime - Summarize off-CPU time by kernel stack trace. Uses Linux eBPF/bcc.

SYNOPSIS

       offcputime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f] [--stack-storage-size STACK_STORAGE_SIZE]
       [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]

DESCRIPTION

       This  program  shows  stack traces and task names that were blocked and "off-CPU", and the total duration
       they were not running: their "off-CPU time".  It works by tracing when threads block and when they return
       to CPU, measuring both the time they were off-CPU and the blocked stack trace and the  task  name.   This
       data is summarized in the kernel using an eBPF map, and by summing the off-CPU time by unique stack trace
       and task name.

       The  output  summary will help you identify reasons why threads were blocking, and quantify the time they
       were off-CPU. This spans all types of blocking activity: disk  I/O,  network  I/O,  locks,  page  faults,
       involuntary context switches, etc.

       This  is  complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent on-CPU. This
       shows the time spent off-CPU, and the output, especially the -f format, can be used to generate an  "off-
       CPU time flame graph".

       See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

       This  tool  only  works  on  Linux 4.6+. It uses the new `BPF_STACK_TRACE` table APIs to generate the in-
       kernel stack traces.  For kernels older than 4.6, see the version under tools/old.

       Note: this tool only traces off-CPU times that began and ended while tracing.

REQUIREMENTS

       CONFIG_BPF and bcc.

OPTIONS

       -h     Print usage message.

       -p PID Trace this process ID only (filtered in-kernel).

       -t TID Trace this thread ID only (filtered in-kernel).

       -u     Only trace user threads (no kernel threads).

       -k     Only trace kernel threads (no user threads).

       -U     Show stacks from user space only (no kernel space stacks).

       -K     Show stacks from kernel space only (no user space stacks).

       -d     Insert delimiter between kernel/user stacks.

       -f     Print output in folded stack format.

       --stack-storage-size STACK_STORAGE_SIZE
              Change the number of unique stack traces that can be stored and displayed.

       -m MIN_BLOCK_TIME
              The minimum time in microseconds over which we store traces (default 1)

       -M MAX_BLOCK_TIME
              The maximum time in microseconds under which we store traces (default U64_MAX)

       --state
              Filter on this thread state bitmask (eg, 2 ==  TASK_UNINTERRUPTIBLE).   See  include/linux/sched.h
              for states.

       duration
              Duration to trace, in seconds.

EXAMPLES

       Trace all thread blocking events, and summarize (in-kernel) by kernel stack trace and total off-CPU time:
              # offcputime

       Trace for 5 seconds only:
              # offcputime 5

       Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs):
              # offcputime -f 5

       Trace PID 185 only:
              # offcputime -p 185

OVERHEAD

       This  summarizes  unique  stack  traces  in-kernel  for efficiency, allowing it to trace a higher rate of
       events than methods that post-process in user space. The stack trace and time data is only copied to user
       space once, when the output is printed. While these techniques greatly lower overhead,  scheduler  events
       are  still  a high frequency event, as they can exceed 1 million events per second, and so caution should
       still be used. Test before production use.

       If the overhead is still a problem, take a look at the MINBLOCK_US tunable in the code. If your aim is to
       chase down longer blocking events, then this could  be  increased  to  filter  shorter  blocking  events,
       further lowering overhead.

SOURCE

       This is from bcc.

              https://github.com/iovisor/bcc

       Also  look  in  the bcc distribution for a companion _examples.txt file containing example usage, output,
       and commentary for this tool.

OS

       Linux

STABILITY

       Unstable - in development.

AUTHOR

       Brendan Gregg

SEE ALSO

       stackcount(8)

USER COMMANDS                                      2016-01-14                                      offcputime(8)