Reading notes of top performance version 2 (II) -- Performance observation tool

《System Performance: Enterprise and the Cloud, 2nd Edition (2020)》 Reading notes and brief notes

Chapter four Observation tools

4. Observation tools

4.1 tool coverage

Tool function overview ：

4.1.1 static performance tools

4.1.2 crisis tools

4.2 tools type

4.2.1 Counter type

Kernels maintain various counters for providing system statistics. They are usually implemented
as unsigned integers that are incremented when events occur.

System level tools

vmstat	Virtual and physical memory statistics, system-wide
mpstat	Per-CPU usage
iostat	Per-disk I/O usage, reported from the block device interface
nstat	TCP/IP stack statistics
sar	Various statistics; can also archive them for historical reporting

Process level tools

ps	Shows process status, shows various process statistics, including memory and CPU usage
top	Shows top processes, sorted by CPU usage or another statistic.
pmap	Lists process memory segments with usage statistics

4.2.2 Analysis type

System level

perf	The standard Linux profiler, which includes profiling subcommands
profile	A BPF-based CPU profiler from the BCC repository (covered in Chapter 15, BPF) that frequency counts stack traces in kernel context
Intel VTune Amplifier XE	Linux and Windows profiling, with a graphical interface including source browsing

Process level

gprof	The GNU profiling tool, which analyzes profiling information added by compilers (e.g., gcc -pg).
cachegrind	A tool from the valgrind toolkit, can profile hardware cache usage (and more) and visualize profiles using kcachegrind
Java Flight Recorder（JER）	Programming languages often have their own special-purpose profilers that can inspect language context. For example, JFR for Java

4.2.3 Tracking tools

System level

tcpdump	Caught tools
biosnoop	Block I/O tracing (uses BCC or bpftrace)
execsnoop	New processes tracing (uses BCC or bpftrace)
perf	The standard Linux profiler, can also trace events
perf trace	A special perf subcommand that traces system calls system-wide
ftrace	The Linux built-in tracer
BCC	A BPF-based tracing library and toolkit
bpftrace	A BPF-based tracer (bpftrace(8)) and toolkit

Process level

strace	System call tracing
gdb	A source-level debugger

4.2.4 monitor

monitor General record keeping of tools statistics, For analysis and use .

sar	Collect, report, or save system activity information
snmp	Devices and operating systems can support SNMP and in some cases provide it by default, avoiding the need to install third-party agents or exporters
agents

4.3 Resources for observation

linux Observable resources , The main source is /proc and /sys Two directories .

linux Track resource summary

4.3.1 /proc file system

/proc is dynamically created by the kernel and is not backed by storage devices (it runs inmemory). It is mostly read-only, providing statistics for observability tools. Some files are writeable, for controlling process and kernel behavior.

Process level statistics

limits	Actual resource constraints
maps	Map memory area
sched	CPU Scheduler Statistics
schedstat	Get CPU The elapsed time 、 Time delay and time slice （runtime、latency、time slice）
smaps	Usage statistics of mapped memory area
stat	Process status and statistics , Including the overall CPU And memory usage
statm	With page Memory usage statistics per unit
status	stat and statm Information about , User readable
fd	（ The open ） File symlink Directory
cgroup	cgroup Group member information
task	Detailed data of each thread

System related statistics

cpuinfo	Physical processor information , Include each virtual CPU、 Name of manufacturer 、 Clock speed 、 Cache size
diskstats	Of all disks I/O statistics
interrupts	Every CPU Interrupt statistics for
loadavg	Load average
meminfo	System memory usage breakdown
net/dev	Network interface summary
net/netstat	The system level networking Data statistics
net/tcp	Activities of the TCP Socket information
pressure	Pressure stall information (PSI) files;cpu、io、memory Pressure blockage record , Analyze, for example OOM problem
schedstat	System level CPU Dispatch Statistics
self	Symbolic link of the current process
slabinfo	kernel slab Cache allocation usage
stat	Summary of kernel and system resource statistics ：CPUs、 disk 、 A page table 、swap、 process
zoneinfo	memory zone Information

4.3.2 /sys file system

differ /proc System ,/sys At first it was for statistics device driver statistics The design of the , But now it has also developed into comprehensive statistics .

4.3.4 Delay accounting

Kernel on CONFIG_TASK_DELAY_ACCT after , The following data will be counted for each task ：

• Scheduler latency: Scheduling delay , Wait to get CPU Time for
• Block I/O： block I/O, Waiting block I/O complete
• Swapping： In exchange for , Waiting for page change （ Memory pressure ）
• Memory reclaim： Memory recovery , Wait for the memory reclamation routine

kernel Documentation/accounting/delay-accounting.txt Help documentation in , And there is an example tools/accounting/getdelays.c

This is the data collected on a high load system ,CPU The delay is serious .

4.3.4 netlink

netlink Mechanism , One of the methods of user mode and kernel mode communication ,genetlink It's easier to expand .

4.3.5 tracepoints

Tracepoints are hard-coded instrumentation points placed at logical locations in kernel code.

give an example ：

In the system call start and end It's about 、 Scheduling events 、 File system operations 、 And disk I/O And so on tarcepoints. There are some tracepoint Kernel support needs to be enabled , such as CONFIG_RCU_TRACE Used to support rcu tracepoints.

tracepoint overhead（ Cost of trace points ）

Activated tracepoints after , Will increase CPU expenses 、 Document operation cost, etc , Does this extra overhead interfere with the performance data that the test cares about , Specific case specific analysis .

4.3.6 kprobes

kprobes (short for kernel probes) is a Linux kernel event source for tracers based on dynamic instrumentation.

kprobes You can trace any kernel function or instruction .

kprobe How to use ： The standard practice is to modify the instructions in the running kernel code to insert the monitoring points we want ; When measuring function entries, you can use the existing ftrace function , Reduce extra overhead expenses .

kprobes and tracepoints contrast ：

kprobe Observable function arguments ,kretprobes Observe the return value of the function

4.3.7 uprobes

uprobes (user-space probes) are similar to kprobes, but for user-space.

4.3.8 USDT

User-level statically-defined tracing (USDT) is the user-space version of tracepoints

4.3.9 Hardware Counters（PMCs）

The processor and other devices commonly support hardware counters for observing activity. The main source are the processors, where they are commonly called performance monitoring counters (PMCs). They are known by other names as well: CPU performance counters (CPCs), performance instrumentation counters (PICs), and performance monitoring unit events (PMU events). These all refer to the same thing: programmable hardware registers on the processor that provide low-level performance information at the CPU cycle level.

Programmable hardware registers on the processor , Can provide CPU Loop level system performance information ;

PMC challenges ：

• The accuracy of overflow sampling
• Availability issues in cloud environments

4.3.10 Other observation resources

MSR: model-specific registers;

ptrace: system call , By gdb For debugging , By strace For tracking

netfilter conntrack：netfilter Connection tracking mechanism ;

4.4 sar Tools

#  How to open sysstat Statistics ？
# vi /etc/default/sysstat
Enable="true"

[email protected]:~# sar -u -n TCP 3 3
Linux 5.4.0-58-generic (ubuntu) 	2020 year 12 month 28 Japan  	_x86_64_	(2 CPU)

18 when 05 branch 02 second      CPU     %user     %nice   %system   %iowait    %steal     %idle
18 when 05 branch 05 second      all     11.59      0.00     38.10      0.21      0.00     50.10

18 when 05 branch 02 second   active/s passive/s    iseg/s    oseg/s
18 when 05 branch 05 second       3.67      0.00     15.00     19.33

18 when 05 branch 05 second      CPU     %user     %nice   %system   %iowait    %steal     %idle
18 when 05 branch 08 second      all      9.64      0.00     32.13      3.61      0.00     54.62

18 when 05 branch 05 second   active/s passive/s    iseg/s    oseg/s
18 when 05 branch 08 second       4.33      0.00     92.33     95.33

18 when 05 branch 08 second      CPU     %user     %nice   %system   %iowait    %steal     %idle
18 when 05 branch 11 second      all     10.82      0.00     48.12      0.22      0.00     40.84

18 when 05 branch 08 second   active/s passive/s    iseg/s    oseg/s
18 when 05 branch 11 second       0.33      0.00     13.67     14.33

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all     10.67      0.00     39.19      1.39      0.00     48.74

Average:     active/s passive/s    iseg/s    oseg/s
Average:         2.78      0.00     40.33     43.00

4.5 tracing Tools

perf	Linux Official analysis tools , Good at CPU analysis （ Sampling analysis ） and PMC Statistics , Can also analyze other event event
ftrace	Linux Official tracking tools , You can run without dependency （ The kernel needs to be enabled CONFIG）
BPF	Extended BPF Tools ,BCC,bpftrace
system tap	A high-level language and tracer with many tapsets (libraries) for tracing different targets. Tools stapbpf Not yet studied
LTTng	A tracer optimized for black-box recording: optimally recording many events for later analysis

perf be used for CPU analysis ,ftrace For kernel code tracing ,BCC/bpftrace Used anywhere else （ Memory 、 file system 、 disk 、 Network and application tracking ）.