Linux Disaster Instruments

You potentially have an outage precipitated by a efficiency bother, you don’t desire to lose precious time correct to set up the tools desired to diagnose it. Here is an inventory of “crisis tools” I imply installing to your Linux servers by default (within the event that they don’t appear to be already), alongside with the (Ubuntu) equipment names that they arrive from:

Equipment	Provides	Notes
procps	ps(1), vmstat(8), uptime(1), high(1)	traditional stats
util-linux	dmesg(1), lsblk(1), lscpu(1)	machine log, tool information
sysstat	iostat(1), mpstat(1), pidstat(1), sar(1)	tool stats
iproute2	ip(8), ss(8), nstat(8), tc(8)	most in sort get hang of tools
numactl	numastat(8)	NUMA stats
tcpdump	tcpdump(8)	Community sniffer
linux-tools-total linux-tools-$(uname -r)	perf(1), turbostat(8)	profiler and PMU stats
bpfcc-tools (bcc)	opensnoop(8), execsnoop(8), runqlat(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), profile(8), and lots of others.	canned eBPF tools[1]
bpftrace	bpftrace, traditional variations of opensnoop(8), execsnoop(8), runqlat(8), biosnoop(8), and lots of others.	eBPF scripting[1]
trace-cmd	trace-cmd(1)	Ftrace CLI
nicstat	nicstat(1)	get hang of tool stats
ethtool	ethtool(8)	get hang of tool information
tiptop	tiptop(1)	PMU/PMC high
cpuid	cpuid(1)	CPU information
msr-tools	rdmsr(8), wrmsr(8)	CPU digging

(Here is in accordance to Table 4.1 “Linux Disaster Instruments” in SysPerf 2.)

Some longer notes: [1] bcc and bpftrace have many overlapping tools: the bcc ones are more capable (e.g., CLI strategies), and the bpftrace ones would possibly per chance per chance furthermore be edited on the cruise. But that’s no longer to inform that one is most attention-grabbing or faster than the diverse: They emit the same BPF bytecode and are equally quick as soon as working. Furthermore deliver that bcc is evolving and migrating tools from Python to libbpf C (with CO-RE and BTF) but we have no longer transformed the equipment but. In the damage “bpfcc-tools” have to smooth get hang of modified with a great smaller “libbpf-tools” equipment that’s correct tool binaries.

This checklist is a minimal. Some servers have accelerators and it’s doubtless you’ll per chance per chance desire their analysis tools put in as successfully: e.g., on Intel GPU servers, the intel-gpu-tools equipment; on NVIDIA, nvidia-smi. Debugging tools, esteem gdb(1), would possibly per chance per chance furthermore be pre-put in for on the spot declare in a crisis.

A must-have analysis tools esteem these don’t alternate that frequently, so this checklist would possibly per chance per chance also completely need updating each few years. When you happen to judge I overlooked a equipment that’s severe this day, please let me know (e.g., within the feedback).

The most critical downside of adding these packages is their on-disk size. On cloud cases, adding Mbytes to the sinful server image can add seconds, or fractions of a 2nd, to instance deployment time. Fortunately the packages I’ve listed are mostly reasonably runt (and bcc will get hang of smaller) and can tag miniature size and time. I’ve seen this size whisper prevent debuginfo (totaling round 1 Gbyte) from being incorporated by default.

Can no longer I correct set up them later when wanted?

Many considerations can happen when searching for to set up tool at some level of a production crisis. I am going to step thru a made-up instance that mixes one of the vital most things I’ve realized the laborious intention:

• 4:00pm: Alert! Your company’s aim goes down. No, some folk inform it’s smooth up. Is it up? It’s up but too tedious to be usable.
• 4:01pm: You look for at your monitoring dashboards and a neighborhood of backend servers are irregular. Is that high disk I/O? What’s causing that?
• 4:02pm: You SSH to 1 server to dig deeper, but SSH takes without a damage in sight to login.
• 4:03pm: You get hang of a login instant and form “iostat -xz 1” for traditional disk stats to open up with. There’s a protracted pause, and sooner or later “Uncover ‘iostat’ no longer stumbled on…Strive: sudo true set up sysstat”. Ugh. Given how tedious the machine is, installing this equipment would possibly per chance per chance also take lots of minutes. You traipse the set up inform.
• 4:07pm: The equipment set up has failed because it could per chance not resolve the repositories. One thing is inferior with the /and lots of others/true configuration. For the explanation that server dwelling owners are now within the SRE chatroom to abet with the outage, you query: “how stop you set up machine packages?” They retort “We by no intention stop. We completely replace our app.” Ugh. You scrutinize a particular server and replica its working /and lots of others/true config over.
• 4:10pm: You will need to traipse “true-get hang of replace” first with the mounted config, but it’s miserably tedious.
• 4:12pm: …have to smooth it undoubtedly be taking this long??
• 4:13pm: true returned “failed: Connection timed out.” Per chance this methodology is simply too tedious with the efficiency bother? Or can not it connect to the repos? You open up network debugging and query the server team: “Develop you use a firewall?” They are saying they don’t know, query the network security team.
• 4:17pm: The network security team have spoke back: Sure, they’ve blocked any sudden traffic, alongside with HTTP/HTTPS/FTP outbound true requests. Gah. “Are you able to edit the rules moral now?” “It’s no longer that straightforward.” “What about turning off the firewall thoroughly?” “Uh, in an emergency, certain.”
• 4:20pm: The firewall is disabled. You traipse true-get hang of replace again. It’s tedious, but works! Then true-get hang of set up, and…permission errors. What!? I am root, that is unnecessary. You fragment your error within the SRE chatroom and somebody facets out: Did not the platform security team save the machine immutable?
• 4:24pm: The platform security team are now within the SRE chatroom explaining that some parts of the file machine would possibly per chance per chance furthermore be written to, but others, notably for executable binaries, are blocked. Gah! “How can we disable this?” “You can not, that is the level. You will need to make fresh server pictures with it disabled.”
• 4:27pm: By now the SRE team has launched a vital outage and instructed the govt. team, who desire long-established situation updates and an ETA for when this would per chance per chance be mounted. Space: Haven’t carried out great but.
• 4:30pm: You open working “cat /proc/diskstats” as a rudimentary iostat(1), but have to utilize time reading the Linux source (admin-manual/iostats.rst) to save sense of it. It correct confirms the disks are busy which you knew anyway from the monitoring dashboard. You undoubtedly favor the disk and file machine tracing tools, esteem biosnoop(8), but it’s doubtless you’ll per chance per chance no longer set up them either. Except it’s doubtless you’ll per chance per chance also hack up rudimentary tracing tools as successfully…You “cd /sys/kernel/debug/tracing” and open taking a ogle the FTrace doctors.
• 4:55pm: Unique server pictures sooner or later open with all writable file systems. You login – gee it’s quick – and “true-get hang of set up sysstat”. Earlier than it’s doubtless you’ll per chance per chance also even traipse iostat there are messages within the chatroom: “Net pages’s again up! Thanks! What did you stop?” “We restarted the servers but we have no longer mounted anything else but.” You’ve gotten the feeling that the outage will return precisely 10 minutes after it’s doubtless you’ll per chance per chance also have fallen asleep tonight.
• 12:50am: Ping! I knew this would happen. You get hang of up and doing and commence your work laptop. The aim is down – it’s been hacked – somebody disabled the firewall and file machine security.

I’ve fortunately no longer skilled the 12:50am event, however the others are in accordance to proper world experiences. In my prior job this sequence can on the total take a particular flip: a “traffic team” would possibly per chance per chance also provoke a cloud aim failover by about the 15 minute designate, so I’d sooner or later get hang of iostat put in but then these systems would be indolent.

Default set up

The above bother explains why you ideally wish to pre-set up crisis tools so it’s doubtless you’ll per chance per chance also open debugging a production bother like a flash at some level of an outage. Some companies already stop this, and have OS teams that make customized server pictures with all the pieces incorporated. But there are reasonably tons of net sites smooth working default variations of Linux that be taught this the laborious intention. I’d imply Linux distros add these crisis tools to their endeavor Linux variants, in declare that companies colossal and runt can hit the ground working when efficiency outages happen.