2. Simple Observations - qyjohn/Hands-on-Linux GitHub Wiki
Launch an EC2 instance with the Ubuntu Server 20.04 LTS (HVM), SSD Volume Type AMI. In Step 4: Add Storage, change the size of the root EBS volume to 20, then use the Add New Volume button to add 2 x 100 EBS volumes to the EC2 instance. After the EC2 instance becomes Running, create two SSH sessions into the EC2 instance, then work on the rest of this tutorial.
You should at least know that you can use the top command to see what keeps the system busy:
$ top
Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 16006.9 total, 15382.3 free, 181.0 used, 443.5 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15549.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 168424 10004 5880 S 0.0 0.1 0:03.31 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-events_highpri
7 root 20 0 0 0 0 I 0.0 0.0 0:00.10 kworker/0:1-events
9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_tasks_rude_
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_tasks_trace
12 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/0
13 root 20 0 0 0 0 I 0.0 0.0 0:00.22 rcu_sched
14 root rt 0 0 0 0 S 0.0 0.0 0:00.03 migration/0
15 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_inject/0
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
18 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_inject/1
19 root rt 0 0 0 0 S 0.0 0.0 0:00.36 migration/1
20 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/1
22 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/1:0H-kblockd
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/2
Also, the uptime command tells you how long the system has been up, the number of active users, and the load average during the past 1, 5 and 15 minutes. The load average is an average of the number of runnable processes over a given time period. For example, for a system with a single CPU core, an hourly load average of 10 would mean that at any time during that hour one could expect to see 1 process running and 9 others ready to run (i.e., not blocked for I/O) waiting for the CPU. If you want to dive deep into this concept, please refer to Brendan Gregg's blog post Linux Load Averages: Solving the Mystery.
$ uptime
01:29:22 up 20:13, 1 user, load average: 0.00, 0.00, 0.00
Now let's use the following Java program (Compute.java) to see how load average changes:
/**
*
* An example to demonstrate the concept of load average by launching multiple compute-intensive threads.
* The number of threads is specified in the command line parameter.
*
*/
import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;
public class Compute extends Thread
{
public AtomicInteger counter;
public Compute(AtomicInteger c)
{
this.counter = c;
}
/**
*
* The run() method simple burns CPU by generating random numbers and perform some calculation.
*
*/
public void run()
{
Random r = new Random();
while (true)
{
for (int i=0; i<1000000; i++)
{
double a = r.nextDouble();
double b = r.nextDouble();
double c = a * b;
}
counter.addAndGet(1);
}
}
public static void main(String[] args)
{
String ANSI_RESET = "\u001B[0m";
String ANSI_BLUE = "\u001B[34m";
try
{
// This is used for counting in multiple threads.
AtomicInteger counter = new AtomicInteger();
// The number of threads, provided by command line parameter
int total = Integer.parseInt(args[0]);
for (int i=0; i<total; i++)
{
Compute t = new Compute(counter);
t.start();
}
int snapshot = 0, delta = 0;
while (true)
{
delta = counter.intValue() - snapshot;
snapshot = counter.intValue();
System.out.print("\r" + ANSI_BLUE + delta + ANSI_RESET + " million operations per second");
Thread.sleep(1000);
}
} catch (Exception e){}
}
}
Install JDK to compile the code, then run the code:
$ sudo apt update
$ sudo apt install openjdk-8-jdk-headless
$ javac Compute.java
$ java Compute 1
1 18 million operations
2 37 million operations
3 56 million operations
4 76 million operations
5 94 million operations
... ...
In a different SSH window, use top to observe the load average and CPU utilization metrics. Run the test program with a variety of threads (from 1 to 100) to see how the change impacts the observations.
You can view the number of CPU cores with the nproc command:
$ nproc
4
A lot more details about your CPU can be found from /proc/cpuinfo:
$ more /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping : 2
microcode : 0x46
cpu MHz : 2400.161
cache size : 30720 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good
nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ab
m cpuid_fault invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips : 4800.11
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
The free command gives you an overview of your memory:
$ free
total used free shared buff/cache available
Mem: 16391064 177724 15798212 836 415128 15933296
Swap: 0 0 0
More information about your memory can be found in /proc/meminfo:
$ more /proc/meminfo
MemTotal: 16391064 kB
MemFree: 15579180 kB
MemAvailable: 15931280 kB
Buffers: 25520 kB
Cached: 563536 kB
SwapCached: 0 kB
Active: 351388 kB
Inactive: 310868 kB
Active(anon): 788 kB
Inactive(anon): 81272 kB
Active(file): 350600 kB
Inactive(file): 229596 kB
Unevictable: 23000 kB
Mlocked: 18464 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 96248 kB
Mapped: 83364 kB
Shmem: 836 kB
KReclaimable: 44888 kB
Slab: 79128 kB
SReclaimable: 44888 kB
SUnreclaim: 34240 kB
KernelStack: 3072 kB
PageTables: 2488 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8195532 kB
Committed_AS: 801724 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 11760 kB
VmallocChunk: 0 kB
Percpu: 13056 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 81920 kB
DirectMap2M: 4112384 kB
DirectMap1G: 13631488 kB
To understand how fast you can access the memory, you can use mbw:
$ sudo apt install mbw -y
$ mbw 4000
Long uses 8 bytes. Allocating 2*524288000 elements = 8388608000 bytes of memory.
Using 262144 bytes as blocks for memcpy block copy test.
Getting down to business... Doing 10 runs per test.
0 Method: MEMCPY Elapsed: 0.75935 MiB: 4000.00000 Copy: 5267.691 MiB/s
1 Method: MEMCPY Elapsed: 0.75818 MiB: 4000.00000 Copy: 5275.757 MiB/s
2 Method: MEMCPY Elapsed: 0.75845 MiB: 4000.00000 Copy: 5273.907 MiB/s
3 Method: MEMCPY Elapsed: 0.75528 MiB: 4000.00000 Copy: 5296.014 MiB/s
4 Method: MEMCPY Elapsed: 0.75734 MiB: 4000.00000 Copy: 5281.623 MiB/s
5 Method: MEMCPY Elapsed: 0.75786 MiB: 4000.00000 Copy: 5278.013 MiB/s
6 Method: MEMCPY Elapsed: 0.75794 MiB: 4000.00000 Copy: 5277.497 MiB/s
7 Method: MEMCPY Elapsed: 0.76026 MiB: 4000.00000 Copy: 5261.365 MiB/s
8 Method: MEMCPY Elapsed: 0.75936 MiB: 4000.00000 Copy: 5267.622 MiB/s
9 Method: MEMCPY Elapsed: 0.76210 MiB: 4000.00000 Copy: 5248.669 MiB/s
AVG Method: MEMCPY Elapsed: 0.75861 MiB: 4000.00000 Copy: 5272.788 MiB/s
0 Method: DUMB Elapsed: 0.45946 MiB: 4000.00000 Copy: 8705.948 MiB/s
1 Method: DUMB Elapsed: 0.45880 MiB: 4000.00000 Copy: 8718.434 MiB/s
2 Method: DUMB Elapsed: 0.46057 MiB: 4000.00000 Copy: 8684.985 MiB/s
3 Method: DUMB Elapsed: 0.45746 MiB: 4000.00000 Copy: 8743.877 MiB/s
4 Method: DUMB Elapsed: 0.45511 MiB: 4000.00000 Copy: 8789.142 MiB/s
5 Method: DUMB Elapsed: 0.46096 MiB: 4000.00000 Copy: 8677.561 MiB/s
6 Method: DUMB Elapsed: 0.46085 MiB: 4000.00000 Copy: 8679.689 MiB/s
7 Method: DUMB Elapsed: 0.46171 MiB: 4000.00000 Copy: 8663.428 MiB/s
8 Method: DUMB Elapsed: 0.46016 MiB: 4000.00000 Copy: 8692.572 MiB/s
9 Method: DUMB Elapsed: 0.46096 MiB: 4000.00000 Copy: 8677.599 MiB/s
AVG Method: DUMB Elapsed: 0.45960 MiB: 4000.00000 Copy: 8703.173 MiB/s
0 Method: MCBLOCK Elapsed: 0.71471 MiB: 4000.00000 Copy: 5596.715 MiB/s
1 Method: MCBLOCK Elapsed: 0.71161 MiB: 4000.00000 Copy: 5621.056 MiB/s
2 Method: MCBLOCK Elapsed: 0.71467 MiB: 4000.00000 Copy: 5597.012 MiB/s
3 Method: MCBLOCK Elapsed: 0.71459 MiB: 4000.00000 Copy: 5597.647 MiB/s
4 Method: MCBLOCK Elapsed: 0.71345 MiB: 4000.00000 Copy: 5606.528 MiB/s
5 Method: MCBLOCK Elapsed: 0.71504 MiB: 4000.00000 Copy: 5594.077 MiB/s
6 Method: MCBLOCK Elapsed: 0.71486 MiB: 4000.00000 Copy: 5595.501 MiB/s
7 Method: MCBLOCK Elapsed: 0.70483 MiB: 4000.00000 Copy: 5675.168 MiB/s
8 Method: MCBLOCK Elapsed: 0.71111 MiB: 4000.00000 Copy: 5625.009 MiB/s
9 Method: MCBLOCK Elapsed: 0.71719 MiB: 4000.00000 Copy: 5577.315 MiB/s
AVG Method: MCBLOCK Elapsed: 0.71320 MiB: 4000.00000 Copy: 5608.486 MiB/s
At this point there is no swap on the system yet. Let's create a 1 GB swap file and use the file as a swap.
$ sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.08437 s, 348 MB/s
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=1fe871cd-f72c-411f-a109-5acfda85f525
$ sudo swapon /swapfile
$ free
total used free shared buff/cache available
Mem: 16391064 194056 14584488 844 1612520 15861340
Swap: 1048572 0 1048572
If you would like to see swap only, you can simply use the swapon command.
$ sudo swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 1024M 0B -2
You should also review what is in /proc/meminfo to see the differences, as below:
$ more /proc/meminfo | grep Swap
SwapCached: 0 kB
SwapTotal: 1048572 kB
SwapFree: 1048572 kB
Swappiness is a Linux kernel property that defines how often the system will use the swap space. Swappiness can have a value between 0 and 100. A low value will make the kernel to try to avoid swapping whenever possible, while a higher value will make the kernel to use the swap space more aggressively. The default swappiness value is 60. You can check the current swappiness value by looking into /proc/sys/vm/swappiness:
$ more /proc/sys/vm/swappiness
60
To set the swappiness value to 10, you would run the following sysctl command:
$ sudo sysctl vm.swappiness=10
To make this parameter persistent across reboots, you need to append the configuration to /etc/sysctl.conf:
vm.swappiness=10
To turn off swap, use swapoff:
$ sudo swapoff -v /swapfile
swapoff /swapfile
$ sudo swapon --show
You already know that the top command gives you the top CPU and memory consumers. By default the output is ordered by CPU usage. If you want the output to be ordered by MEM usage, you can press SHIFT + m. To switch back to ordered by CPU usage, press SHIFT +p. The following is an example of the top output, order by MEM usage. You should note that the processed are ordered by RES (residence memory) instead of VIRT (virtual memory).
top - 05:51:14 up 5:55, 2 users, load average: 0.00, 0.12, 0.20
Tasks: 128 total, 1 running, 127 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 16006.9 total, 244.7 free, 192.7 used, 15569.5 buff/cache
MiB Swap: 1024.0 total, 1024.0 free, 0.0 used. 15475.0 avail Mem
scroll coordinates: y = 1/128 (tasks), x = 1/12 (fields)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
560 root 20 0 1221900 19880 5920 S 0.0 0.1 0:02.38 snapd
336 root rt 0 280144 17916 8204 S 0.0 0.1 0:02.40 multipathd
1276 root 20 0 1469260 17672 4296 S 0.0 0.1 0:01.66 ssm-agent-worke
556 root 20 0 29204 10968 3304 S 0.0 0.1 0:00.07 networkd-dispat
184 root 19 -1 53344 10808 9776 S 0.0 0.1 0:00.44 systemd-journal
631 root 20 0 108116 10808 3144 S 0.0 0.1 0:00.06 unattended-upgr
1 root 20 0 168564 10628 6412 S 0.0 0.1 0:03.90 systemd
1235 root 20 0 1456612 9640 1860 S 0.0 0.1 0:01.76 amazon-ssm-agen
7062 root 20 0 13804 8868 7424 S 0.0 0.1 0:00.01 sshd
563 root 20 0 394700 7912 6052 S 0.0 0.0 0:00.06 udisksd
453 systemd+ 20 0 23904 7776 3732 S 0.0 0.0 0:00.12 systemd-resolve
1321 ubuntu 20 0 18288 6712 5320 S 0.0 0.0 0:00.03 systemd
7159 ubuntu 20 0 13936 5884 4420 S 0.0 0.0 0:00.16 sshd
7160 ubuntu 20 0 10040 5176 3472 S 0.0 0.0 0:00.12 bash
217 root 20 0 19620 4984 2988 S 0.0 0.0 0:00.18 systemd-udevd
1423 ubuntu 20 0 10168 4780 2992 S 0.0 0.0 0:00.13 bash
367 systemd+ 20 0 90008 4512 3740 S 0.0 0.0 0:00.08 systemd-timesyn
450 systemd+ 20 0 26612 4332 3388 S 0.0 0.0 0:00.13 systemd-network
1322 ubuntu 20 0 169640 4264 0 S 0.0 0.0 0:00.00 (sd-pam)
1318 root 20 0 13804 4152 2708 S 0.0 0.0 0:00.00 sshd
754 root 20 0 12184 4036 3112 S 0.0 0.0 0:00.01 sshd
659 root 20 0 236432 4032 3100 S 0.0 0.0 0:00.02 polkitd
538 root 20 0 241044 3896 2900 S 0.0 0.0 0:00.41 accounts-daemon
1546729 ubuntu 20 0 11012 3892 3248 R 0.0 0.0 0:00.17 top
561 root 20 0 16840 3836 2968 S 0.0 0.0 0:00.16 systemd-logind
558 syslog 20 0 224508 3792 2528 S 0.0 0.0 0:00.06 rsyslogd
544 message+ 20 0 7548 3496 2836 S 0.0 0.0 0:00.20 dbus-daemon
1422 ubuntu 20 0 13936 2776 1308 S 0.0 0.0 0:01.04 sshd
555 root 20 0 81836 2584 2284 S 0.0 0.0 0:00.78 irqbalance
543 root 20 0 8544 2564 2288 S 0.0 0.0 0:00.03 cron
564 daemon 20 0 3800 2108 1928 S 0.0 0.0 0:00.00 atd
600 root 20 0 7360 1752 1628 S 0.0 0.0 0:00.01 agetty
617 root 20 0 5836 1540 1428 S 0.0 0.0 0:00.02 agetty
539 root 20 0 2548 668 600 S 0.0 0.0 0:00.00 acpid
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
To understand the impact of swapping on application, do the following test with mbw. Here we use X000 to represent 1/2 of your total memory. For example, if you have 8 GB RAM than X000 = 4000.
-
With swap on, perform an mbw test with a parameter that is slightly less than X.
-
With swap on, perform an mbw test with a parameter that is slightly more than X.
-
With swap off, perform an mbw test with a parameter that is slightly more than X.
Use lsblk to see the block devices on the system. You have three disks (xvda, xvdb, xvdc) on the system. Under disk xvda there is a partition xvda1, which is mounted to /. Disks xvdb and xvdc do not have any partitions and they are not mounted.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 25M 1 loop /snap/amazon-ssm-agent/4046
loop1 7:1 0 55.4M 1 loop /snap/core18/2128
loop2 7:2 0 67.3M 1 loop /snap/lxd/21545
loop3 7:3 0 61.9M 1 loop /snap/core20/1169
loop4 7:4 0 32.5M 1 loop /snap/snapd/13640
xvda 202:0 0 20G 0 disk
ββxvda1 202:1 0 20G 0 part /
xvdb 202:16 0 100G 0 disk
xvdc 202:32 0 100G 0 disk
If you want to see size in bytes, use the "-b" flag. Note that 1 GB is 1024 x 1024 x 1024 = 1073741824 bytes.
$ lsblk -b
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 26189824 1 loop /snap/amazon-ssm-agent/4046
loop1 7:1 0 58130432 1 loop /snap/core18/2128
loop2 7:2 0 70516736 1 loop /snap/lxd/21545
loop3 7:3 0 64835584 1 loop /snap/core20/1169
loop4 7:4 0 34017280 1 loop /snap/snapd/13640
xvda 202:0 0 21474836480 0 disk
ββxvda1 202:1 0 21473771008 0 part /
xvdb 202:16 0 107374182400 0 disk
xvdc 202:32 0 107374182400 0 disk
Use mount to see what are mounted:
$ mount
/dev/xvda1 on / type ext4 (rw,relatime,discard)
devtmpfs on /dev type devtmpfs (rw,relatime,size=8189548k,nr_inodes=2047387,mode=755,inode64)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=1639108k,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16690)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/var/lib/snapd/snaps/core18_2128.snap on /snap/core18/2128 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/snaps/amazon-ssm-agent_4046.snap on /snap/amazon-ssm-agent/4046 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/snaps/snapd_13640.snap on /snap/snapd/13640 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/snaps/lxd_21545.snap on /snap/lxd/21545 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/snaps/core20_1169.snap on /snap/core20/1169 type squashfs (ro,nodev,relatime,x-gdu.hide)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,size=1639108k,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1639104k,mode=700,uid=1000,gid=1000,inode64)
Use df -i to see the mounted storage devices:
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 2560000 77586 2482414 4% /
devtmpfs 2047387 335 2047052 1% /dev
tmpfs 2048883 2 2048881 1% /dev/shm
tmpfs 2048883 518 2048365 1% /run
tmpfs 2048883 3 2048880 1% /run/lock
tmpfs 2048883 18 2048865 1% /sys/fs/cgroup
/dev/loop1 10803 10803 0 100% /snap/core18/2128
/dev/loop0 16 16 0 100% /snap/amazon-ssm-agent/4046
/dev/loop4 479 479 0 100% /snap/snapd/13640
/dev/loop2 796 796 0 100% /snap/lxd/21545
/dev/loop3 11732 11732 0 100% /snap/core20/1169
tmpfs 2048883 22 2048861 1% /run/user/1000
All Unix filesystems use two basic components to organize and store data: blocks and inodes. Just as a physical disk is organized into sectors, data on a filesystem is abstracted into blocks. Blocks have a fixed size, determined at the time the filesystem is created. The block size of a filesystem determines how many bytes are allocated to each block on the filesystem. Inodes are used to map blocks to physical disk locations on Unix filesystems. Every file created, whether itβs a directory, normal file, or special file, is assigned an inode. Inodes work in much the same way as pointers do in programming languages such as C. Inodes also store information about a file, including its type, size, and parent directory.
You can use the tune2fs command to find out information about blocks and inodes for a particular file system:
$ sudo tune2fs -l /dev/root
tune2fs 1.45.5 (07-Jan-2020)
Filesystem volume name: cloudimg-rootfs
Last mounted on: /
Filesystem UUID: 2a29f520-1100-4824-b5d9-d841f1267838
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 2560000
Block count: 5242619
Reserved block count: 0
Free blocks: 4713831
Free inodes: 2487491
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 253
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16000
Inode blocks per group: 1000
Flex block group size: 16
Filesystem created: Thu Oct 21 23:27:59 2021
Last mount time: Wed Dec 8 23:55:47 2021
Last write time: Thu Oct 21 23:30:40 2021
Mount count: 1
Maximum mount count: -1
Last checked: Thu Oct 21 23:30:39 2021
Check interval: 0 (<none>)
Lifetime writes: 2503 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 4945c69a-2ba2-4917-97e0-492ef27c0f45
Journal backup: inode blocks
Checksum type: crc32c
Checksum: 0x0b2afb06
Now let's do some test to see how fast the EBS volumes are. Here we use iostat, which is part of the sysstat package. In one of the SSH window, do the following to install *sysstat and monitor disk I/O with a 1-second refresh interval.
$ sudo apt install sysstat -y
$ iostat 1
In the other SSH window, use dd to do some writes on xvdb.
$ sudo dd if=/dev/zero of=/dev/xvdb bs=1M count=20000 oflag=direct
In the SSH window running iostat 1, you should see output similar to the following. On xvdb, we are seeing about 1000 IO requests per second (TPS), with a write speed of 125 MB/s. The system spends 0.50% CPU time on making system calls, 1.25% CPU time on waiting for disk I/O, 0.50% of the CPU time is stolen by the underlying virtualization layer, and is idling in 97.76% of the time.
$ iostat 1
... ...
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.50 1.25 0.50 97.76
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.00 0.00 0.00 0.00 0 0 0
loop1 0.00 0.00 0.00 0.00 0 0 0
loop2 0.00 0.00 0.00 0.00 0 0 0
loop3 0.00 0.00 0.00 0.00 0 0 0
loop4 0.00 0.00 0.00 0.00 0 0 0
loop5 0.00 0.00 0.00 0.00 0 0 0
xvda 0.00 0.00 0.00 0.00 0 0 0
xvdb 977.00 0.00 125056.00 0.00 0 125056 0
xvdc 0.00 0.00 0.00 0.00 0 0 0
After dd completes, it produces a summary of test result.
$ sudo dd if=/dev/zero of=/dev/xvdb bs=1M count=20000 oflag=direct
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 162.852 s, 129 MB/s
Now let's create an EXT4 filesystem on xvdb and mount it to /mnt.
$ sudo mkfs.ext4 /dev/xvdb
mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: 9ba73b96-a8d9-4f7b-bf85-46f278b7f237
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mount /dev/xvdb /mnt
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 25M 1 loop /snap/amazon-ssm-agent/4046
loop1 7:1 0 55.4M 1 loop /snap/core18/2128
loop2 7:2 0 67.3M 1 loop /snap/lxd/21545
loop3 7:3 0 61.9M 1 loop /snap/core20/1169
loop4 7:4 0 32.5M 1 loop /snap/snapd/13640
xvda 202:0 0 20G 0 disk
ββxvda1 202:1 0 20G 0 part /
xvdb 202:16 0 100G 0 disk /mnt
xvdc 202:32 0 100G 0 disk
Now let's create a 20 GB file on /mnt to see how fast we can go. In one of the SSH window we do the following:
$ sudo dd if=/dev/zero of=/mnt/testfile bs=1M count=20000 oflag=direct
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 162.857 s, 129 MB/s
While waiting fort dd to complete, we observe the disk I/O with iostat 1. In the following example, we are spending 5% of CPU time in iowait (this number does change over time).
$ iostat 1
... ...
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.50 5.26 0.25 93.98
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.00 0.00 0.00 0.00 0 0 0
loop1 0.00 0.00 0.00 0.00 0 0 0
loop2 0.00 0.00 0.00 0.00 0 0 0
loop3 0.00 0.00 0.00 0.00 0 0 0
loop4 0.00 0.00 0.00 0.00 0 0 0
loop5 0.00 0.00 0.00 0.00 0 0 0
xvda 0.00 0.00 0.00 0.00 0 0 0
xvdb 951.00 0.00 121728.00 0.00 0 121728 0
xvdc 0.00 0.00 0.00 0.00 0 0 0
Now let's see how fast we can read from that file:
$ sudo dd if=/mnt/testfile of=/dev/null bs=1M
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 162.845 s, 129 MB/s
While waiting fort dd to complete, we observe the disk I/O with iostat 1.
$ iostat 1
... ...
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 2.01 22.31 0.50 75.19
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.00 0.00 0.00 0.00 0 0 0
loop1 0.00 0.00 0.00 0.00 0 0 0
loop2 0.00 0.00 0.00 0.00 0 0 0
loop3 0.00 0.00 0.00 0.00 0 0 0
loop4 0.00 0.00 0.00 0.00 0 0 0
loop5 0.00 0.00 0.00 0.00 0 0 0
xvda 0.00 0.00 0.00 0.00 0 0 0
xvdb 977.00 125056.00 0.00 0.00 125056 0 0
xvdc 0.00 0.00 0.00 0.00 0 0 0
Assuming that you are sharing the same system with some other users. You notice some iowait when performing iostat 1. How do you identify which user or process is doing that? In this case, iotop is going to be very handy. Let's run a dd again to write a 20 GB file into /mnt/testfile2, then use iotop to observe the situation:
$ sudo apt install iotop -y
$ sudo iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 122.36 M/s
Current DISK READ: 0.00 B/s | Current DISK WRITE: 122.36 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1546297 be/4 root 0.00 B/s 122.36 M/s 0.00 % 97.78 % dd if=/dev/zero of=/mnt/t~ count=20000 oflag=direct
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
6 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H-events_highpri]
7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:1-events]
9 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq]
10 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_rude_]
11 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_trace]
12 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
13 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
14 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
15 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [idle_inject/0]
16 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/0]
17 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/1]
18 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [idle_inject/1]
19 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
20 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1]
22 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H-kblockd]
23 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/2]
24 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [idle_inject/2]
25 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/2]
keys: any: refresh q: quit i: ionice o: active p: procs a: accum
sort: r: asc left: SWAPIN right: COMMAND home: TID end: COMMAND
The dd command is good enough in determining the capacity of a block device (and filesystem). However, dd runs in a single-thread fashion. If we want to see how the underlying storage performs under multiple workload, IOZone is a good tool to use.
$ sudo apt install iozone3 -y
Now let's use IOZone to perform 5 writes in parallel:
$ sudo iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
If we observe the situation with iostat 1, we will see that we are achieveing approximately 1000 TPS and 125 MB/s in write throughput. However, we are now spending 98% of our CPU time in iowait. That is, with multiple threads competing for disk I/O, disk I/O capacity becomes the major bottleneck in achieving better performance.
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 1.30 98.44 0.26 0.00
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.00 0.00 0.00 0.00 0 0 0
loop1 0.00 0.00 0.00 0.00 0 0 0
loop2 0.00 0.00 0.00 0.00 0 0 0
loop3 0.00 0.00 0.00 0.00 0 0 0
loop4 0.00 0.00 0.00 0.00 0 0 0
loop5 0.00 0.00 0.00 0.00 0 0 0
xvda 0.00 0.00 0.00 0.00 0 0 0
xvdb 978.00 0.00 124936.00 0.00 0 124936 0
xvdc 0.00 0.00 0.00 0.00 0 0 0
If we observe the situation with iotop, we can see 5 processes are created by iozone.
Total DISK READ: 121.59 M/s | Total DISK WRITE: 3.83 K/s
Current DISK READ: 122.14 M/s | Current DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1547254 be/4 root 24.39 M/s 0.00 B/s 0.00 % 97.73 % iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
1547257 be/4 root 24.27 M/s 0.00 B/s 0.00 % 97.52 % iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
1547253 be/4 root 24.39 M/s 0.00 B/s 0.00 % 97.48 % iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
1547256 be/4 root 24.27 M/s 0.00 B/s 0.00 % 97.19 % iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
1547255 be/4 root 24.27 M/s 0.00 B/s 0.00 % 97.14 % iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
1547606 be/4 root 7.65 K/s 3.83 K/s 0.00 % 1.75 % sh /usr/sbin/mkinitramfs -o /boot/init~mg-5.11.0-1022-aws.new 5.11.0-1022-aws
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
6 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H-events_highpri]
9 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq]
10 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_rude_]
We can identify the iozone process with the ps command, then visualize the relationship between the above-mentioned processes with pstree.
$ ps -ef | grep iozone
root 1547234 1423 0 06:06 pts/0 00:00:00 sudo iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
root 1547235 1547234 0 06:06 pts/0 00:00:00 iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
root 1552196 1547235 2 06:14 pts/0 00:00:02 iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
root 1552197 1547235 2 06:14 pts/0 00:00:02 iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
root 1552198 1547235 2 06:14 pts/0 00:00:02 iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
root 1552199 1547235 2 06:14 pts/0 00:00:02 iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
root 1552200 1547235 2 06:14 pts/0 00:00:02 iozone -s 4000000 -l5 -u5 -F /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5
ubuntu 1560730 7160 0 06:16 pts/1 00:00:00 grep --color=auto iozone
$ pstree -p 1547234
sudo(1547234)βββiozone(1547235)ββ¬βiozone(1552196)
ββiozone(1552197)
ββiozone(1552198)
ββiozone(1552199)
ββiozone(1552200)
First of all, you need to know your network devices and their IP addresses. This requires the net-tools package to be installed.
$ sudo apt install net-tools -y
You should at least know ifconfig, which give you a lot of information about network interfaces, IP address, netmast, etc. For eth0, the default MTU is 9001. We encourage you to read EC2 MTU for more information.
$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 172.31.77.36 netmask 255.255.240.0 broadcast 172.31.79.255
inet6 fe80::cf:72ff:fe3c:5d41 prefixlen 64 scopeid 0x20<link>
ether 02:cf:72:3c:5d:41 txqueuelen 1000 (Ethernet)
RX packets 120363 bytes 130552208 (130.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 41177 bytes 12482698 (12.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 402 bytes 36532 (36.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 402 bytes 36532 (36.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Now let's look at the route table on the operating system level:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-172-31-64-1. 0.0.0.0 UG 100 0 0 eth0
172.31.64.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
ip-172-31-64-1. 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
Your DNS configuration is in /etc/resolv.conf. The following configuration says your DNS server is 127.0.0.53, which is a DNS cache process running on the same EC2 instance (systemd-resolved).
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search ec2.internal
To know more about systemd-resolved, use the following command:
$ sudo systemctl status systemd-resolved
β systemd-resolved.service - Network Name Resolution
Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-12-08 23:55:51 UTC; 6h ago
Docs: man:systemd-resolved.service(8)
https://www.freedesktop.org/wiki/Software/systemd/resolved
https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 453 (systemd-resolve)
Status: "Processing requests..."
Tasks: 1 (limit: 19194)
Memory: 4.9M
CGroup: /system.slice/systemd-resolved.service
ββ453 /lib/systemd/systemd-resolved
Dec 08 23:55:51 ip-172-31-77-36 systemd[1]: Starting Network Name Resolution...
Dec 08 23:55:51 ip-172-31-77-36 systemd-resolved[453]: Positive Trust Anchors:
Dec 08 23:55:51 ip-172-31-77-36 systemd-resolved[453]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 08 23:55:51 ip-172-31-77-36 systemd-resolved[453]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.>
Dec 08 23:55:51 ip-172-31-77-36 systemd-resolved[453]: Using system hostname 'ip-172-31-77-36'.
Dec 08 23:55:51 ip-172-31-77-36 systemd[1]: Started Network Name Resolution.
The systemd-resolved is a service daemon and it listens on port 53. To see what ports are open on the EC2 instance, netstat -l can be quite handy. In the following output, domain refers to port 53, which is used for DNS server, ssh refers to port 22, which is used for SSH server, bootpc refers to port 68. The mappings between port number and names come from configuration file /etc/services.
$ netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:domain 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
udp 0 0 localhost:domain 0.0.0.0:*
udp 0 0 ip-172-31-77-36.:bootpc 0.0.0.0:*
raw6 0 0 [::]:ipv6-icmp [::]:* 7
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 24554 /run/user/1000/systemd/private
unix 2 [ ACC ] STREAM LISTENING 24559 /run/user/1000/bus
unix 2 [ ACC ] STREAM LISTENING 24560 /run/user/1000/gnupg/S.dirmngr
unix 2 [ ACC ] STREAM LISTENING 16694 @/org/kernel/linux/storage/multipathd
unix 2 [ ACC ] STREAM LISTENING 24561 /run/user/1000/gnupg/S.gpg-agent.browser
unix 2 [ ACC ] STREAM LISTENING 24562 /run/user/1000/gnupg/S.gpg-agent.extra
unix 2 [ ACC ] STREAM LISTENING 24563 /run/user/1000/gnupg/S.gpg-agent.ssh
unix 2 [ ACC ] STREAM LISTENING 24564 /run/user/1000/gnupg/S.gpg-agent
unix 2 [ ACC ] STREAM LISTENING 24565 /run/user/1000/pk-debconf-socket
unix 2 [ ACC ] STREAM LISTENING 24566 /run/user/1000/snapd-session-agent.socket
unix 2 [ ACC ] STREAM LISTENING 16680 /run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 16682 /run/systemd/userdb/io.systemd.DynamicUser
unix 2 [ ACC ] STREAM LISTENING 16692 /run/lvm/lvmpolld.socket
unix 2 [ ACC ] STREAM LISTENING 16697 /run/systemd/fsck.progress
unix 2 [ ACC ] STREAM LISTENING 21690 /run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 16707 /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 21692 /run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 21695 /run/snapd.socket
unix 2 [ ACC ] SEQPACKET LISTENING 16712 /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 21697 /run/snapd-snap.socket
unix 2 [ ACC ] STREAM LISTENING 21699 /run/uuidd/request
unix 2 [ ACC ] STREAM LISTENING 15569 /run/systemd/journal/io.systemd.journal
unix 2 [ ACC ] STREAM LISTENING 22774 /run/irqbalance//irqbalance555.sock
unix 2 [ ACC ] STREAM LISTENING 24084 /var/snap/lxd/common/lxd/unix.socket
unix 2 [ ACC ] STREAM LISTENING 26749 /var/lib/amazon/ssm/ipc/health
unix 2 [ ACC ] STREAM LISTENING 26750 /var/lib/amazon/ssm/ipc/termination
unix 2 [ ACC ] STREAM LISTENING 21694 @ISCSIADM_ABSTRACT_NAMESPACE
Now let's do a few DNS resolution to see what happens. The domain name under test is www.amazon.com. The first two queries returned 54.192.29.129, with different TTL. The next two queries returned 162.219.225.118, with different TTL.
$ dig www.amazon.com
; <<>> DiG 9.16.1-Ubuntu <<>> www.amazon.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60287
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.amazon.com. IN A
;; ANSWER SECTION:
www.amazon.com. 229 IN CNAME tp.47cf2c8c9-frontier.amazon.com.
tp.47cf2c8c9-frontier.amazon.com. 5 IN CNAME d3ag4hukkh62yn.cloudfront.net.
d3ag4hukkh62yn.cloudfront.net. 43 IN A 54.192.29.129
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Dec 09 07:13:21 UTC 2021
;; MSG SIZE rcvd: 138
$ dig www.amazon.com
; <<>> DiG 9.16.1-Ubuntu <<>> www.amazon.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10424
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.amazon.com. IN A
;; ANSWER SECTION:
www.amazon.com. 223 IN CNAME tp.47cf2c8c9-frontier.amazon.com.
tp.47cf2c8c9-frontier.amazon.com. 0 IN CNAME d3ag4hukkh62yn.cloudfront.net.
d3ag4hukkh62yn.cloudfront.net. 38 IN A 54.192.29.129
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Dec 09 07:13:27 UTC 2021
;; MSG SIZE rcvd: 138
$ dig www.amazon.com
; <<>> DiG 9.16.1-Ubuntu <<>> www.amazon.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46537
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.amazon.com. IN A
;; ANSWER SECTION:
www.amazon.com. 222 IN CNAME tp.47cf2c8c9-frontier.amazon.com.
tp.47cf2c8c9-frontier.amazon.com. 33 IN CNAME www-amazon-com.customer.fastly.net.
www-amazon-com.customer.fastly.net. 299 IN A 162.219.225.118
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Dec 09 07:13:28 UTC 2021
;; MSG SIZE rcvd: 143
$ dig www.amazon.com
; <<>> DiG 9.16.1-Ubuntu <<>> www.amazon.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38949
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.amazon.com. IN A
;; ANSWER SECTION:
www.amazon.com. 202 IN CNAME tp.47cf2c8c9-frontier.amazon.com.
tp.47cf2c8c9-frontier.amazon.com. 13 IN CNAME www-amazon-com.customer.fastly.net.
www-amazon-com.customer.fastly.net. 280 IN A 162.219.225.118
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Dec 09 07:13:48 UTC 2021
;; MSG SIZE rcvd: 143
If you want to know the network bandwidth between two EC2 instance, you can use iperf. In short, you install iperf on both EC2 instances, then start iperf as the server on one of them, then start iperf as the client on the other. This tool is quit useful in troubleshooting situations where your latency is low but your throughput is high. In this example, we simply use two SSH windows for a demo, one as the server and the other as the client.
$ sudo apt install iperf -y
In one of the SSH window, run the iperf server:
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
In the other SSH window, run the iperf client. In this example, we see 20.1 Gbits/sec because both the server and the client are running on the same EC2 instance.
$ iperf -c 172.31.77.36
------------------------------------------------------------
Client connecting to 172.31.77.36, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 172.31.77.36 port 56886 connected with 172.31.77.36 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 23.3 GBytes 20.1 Gbits/sec
To monitor the network traffic in and out of your EC2 instance, you might use iftop as a quick start.
$ sudo apt-get install iftop -y
The following output is an example of what happens during an iperf test between two different EC2 instances in the same VPC.
$ sudo iftop
interface: eth0
IP address is: 172.31.77.36
MAC address is: 02:cf:72:3c:5d:41
1.86Gb 3.73Gb 5.59Gb 7.45Gb 9.31Gb
ββββββββββββββββββββββ΄βββββββββββββββββββββ΄βββββββββββββββββββββ΄βββββββββββββββββββββ΄βββββββββββββββββββββ
ip-172-31-77-36.ec2.internal => 72-21-198-65.amazon.com 608b 17.0Kb 7.93Kb
<= 208b 3.15Kb 1.34Kb
ip-172-31-77-36.ec2.internal => ip-172-31-64-126.ec2.internal 0b 0b 651Kb
<= 0b 0b 243Mb
ip-172-31-77-36.ec2.internal => ip-172-31-0-2.ec2.internal 0b 0b 17b
<= 0b 0b 25b
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
TX: cum: 3.27MB peak: 2.58Mb rates: 608b 17.0Kb 659Kb
RX: 1.19GB 992Mb 208b 3.15Kb 243Mb
TOTAL: 1.19GB 994Mb 816b 20.1Kb 244Mb
Now let's talk a look at the topology of the system:
$ sudo apt-get update -y
$ sudo apt install hwloc -y
$ sudo lstopo
Machine (16GB total)
Package L#0
NUMANode L#0 (P#0 16GB)
L3 L#0 (30MB)
L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#1)
L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#2)
L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#3)
HostBridge
PCI 00:01.1 (IDE)
PCI 00:02.0 (VGA)
Block "xvdc"
Block "xvda"
Block "xvdb"
Net "eth0"
The lstopo-no-graphics -.ascii command produces output in ASC arts.
$ sudo lstopo-no-graphics -.ascii
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Machine (16GB total) β
β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ ββ€βΆββ¬ββββββ¬ββββββββββββββ β
β β Package L#0 β β β PCI 00:01.1 β β
β β β β βββββββββββββββ β
β β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β
β β β NUMANode L#0 P#0 (16GB) β β βββββββ¬ββββββββββββββ β
β β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β PCI 00:02.0 β β
β β β βββββββββββββββ β
β β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β
β β β L3 (30MB) β β β
β β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β
β β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β L2 (256KB) β β L2 (256KB) β β L2 (256KB) β β L2 (256KB) β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β L1d (32KB) β β L1d (32KB) β β L1d (32KB) β β L1d (32KB) β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β L1i (32KB) β β L1i (32KB) β β L1i (32KB) β β L1i (32KB) β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β Core L#0 β β Core L#1 β β Core L#2 β β Core L#3 β β β
β β β β β β β β β β β β
β β β ββββββββββ β β ββββββββββ β β ββββββββββ β β ββββββββββ β β β
β β β β PU L#0 β β β β PU L#1 β β β β PU L#2 β β β β PU L#3 β β β β
β β β β β β β β β β β β β β β β β β β β
β β β β P#0 β β β β P#1 β β β β P#2 β β β β P#3 β β β β
β β β ββββββββββ β β ββββββββββ β β ββββββββββ β β ββββββββββ β β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β ββββββββββ βββββββββ β
β β xvdc β β xvda β β
β β β β β β
β β 100 GB β β 20 GB β β
β ββββββββββ βββββββββ β
β β
β ββββββββββ ββββββββ β
β β xvdb β β eth0 β β
β β β ββββββββ β
β β 100 GB β β
β ββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Host: ip-172-31-77-36 β
β β
β Date: Thu Dec 9 00:54:47 2021 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
UnixBench is a commonly used micro benchmark suite to evaluate the overall performance of computer systems. UnixBench includes a number of individual tests. Each individual test was designed to evaluate the performance of certain components on a computer system. When there are more than one CPU cores on the system, UnixBench executes the same test twice in sequence, one with a single thread and the other with multiple threads, where the number of threads equals to the number of CPU cores. These tests are executed with a fixed time, and the test result is reported as index score. The index score reflects how much work is done during the fixed time, as compared to the amount of work done during the same fixed time on a SPARCstation 20-61 5. The benchmark suite produces a single-thread index score from the single-thread execution, and a multi-thread index score from the multi-thread execution.
Download and build UnixBench:
$ git clone https://github.com/kdlucas/byte-unixbench
$ cd byte-unixbench/UnixBench/
$ sudo apt install gcc make -y
$ make
Run UnixBench and view the results:
$ ./Run
sh: 1: 3dinfo: not found
# # # # # # # ##### ###### # # #### # #
# # ## # # # # # # # ## # # # # #
# # # # # # ## ##### ##### # # # # ######
# # # # # # ## # # # # # # # # #
# # # ## # # # # # # # ## # # # #
#### # # # # # ##### ###### # # #### # #
Version 5.1.3 Based on the Byte Magazine Unix Benchmark
Multi-CPU version Version 5 revisions by Ian Smith,
Sunnyvale, CA, USA
January 13, 2011 johantheghost at yahoo period com
------------------------------------------------------------------------------
Use directories for:
* File I/O tests (named fs***) = /home/ubuntu/byte-unixbench/UnixBench/tmp
* Results = /home/ubuntu/byte-unixbench/UnixBench/results
------------------------------------------------------------------------------
1 x Dhrystone 2 using register variables 1 2 3 4 5 6 7 8 9 10
1 x Double-Precision Whetstone 1 2 3 4 5 6 7 8 9 10
1 x Execl Throughput 1 2 3
1 x File Copy 1024 bufsize 2000 maxblocks 1 2 3
1 x File Copy 256 bufsize 500 maxblocks 1 2 3
1 x File Copy 4096 bufsize 8000 maxblocks 1 2 3
1 x Pipe Throughput 1 2 3 4 5 6 7 8 9 10
1 x Pipe-based Context Switching 1 2 3 4 5 6 7 8 9 10
1 x Process Creation 1 2 3
1 x System Call Overhead 1 2 3 4 5 6 7 8 9 10
1 x Shell Scripts (1 concurrent) 1 2 3
1 x Shell Scripts (8 concurrent) 1 2 3
4 x Dhrystone 2 using register variables 1 2 3 4 5 6 7 8 9 10
4 x Double-Precision Whetstone 1 2 3 4 5 6 7 8 9 10
4 x Execl Throughput 1 2 3
4 x File Copy 1024 bufsize 2000 maxblocks 1 2 3
4 x File Copy 256 bufsize 500 maxblocks 1 2 3
4 x File Copy 4096 bufsize 8000 maxblocks 1 2 3
4 x Pipe Throughput 1 2 3 4 5 6 7 8 9 10
4 x Pipe-based Context Switching 1 2 3 4 5 6 7 8 9 10
4 x Process Creation 1 2 3
4 x System Call Overhead 1 2 3 4 5 6 7 8 9 10
4 x Shell Scripts (1 concurrent) 1 2 3
4 x Shell Scripts (8 concurrent) 1 2 3
========================================================================
BYTE UNIX Benchmarks (Version 5.1.3)
System: ip-172-31-77-36: GNU/Linux
OS: GNU/Linux -- 5.11.0-1020-aws -- #21~20.04.2-Ubuntu SMP Fri Oct 1 13:03:59 UTC 2021
Machine: x86_64 (x86_64)
Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
CPU 0: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (4800.1 bogomips)
Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
CPU 1: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (4800.1 bogomips)
Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
CPU 2: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (4800.1 bogomips)
Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
CPU 3: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (4800.1 bogomips)
Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
01:52:05 up 1:56, 1 user, load average: 0.45, 0.12, 0.04; runlevel 2021-12-08
------------------------------------------------------------------------
Benchmark Run: Thu Dec 09 2021 01:52:05 - 02:20:08
4 CPUs in system; running 1 parallel copy of tests
Dhrystone 2 using register variables 33546247.8 lps (10.0 s, 7 samples)
Double-Precision Whetstone 5392.2 MWIPS (9.9 s, 7 samples)
Execl Throughput 3168.9 lps (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks 535164.8 KBps (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks 136331.0 KBps (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks 1841354.8 KBps (30.0 s, 2 samples)
Pipe Throughput 710269.7 lps (10.0 s, 7 samples)
Pipe-based Context Switching 41683.7 lps (10.0 s, 7 samples)
Process Creation 6617.0 lps (30.0 s, 2 samples)
Shell Scripts (1 concurrent) 9473.7 lpm (60.0 s, 2 samples)
Shell Scripts (8 concurrent) 2984.1 lpm (60.0 s, 2 samples)
System Call Overhead 410413.8 lps (10.0 s, 7 samples)
System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 33546247.8 2874.6
Double-Precision Whetstone 55.0 5392.2 980.4
Execl Throughput 43.0 3168.9 737.0
File Copy 1024 bufsize 2000 maxblocks 3960.0 535164.8 1351.4
File Copy 256 bufsize 500 maxblocks 1655.0 136331.0 823.8
File Copy 4096 bufsize 8000 maxblocks 5800.0 1841354.8 3174.7
Pipe Throughput 12440.0 710269.7 571.0
Pipe-based Context Switching 4000.0 41683.7 104.2
Process Creation 126.0 6617.0 525.2
Shell Scripts (1 concurrent) 42.4 9473.7 2234.4
Shell Scripts (8 concurrent) 6.0 2984.1 4973.5
System Call Overhead 15000.0 410413.8 273.6
========
System Benchmarks Index Score 970.4
------------------------------------------------------------------------
Benchmark Run: Thu Dec 09 2021 02:20:08 - 02:48:17
4 CPUs in system; running 4 parallel copies of tests
Dhrystone 2 using register variables 131028375.7 lps (10.0 s, 7 samples)
Double-Precision Whetstone 21014.2 MWIPS (10.0 s, 7 samples)
Execl Throughput 11473.6 lps (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks 1049984.8 KBps (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks 275655.6 KBps (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks 3426155.1 KBps (30.0 s, 2 samples)
Pipe Throughput 2744140.9 lps (10.0 s, 7 samples)
Pipe-based Context Switching 576368.6 lps (10.0 s, 7 samples)
Process Creation 30486.4 lps (30.0 s, 2 samples)
Shell Scripts (1 concurrent) 24279.0 lpm (60.0 s, 2 samples)
Shell Scripts (8 concurrent) 3418.3 lpm (60.0 s, 2 samples)
System Call Overhead 1512360.2 lps (10.0 s, 7 samples)
System Benchmarks Index Values BASELINE RESULT INDEX
Dhrystone 2 using register variables 116700.0 131028375.7 11227.8
Double-Precision Whetstone 55.0 21014.2 3820.8
Execl Throughput 43.0 11473.6 2668.3
File Copy 1024 bufsize 2000 maxblocks 3960.0 1049984.8 2651.5
File Copy 256 bufsize 500 maxblocks 1655.0 275655.6 1665.6
File Copy 4096 bufsize 8000 maxblocks 5800.0 3426155.1 5907.2
Pipe Throughput 12440.0 2744140.9 2205.9
Pipe-based Context Switching 4000.0 576368.6 1440.9
Process Creation 126.0 30486.4 2419.6
Shell Scripts (1 concurrent) 42.4 24279.0 5726.2
Shell Scripts (8 concurrent) 6.0 3418.3 5697.2
System Call Overhead 15000.0 1512360.2 1008.2
========
System Benchmarks Index Score 3089.2
In Linux there is perf - the performance analysis tool for Linux. We can use perf to collect performance counters, including context switches, CPU cycles, instructions, cache misses, etc. These are hardware level performance counters, which can be enabled via certain instructions. Once enabled, they can be read by accessing specific processor registers. Due to the multi-tenant nature in public clouds, on AWS EC2 instances access to these processor registers is intentionally disabled for security reasons. It is an undocumented feature that these processor registers are made available on very few instance types running on dedicated hosts, including i3.16xlarge and the Graviton instance types (A1, M5G, M6G, etc).
In this example, we allocate two dedicated hosts, one for a1.xlarge and the other for m6g.xlarge. Then launch an an a1.xlarge and an m6g.xlarge instance running on those dedicated hosts. After that, SSH into the each of them to install perf.
$ sudo apt update
$ sudo apt install linux-oem-5.6-tools-common -y
$ sudo apt install linux-tools-5.11.0-1020-aws linux-cloud-tools-5.11.0-1020-aws -y
$ sudo apt install hwloc -y
For a comparision, we run a simple bash script (pi.sh) to calculate the value of Pi. From a computation point of view, the amount of work needed is the same on both a1.xlarge and m6g.xlarge.
#!/bin/bash
{ echo -n "scale=100;"; seq 1 2 200 | xargs -n1 -I{} echo '(16*(1/5)^{}/{}-4*(1/239)^{}/{})';} | paste -sd-+ | bc -l
On a1.xlarge we have the following result. The total time is 3.33 seconds, with 3.17 seconds in user mode and 0.21 seconds in kernel mode.
$ sudo perf stat ./pi.sh
3.141592653589793238462643383279502884197169399375105820974944592307\
8164062862089986280348253421170679
Performance counter stats for './pi.sh':
3327.01 msec task-clock # 0.998 CPUs utilized
327 context-switches # 0.098 K/sec
41 cpu-migrations # 0.012 K/sec
10540 page-faults # 0.003 M/sec
7525313978 cycles # 2.262 GHz
10900393764 instructions # 1.45 insn per cycle
<not supported> branches
133879007 branch-misses
3.334672665 seconds time elapsed
3.174069000 seconds user
0.217353000 seconds sys
$ sudo lstopo
Machine (7667MB total)
Package L#0
NUMANode L#0 (P#0 7667MB)
L2 L#0 (2048KB)
Die L#0 + L1d L#0 (32KB) + L1i L#0 (48KB) + Core L#0 + PU L#0 (P#0)
Die L#1 + L1d L#1 (32KB) + L1i L#1 (48KB) + Core L#1 + PU L#1 (P#1)
Die L#2 + L1d L#2 (32KB) + L1i L#2 (48KB) + Core L#2 + PU L#2 (P#2)
Die L#3 + L1d L#3 (32KB) + L1i L#3 (48KB) + Core L#3 + PU L#3 (P#3)
HostBridge
PCI 00:04.0 (NVMExp)
Block(Disk) "nvme0n1"
PCI 00:05.0 (Ethernet)
Net "ens5"
$ more /proc/cpuinfo
processor : 0
BogoMIPS : 166.66
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
processor : 1
BogoMIPS : 166.66
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
processor : 2
BogoMIPS : 166.66
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
processor : 3
BogoMIPS : 166.66
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
On m6g.xlarge we have the following result. The total time is 2.23 seconds, with 2.20 seconds in user mode and 0.05 seconds in kernel mode.
$ sudo perf stat ./pi.sh
3.141592653589793238462643383279502884197169399375105820974944592307\
8164062862089986280348253421170679
Performance counter stats for './pi.sh':
2221.97 msec task-clock # 0.998 CPUs utilized
319 context-switches # 0.144 K/sec
61 cpu-migrations # 0.027 K/sec
9655 page-faults # 0.004 M/sec
5511164813 cycles # 2.480 GHz
10886918709 instructions # 1.98 insn per cycle
<not supported> branches
101860518 branch-misses
2.225711878 seconds time elapsed
2.199616000 seconds user
0.046338000 seconds sys
$ sudo lstopo
Machine (15GB total)
Package L#0
NUMANode L#0 (P#0 15GB)
L3 L#0 (32MB)
Die L#0 + L2 L#0 (1024KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 (P#0)
Die L#1 + L2 L#1 (1024KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 (P#1)
Die L#2 + L2 L#2 (1024KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 (P#2)
Die L#3 + L2 L#3 (1024KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 (P#3)
HostBridge
PCI 00:04.0 (NVMExp)
Block(Disk) "nvme0n1"
PCI 00:05.0 (Ethernet)
Net "ens5"
$ more /proc/cpuinfo
processor : 0
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
processor : 1
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
processor : 2
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
processor : 3
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
The performance difference, 2.23 seconds (m6g.xlarge) vs 3.33 seconds, is significant. What contributes to the performance difference? Let's compare the above-mentioned numbers in a table:
#
# System Metrics
#
Metrics a1.xlarge m6g.xlarge difference (m6g/a1)
BogoMIPS 166.66 243.75 1.46
L3 Cache N/A 32 MB
L2 Cache 2048 KB (shared) 1024 KB (dedicated) 2.00
L1d Cache 32 KB 64 KB 2.00
L1i Cache 48 KB 64 KB 1.33
#
# Runtime Metrics
#
task-clock 3327.01 msec 2221.97 msec
CPUs utilized 0.998 0.998
context-switches 327 319
cpu-migrations 41 61
page-faults 10540 9655
cycles 7525313978 5511164813
instructions 10900393764 10886918709
branch-misses 133879007 101860518
user 3.17 seconds 2.20 seconds
sys 0.21 seconds 0.05 seconds
We notice that the CPU cores on m6g.xlarge is faster than the a1.xlarge (243.75 BogoMIPS vs 166.66 BogoMIPS). For the same work, on m6g.xlarge it requires approximately the same number of instructions. Assuming that all 4 cores are fully utilized during the computation, we can guesstimate the time needed for the computation with the following equation:
time = instruction / (cores x BogoMIPS)
With approximately the same number of instructions, we have the following theoretical estimate:
time_on_a1 / time_on_m6g = BogoMIPS_onm6g / BogoMIPS_on_a1
= 243.75 / 166.66 = 1.46
Based on our test results, we have the following actual number:
time_on_a1 / time_on_m6g = 3.33 / 2.23 = 1.49
Although the above-mentioned numbers do not exactly match, they clearly indicate that the difference in BogoMIPS is the major contributor to the performance difference observed in this example.
You can use perf top to make observations. For example, if we run our Java test program in one SSH window, and perf top --sort comm,dso in another SSH window, we will see the following:
$ sudo perf top --sort comm,dso
Samples: 414K of event 'cpu-clock:pppH', 4000 Hz, Event count (approx.): 52754843482 lost: 0/0 drop: 0/0
Overhead Command Shared Object
20.02% Thread-0 [JIT] tid 1568719
19.96% Thread-3 [JIT] tid 1568719
19.88% Thread-2 [JIT] tid 1568719
19.83% Thread-4 [JIT] tid 1568719
19.80% Thread-1 [JIT] tid 1568719
0.22% perf perf
0.07% VM Thread [kernel]
0.06% VM Thread libc-2.31.so
0.05% perf [kernel]
0.04% perf libc-2.31.so
0.02% VM Thread libjvm.so
0.01% perf libpthread-2.31.so
0.00% C1 CompilerThre libjvm.so
0.00% perf-top-UI libslang.so.2.3.2
0.00% ssm-agent-worke ssm-agent-worker
0.00% ps [kernel]
0.00% sshd [kernel]
0.00% sshd sshd
0.00% java libjvm.so
0.00% ssm-agent-worke [kernel]
0.00% Thread-4 [kernel]
0.00% Thread-1 [kernel]
0.00% perf-top-UI [kernel]
0.00% irqbalance irqbalance
0.00% Thread-3 [kernel]
0.00% java [JIT] tid 1568719
0.00% ps libc-2.31.so
0.00% perf libelf-0.176.so
0.00% sshd libc-2.31.so
0.00% C1 CompilerThre [JIT] tid 1568719
0.00% perf-top-UI libc-2.31.so
0.00% systemd libsystemd-shared-245.so
0.00% Thread-2 libjvm.so
0.00% Thread-2 [kernel]
0.00% ssm-agent-worke libc-2.31.so
0.00% systemd-resolve libsystemd-shared-245.so
0.00% sshd [vdso]
0.00% amazon-ssm-agen [kernel]
0.00% ps ld-2.31.so
0.00% ssm-agent-worke [vdso]
For a higher level overview, try: perf top --sort comm,dso
To know what happens under the hook when you run something, strace can give you a whole lot of details. Try the following:
$ strace ls
$ strace dig www.amazon.com