Perf team scripts - ntapse/scripts GitHub Wiki

This goal of this page is to track the scripting efforts of the performance team in an effort to mitigate duplicate efforts. To keep it simple, each participant gets their own section that they can edit independently.

See also: EPS Perf Scripts

Table of Contents

Kendrick Gay

  • flush.sh
    • Path: /u/kendricg/scripts/flush.sh
    • Keywords: lush,flush
    • About: Just a simple front-end for lush for those more familiar with standard shell binaries usage.
    • Example:
[root@vm01 03052014]# flush -p 152907 -c "wafl_susp -w" -i 1-3 -f "grep cp_from_cp[[:space:]]" -h contsite12
[Executing /opt/latxlib/lush.py perfid=152907 sectname=wafl_susp -w iternum=1-3 contsite12 output | grep cp_from_cp[[:space:]]...

[ Iteration #1 ]

   [ Host contsite12 ]
        cp_from_cp               =         17

[ Iteration #2 ]

   [ Host contsite12 ]
        cp_from_cp               =         24

[ Iteration #3 ]

   [ Host contsite12 ]
        cp_from_cp               =         15

Josh Dorion

  • cifspercentages.sh
    • Path: /u/djosh/scripts/cifspercentages.sh
    • Keywords: cifs
    • About: This script prints the CIFS latency and op size histograms and their percentages (useful for understanding workloads) by URL.
    • Example:
[root@machine]# ./cifspercentages.sh [http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521431/name/stats_stop_-I_perfstat_cifs.txt http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521431/name/stats_stop_-I_perfstat_cifs.txt]<br/>####### read_latency_hist #######<br/>Total ops: 75504<br/>cifs:cifs:cifs_read_latency_hist.0 - <1ms:28736 = 38.058900&nbsp;%<br/>etc
  • csf.sh
    • Path: /u/djosh/scripts/csf.sh
    • Keywords: stingray, case
    • About: This script lists all files in a case directory, by full path, sorted by last modified.
    • Example:
[root@machine]# ./csf.sh 2004895353 /x/eng/cs-data/2004895353/<br/>/x/eng/cs-data/2004895353/2004895353.2004895353_perfstat_20140311_1137.out<br/>/x/eng/cs-data/2004895353/2004895353_perfstat_20140311_1137.out
  • dupeperfstatfix.sh
    • Path: /u/djosh/scripts/dupeperfstatfix.sh
    • Keywords: perfstat, stingray
    • About: This script modifies a perfstat to have a line break at the beginning (useful for uploading duplicate data to LatX).
    • Example:
[root@machine]# ./dupeperfstatfix.sh perfstat_complete.out
  • lunpercentages.sh
    • Path: /u/djosh/scripts/lunpercentages.sh
    • Keywords: iscsi, fcp
    • About: This script prints the iSCSI/FCP latency and op size histograms and their percentages (useful for understanding workloads) by URL.
    • Example:
[root@machine]# ./lunpercentages.sh [http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521440/name/stats_stop_-I_perfstat_fcp.txt http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521440/name/stats_stop_-I_perfstat_fcp.txt]<br/>####### read_latency_hist #######<br/>Total ops: 75504<br/>fcp:fcp:fcp_latency_hist.0 - <1ms:28736 = 38.058900&nbsp;%<br/>etc
  • misalignment.py
    • Path: /u/djosh/scripts/misalignment.py -p 233078
    • Keywords: misalignment
    • About: This script isolates the iteration with the highest partial writes percentage and prints both the iteration and partial writes percentages out.
    • Example:
[root@machine]# misalignment.py 4 iterations found<br/>AEDR-ST02.aeres.local - loading sections...<br/>There are 227194 total write operations during the iteration.<br/>The highest partial write value is 27.4% in iteration 4
  • nfsmisalignment.sh
    • Path: /u/djosh/scripts/nfsmisalignment.sh
    • Keywords: nfs, misalignment, vmware
    • About: This script prints a list of misaligned VMDKs from nfsstat -d, but removes delta files and duplicate listings.
    • Example:
[root@machine]# ./nfsmisalignment.sh [http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521541/name/nfsstat_-d.txt http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521541/name/nfsstat_-d.txt]<br/>Filename=perat2nfs2_vol17_p/q_01/PERAT2NT8PRINT1/PERAT2NT8PRINT1-flat.vmdk<br/>Filename=perat2nfs2_vol18_p/q_01/PERAT2NTTS3C08/PERAT2NTTS3C08-flat.vmdk<br/>Filename=perat2nfs2_vol18_p/q_01/PERAT2W8SSOP/PERAT2W8SSOP-flat.vmdk<br/>etc
  • nfspercentages.sh
    • Path: /u/djosh/scripts/nfspercentages.sh
    • Keywords: nfs
    • About: This script prints the NFS latency and op size histograms and their percentages (useful for understanding workloads) by URL.
    • Example:
[root@machine]# ./nfspercentages.sh [http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521458/name/stats_stop_-I_perfstat_nfs.txt http://api.latx.corp.netapp.com/ws/outputtxt/dataid/754521458/name/stats_stop_-I_perfstat_nfs.txt]<br/>####### read_latency_hist #######<br/>Total ops: 75504<br/>nfsv3:nfs:nfsv3_read_latency_hist.0 - <1ms:28736 = 38.058900&nbsp;%<br/>etc

Ken Mencher

Dan Small

  • cpdumper.pl
    • Path: /u/smalld/scripts/cpdumper.pl
    • Keywords: cpdump
    • About:This script takes the cpdump file from pre-8.1 releases and generates a CSV file from its contents, which is way easier than reading a giant text file.
  • priority.py
    • Path: /u/smalld/scripts/priority.py
    • Keywords: DFM, PA, priority
    • About: This script queries the DFM server, retrieves a list of controllers and then passes commands through the DFM server to see if priority is enabled. If it is, a command is relayed to disable it. By passing the command through DFM the credentials stored in DFM are used.
  • disk_ut.py
    • Path: /u/smalld/scripts/disks_ut.py
    • Keywords: CM disk, counter manager disk, raw, M/D/1, Balance
    • About: This script can be ran against a LatxID or an input file conntaining one or more samples of 'stat disk' output. this could be the console output of'stats start disk' and 'stats stop' or the output of 'stats show -i xxx -n x disk'. If multple samples are present it processes them as iterations. It calculates several statistics, like the mean of the various counters. The primary purpose was to compare ONTAP's disk busy to the Balance Utilizaiton. Unlike the disk related Flexible Reports this script does not exclude parity disks. Output is CSV.
  • ifnet.py
    • Path: /u/smalld/scripts/ifnet.py
    • Keywords: ifnet, 559910, 583491
    • About: Mines the "ip packets received" and "ip packets with bad headers discarded" counter values from 10Gb interfaces. The values printed after iteration one are deltas.
  • disk.py
    • Path: /u/smalld/scripts/latxlib/disk.py
    • Keywords: CM Disk
    • About: A function that takes a "getPerfstat" object, filername and iteration number. Returns a ODWN of aggregate over plex over counter.
The counters are:

disk_busy_mean, disk_busy_cv,total_transfers_sum, total_transfers_mean, total_transfers_cv,reads_sum, reads_mean, reads_cv, read_blocks, read_xput, read_chain_mean, read_chain_cv, read_avg_size,writes_sum, writes_mean, writes_cv, write_blocks, write_xput, write_chain_mean, write_chain_cv, write_avg_size,cp_reads_sum, cp_reads_mean, cp_reads_cv, cp_read_blocks, cp_read_xput, cp_read_chain_mean, cp_read_chain_cv, cp_read_avg_size,guarenteed_reads_sum, guarenteed_reads_mean, guarenteed_reads_cv, guarenteed_read_blocks, guarenteed_read_xput, guarenteed_read_chain_mean, guarenteed_read_chain_cv, guarenteed_read_avg_size,guarenteed_writes_sum, guarenteed_writes_mean, guarenteed_writes_cv, guarenteed_write_blocks, guarenteed_write_xput, guarenteed_write_chain_mean, guarenteed_write_chain_cv, guarenteed_write_avg_size,io_pending_sum, io_pending_mean, io_pending_cv,io_queued_sum, io_queued_mean, io_queued_cv Note, xput is in KB/s and parity drives are not included. This function is called by the a_disk_basic.py and a_disk_detailed.py Flexible Reports.

  • workload.py
    • Path: /u/smalld/scripts/latxfr/workload.py
    • Keywords: Flexible Report, FR, workload, workload detailed
    • About: Although a FR, an alternative option to print directly to a CSV exists.
  • cmstats_target.py
    • Path: /u/smalld/scripts/latxfr/cmstats_target.py
    • Keywords: section views, CM, Counter Manager
    • About: This is the target script called by all CM FR's under 'Section Views'. This script is passed three arguments: latxID, hostname and CM object.

Flexible ReportsThe output is XML, but a wrapper could probably be writtin to print to the console. Additionally, they provide a pretty good reference.
  • a_disk_basic.py
    • Path: /u/smalld/scripts/latxfr/a_disk_basic.py
    • Keywords: Flexible Report, FR, Disk
    • About: Uses the disk.py function and returns a flexible report (xml) of basic disk statistcs such as the mean of the avalble counters.
  • a_disk_detailed.py
    • Path: /u/smalld/scripts/latxfr/a_disk_basic.py
    • Keywords: Flexible Report, FR, Disk
    • About: Just like a_disk_basic, howver this script displays all of the statistics returned by the disk.py function.
  • cifs_lat_rep.py
    • Path: /u/smalld/scripts/latxfr/cifs_lat_rep.py
    • Keywords: cifs, latency report, cdot, cmode, c-mode, CM, countermanager
    • About: FR that is cDOT aware, mimics the CIFS latency report seen in native Latx
  • cp.py
    • Path: /u/smalld/scripts/latxfr/cp.py
    • Keywords: cp, prefetch, cpstats, Flexible Report
    • About: Grabs CP related counters from 'wafl_susp -w', 'wafl cpstats', CM readahed and CM wafl. Output is FR.
  • vol_lat_rep.py
    • Path: /u/smalld/scripts/latxfr/vol_lat_rep.py
    • Keywords: vol, volume, latency report, cdot, cmode, c-mode, CM, countermanager
    • About: FR that is cDOT aware, mimics the volume latency report seen in native Latx
  • nfsv3_op_counts.py
    • Path: /u/smalld/scripts/latxfr/nfsv3_op_counts.py
    • Keywords: nfsv3,latency report, cdot, cmode, c-mode, CM, countermanager
    • About: FR that is cDOT aware, mimics the nfsv3 latency report seen in native latx
  • lun_lat_rep.py
    • Path: /u/smalld/scripts/latxfr/vol_lat_rep.py
    • Keywords: lun, latency report, cdot, cmode, c-mode, CM, countermanager
    • About: FR that is cDOT aware, mimics the lun latency report seen in native Latx
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️