B Qc @ s> d Z d d l Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
Z
d d l Z d d l Z d d l
Z
d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d
l m Z d d l m Z d d l m Z d d
l m Z d d l m Z d d l m Z d d l! m" Z" d d l! m# Z# d d l! m$ Z$ d d l! m% Z% d d l! m& Z& d d l! m' Z' d d l( m) Z) d d l( m* Z* d d l( m+ Z+ d d l( m, Z, d Z- d e f d YZ. d S( s% Contains the perfdiag gsutil command.iN( t defaultdict( t Command( t COMMAND_NAME( t COMMAND_NAME_ALIASES( t CONFIG_REQUIRED( t FILE_URIS_OK( t MAX_ARGS( t MIN_ARGS( t PROVIDER_URIS_OK( t SUPPORTED_SUB_ARGS( t URIS_START_ARG( t config( t CommandException( t HELP_NAME( t HELP_NAME_ALIASES( t HELP_ONE_LINE_SUMMARY( t HELP_TEXT( t HELP_TYPE( t HelpType( t IS_LINUX( t MakeBitsHumanReadable( t MakeHumanReadable( t
Percentiles
SYNOPSIS
gsutil perfdiag [-i in.json] [-o out.json]
[-n iterations] [-c concurrency] [-s size] [-t tests] uri...
DESCRIPTION
The perfdiag command runs a suite of diagnostic tests for a given Google
Storage bucket.
The 'uri' parameter must name an existing bucket (e.g. gs://foo) to which
the user has write permission. Several test files will be uploaded to and
downloaded from this bucket. All test files will be deleted at the completion
of the diagnostic if it finishes successfully.
gsutil performance can be impacted by many factors at the client, server,
and in-between, such as: CPU speed; available memory; the access path to the
local disk; network bandwidth; contention and error rates along the path
between gsutil and Google; operating system buffering configuration; and
firewalls and other network elements. The perfdiag command is provided so
that customers can run a known measurement suite when troubleshooting
performance problems.
PROVIDING DIAGNOSTIC OUTPUT TO GOOGLE CLOUD STORAGE TEAM
If the Google Cloud Storage Team asks you to run a performance diagnostic
please use the following command, and email the output file (output.json)
to gs-team@google.com:
gsutil perfdiag -o output.json gs://your-bucket
OPTIONS
-n Sets the number of iterations performed when downloading and
uploading files during latency and throughput tests. Defaults to
5.
-c Sets the level of concurrency to use while running throughput
experiments. The default value of 1 will only run a single read
or write operation concurrently.
-s Sets the size (in bytes) of the test file used to perform read
and write throughput tests. The default is 1 MiB.
-t Sets the list of diagnostic tests to perform. The default is to
run all diagnostic tests. Must be a comma-separated list
containing one or more of the following:
lat: Runs N iterations (set with -n) of writing the file,
retrieving its metadata, reading the file, and deleting
the file. Records the latency of each operation.
rthru: Runs N (set with -n) read operations, with at most C
(set with -c) reads outstanding at any given time.
wthru: Runs N (set with -n) write operations, with at most C
(set with -c) writes outstanding at any given time.
-o Writes the results of the diagnostic to an output file. The output
is a JSON file containing system information and performance
diagnostic results. The file can be read and reported later using
the -i option.
-i Reads the JSON output file created using the -o command and prints
a formatted description of the results.
NOTE
The perfdiag command collects system information. It collects your IP address,
executes DNS queries to Google servers and collects the results, and collects
network statistics information from the output of netstat -s. None of this
information will be sent to Google unless you choose to send it.
t PerfDiagCommandc B s` e Z d Z i d e 6d d d d g e 6d e 6d e 6d e 6e e 6e e
6d e 6e e
6Z i d e 6g e 6e j e 6d e 6e e 6Z d! Z d" Z e j j j j Z e d Z e e e d Z! e e e d Z" d Z# d Z$ e% j& d Z' d Z( d Z) d Z* d Z+ d Z, d Z- d Z. d Z/ d Z0 d Z1 d Z2 RS(# s* Implementation of gsutil perfdiag command.t perfdiagt diagt
diagnostict perft performancei i s n:c:s:t:i:o:s Run performance diagnostici i i t rthrut wthrut latc C s< | j r d G| GHn t t j d } d } g } xt | sN | | k r7xf t | | k r | t | | k r d d j | GHt j | d | d | } | j | qQ W| } g } xn | D]f } | j }
|
d
k r | j | q | r&|
r&t d |
d j | f q | d 7} q Wq6 Wd
S( s Executes a command n times with a window size of w.
Up to w instances of the command will be executed and left outstanding at a
time until n instances of the command have completed.
Args:
cmd: List containing the command to execute.
n: Number of times the command will be executed.
w: Window size of outstanding commands being executed.
raise_on_error: See _Exec.
Raises:
Exception: If raise_on_error is set to True and any process exits with a
non-zero return code.
s Running command:t wi s Starting concurrent command: %st t stdoutt stderrs8 Received non-zero return code (%d) from subprocess '%s'.i N( t debugt opent ost devnullt lent joint
subprocesst Popent appendt pollt NoneR ( t selft cmdt nR t raise_on_errort devnull_ft num_finishedt runningt pt prev_runningt retcode( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt
_WindowedExec s( +
c C s | j r d G| GHn | r$ t j n d } t j | d t j d | } | j \ } } | r | j r t d | j d j | f n | r | S| j S( s Executes a command in a subprocess.
Args:
cmd: List containing the command to execute.
raise_on_error: Whether or not to raise an exception when a process exits
with a non-zero return code.
return_output: If set to True, the return value of the function is the
stdout of the process.
mute_stderr: If set to True, the stderr of the process is not printed to
the console.
Returns:
The return code of the process or the stdout if return_output is set.
Raises:
Exception: If raise_on_error is set to True and any process exits with a
non-zero return code.
s Running command:R" R# s8 Received non-zero return code (%d) from subprocess '%s'.R! N( R$ R* t PIPER. R+ t communicatet
returncodeR R) ( R/ R0 R2 t
return_outputt mute_stderrR# R6 t
stdoutdatat
stderrdata( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _Exec s c C s, | j | } | j | d | d | d | S( sX Executes a gsutil command in a subprocess.
Args:
cmd: A list containing the arguments to the gsutil program, e.g. ['ls',
'gs://foo'].
raise_on_error: see _Exec.
return_output: see _Exec.
mute_stderr: see _Exec.
Returns:
The return code of the process or the stdout if return_output is set.
R2 R= R> ( t gsutil_exec_listRA ( R/ R0 R2 R= R> ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _GsUtil s
c s i _ g _ i _ i _ f d } x- j D]" } | | } j j | q= W| j _ t j t
j j j _
d S( s? Performs setup operations needed before diagnostics can be run.c s t j d d d d d t \ } } | j | q Xq W| j d | j g d t d t d S( sD Performs operations to clean things up after performing diagnostics.t rmR2 R> N( RZ R] R& t removet OSErrorRC Rb RL t True( R/ RV ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _TearDown6 s
c c sH | Gd GHt j } d Vt j } | | j | | | Gd GHd S( s A context manager that measures time.
A context manager that prints a status message before and after executing
the inner command and times how long the inner command takes. Keeps track of
the timing, aggregated by the given key.
Args:
key: The key to insert the timing value into a dictionary bucket.
bucket: A dictionary to place the timing value in.
Yields:
For the context manager.
s starting...Ns done.( t timeR, ( R/ t keyt buckett t0t t1( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _TimeA s c
C s t t | j d } | j \ } } } } | d j r5 | j | q5 q5 WWd QXi } t d d } x | D] } | j d } | \ } } } }
} } } }
} } } | | k r t | | } t | | } t |
}
t |
}
t | } t | } |
|
| | | | f | | s s segments receiveds packets receivedc s s$ | ] } | t j k r | Vq d S( N( R R ( R R ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys s s segments send outs packets sents
segments sentc s s$ | ] } | t j k r | Vq d S( N( R R ( R R ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys s t tcp_retransmitt tcp_receivedt tcp_sentN( RA Rg RL t stript lowerR. R R) R t
ValueErrort TypeError( R/ t netstat_outputt found_tcpR R R R t result( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _GetTcpStats s>
c C s i } t j t j | d [^.]+)\.t googservt googserv_routet googserv_ipst googserv_hostnamess -type=TXTs o-o.myaddr.google.com.s text\s+=\s+"(?P[\.\d]+)"t dnsips
dns_o-o_ipt cpu_countt load_avgs
/proc/meminfoR t MemTotalR c s s$ | ] } | t j k r | Vq d S( N( R R ( R R ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys P s i t MemFreec s s$ | ] } | t j k r | Vq d S( N( R R ( R R ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys S s t Buffersc s s$ | ] } | t j k r | Vq d S( N( R R ( R R ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys V s t Cachedc s s$ | ] } | t j k r | Vq d S( N( R R ( R R ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys Y s Nt mem_totalt mem_freet mem_bufferst
mem_cachedt meminfot
gsutil_configs
t sysinfo(' t sockett
gethostbynamet gethostnameRJ t
gettempdirRi t strftimet gmtimet GOOGLE_API_HOSTRA Rg t ret searcht groupR. t gethostbyname_ext
gethostbyaddrR, t multiprocessingR t NotImplementedErrorRr R& t
getloadavgt AttributeErrorRf R% t
startswithR R) t IOErrorR t dirR t getattrt isuppert
isinstancet
basestringRY ( R/ R R0 t nslookup_cname_outputt mt hostnamet aliaslistt
ipaddrlistt googserv_ipt nslookup_txt_outputR R R R RW R t attrt
attr_value( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _CollectSysInfo sx
c s t | } t t | | t j t f d | D | } t | j d Gd Gd d j d Gd Gd | d j d Gd Gd t | d d j d Gd Gd t | d
d j d Gd GHd S( sA Prints out mean, standard deviation, median, and 90th percentile.c 3 s | ] } | d Vq d S( i N( ( R t x( t mean( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pys r s i R s %.1fi i i g ?i g?N( R( t floatt sumt matht sqrtR^ t rjustR ( R/ t trialsR1 t stdev( ( R s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt
_DisplayStatsn s )"c $
C sa Hd d GHd j d GHd d GHd | j k rHd d GHd j d GHd d GHd GHd GHxft | j d D]N} t | j d | } | j d \ } } t | } | d
k r d j d Gd
Gt | j d Gd
G| j | n | d k r5d j d Gd
Gt | j d Gd
G| j | n | d k ryd j d Gd
Gt | j d Gd
G| j | n | d k ro d j d Gd
Gt | j d Gd
G| j | qo qo Wn d | j k rEHd d GHd j d GHd d GH| j d } d t | d | d t | d f GHd t | d d GHn d | j k rHd d GHd j d GHd d GH| j d } d t | d | d t | d f GHd t | d d GHn d! | j k rHd d GHd" j d GHd d GH| j d! } d# | d$ GHd% | d& GHd' | j d( GHd) | k r| d) } dV } y t
j | d* } Wn t k
rrn X| rt
j | }
t j j |
} d+ | j d, GHqn d- | d. GHd/ d0 j | d1 GHd2 d0 j | d3 GHd4 | d5 GHd6 | d7 GHd8 | d9 GHyJ d: t | d; d< GHd= t | d; d> | d; d? | d; d@ GHWn t k
ron X| dA } | dB }
xO dX D]G } y- | dF | |
dF | } dG | | f GHWqt k
rqXqWdH | k rdI | k rdJ G| dH } | dI } d
GdK j dL Gx- dM dN dO dP dQ dR g D] } | j d Gq0WHx t | D] } | | } | | } | \ } } } } } } | \ } } } } } }! d
G| j dL G| | | | | | | | | | |! | g }" x! |" D] } t | j d GqWHqUWqn | j r\t | j dS }# t j | j |# dT dU WdV QXHdW | j GHn HdV S(Y s/ Displays results collected from diagnostic run.t =iN s DIAGNOSTIC RESULTSRo t -t LatencysO Operation Size Trials Mean (ms) Std Dev (ms) Median (ms) 90th % (ms)sO ========= ========= ====== ========= ============ =========== ===========R t METADATAt Metadatai R t DOWNLOADt Downloadt UPLOADt Uploadt DELETEt DeleteR s Write Throughputs: Copied a %s file %d times for a total transfer size of %s.RT R R s Write throughput: %s/s.R i R s Read ThroughputR s Read throughput: %s/s.R s System Informations IP Address:
%sR s Temporary Directory:
%sR s Bucket URI:
%sR_ R s %a, %d %b %Y %H:%M:%S +0000s Measurement time:
%ss %Y-%m-%d %I-%M-%S %p %Zs Google Server:
%sR s! Google Server IP Addresses:
%ss
R s Google Server Hostnames:
%sR s# Google DNS thinks your IP is:
%ss
dns_o-o_ips CPU Count:
%sR s CPU Load Average:
%sR s Total Memory:
%sR R s Free Memory:
%sR R R t netstat_endt
netstat_startt sentt receivedt
retransmits tcp_%ss! TCP segments %s during test:
%dt disk_counters_endt disk_counters_starts Disk Counter Deltas:
t diski R R R R R R R t indenti Ns Output file written to '%s'.( s sentR R ( t centerRY t sortedR R R R R R R. Ri t strptimeR t calendart timegmt datetimet
fromtimestampR R) R R^ t output_fileR% t jsont dump($ R/ Rj R t opt numbytest
write_thrut read_thrut infot ts_stringt timetuplet localtimet localdtt
netstat_aftert netstat_beforet tcp_typet deltat
disk_aftert disk_beforet colnamet disknamet beforet aftert reads1t writes1t rbytes1t wbytes1t rtime1t wtime1t reads2t writes2t rbytes2t wbytes2t rtime2t wtime2t deltasRW ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _DisplayResultsz s
c C sS y/ t | } | d k r* t | n | SWn t k
rN t | n Xd S( s^ Tries to convert val argument to a positive integer.
Args:
val: The value (as a string) to convert to a positive integer.
msg: The error message to place in the CommandException on an error.
Returns:
A valid positive integer.
Raises:
CommandException: If the supplied value is not a valid positive integer.
i N( R R R ( R/ t valt msg( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt _ParsePositiveInteger
s
c C s| d | _ d | _ d | _ | j | _ d | _ d | _ | j r
x| j D]\ } } | d k r| | j | d | _ n | d k r | j | d | _ n | d k r | j | d | _ n | d
k r4g | _ xX | j
j d D]> } | j | j k r t
d | n | j j | q Wn | d
k rXt j j | | _ n | d k rL t j j | | _ t j j | j st
d | n y= t | j d % } t j | | _ d | j GHWd QXWn! t k
rt
d | n Xd SqL Wn | j s%t
d n | j j | j d | _ | j j sft
d | j n | j j | _ d S( s& Parses arguments for perfdiag command.i i i s -ns, The -n parameter must be a positive integer.s -cs, The -c parameter must be a positive integer.s -ss, The -s parameter must be a positive integer.s -tt ,s8 List of test names (-t) contains invalid test name '%s'.s -os -is Invalid input file (-i): '%s'.R s Read input file: '%s'.Ns' Could not decode input file (-i): '%s'.s1 Wrong number of arguments for "perfdiag" command.i sO The perfdiag command requires a URI that specifies a bucket.
"%s" is not valid.( Rt R R\ t ALL_DIAG_TESTSt
diag_testsR. R t
input_filet sub_optsR1 R R R R R, R& R` t abspatht isfileR% R t loadRY R t argst suri_buildert
StorageUriR_ t names_buckett
get_bucketRk ( R/ t ot at test_nameRW ( ( s5 /tmp/tmp.yUYbTOKr8o/gsutil/gslib/commands/perfdiag.pyt
_ParseArgs" sZ
c C sx | j | j r! | j d Sd | j GHd | j GHd | j GHd t | j GHd d j | j GHz | j
| j | j | j
d d sN J