BQc@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddlm Z ddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddlm!Z!ddl"m#Z#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,dZ-dZ.de/fdYZ0dS(sBase class for gsutil commands. In addition to base class code, this file contains helpers that depend on base class state (such as GetAclCommandHelper, which depends on self.gsutil_bin_dir, self.bucket_storage_uri_class, etc.) In general, functions that depend on class state and that are used by multiple commands belong in this file. Functions that don't depend on class state belong in util.py, and non-shared helpers belong in individual subclasses. iN(thandler(t StorageUri(t GetoptError(tutil(tCommandException(t HelpProvider(tNameExpansionIterator(tNameExpansionIteratorQueue(tProjectIdHandler(tStorageUriBuilder(t ThreadPool(t HAVE_OAUTH2(tNO_MAX(tContainsWildcardcCs[tjd}t|_|jtjtj}|jtjd|j ||S(sCreates a logger that resembles 'print' output, but is thread safe. The logger will display all messages logged with level INFO or above. Log propagation is disabled. Returns: A logger object. sthreaded-loggings %(message)s( tloggingt getLoggertFalset propagatetsetLeveltINFOt StreamHandlert setFormattert Formattert addHandler(tlogt log_handler((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyt_ThreadedLogger6s    t command_nametcommand_name_aliasestmin_argstmax_argstsupported_sub_argst file_uri_oktprovider_uri_okturis_start_argtconfig_requiredtEOFtCommandcBs eZeZegZi de6ge6de6e e 6de 6e e 6e e6de6ee6ZeZejZgZdZeeZddZe dZdZdZdZdZd Z d Z!dd Z"d Z#d Z$dZ%dddZ&RS(itcCs |jtS(N(t command_spect COMMAND_NAME(tself((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyt_GetDefaultCommandNamesc CsF||_||_||_||_||_||_||_||_||_| |_ | |_ | |_ t |_ t |_t |_xO|jD]D} | |jks|j| d krtd|j| fqqW|j} | j|j| |_~ |js ny)tj||jt\|_|_Wn/tk rd}td|j|jfnXt|j|jtkst|j|jtkrtd|jn|jt  r|j!|j|jt"rtd|jn|jt# r5|j$|j|jt"r5td|jn|jt%rO|j&nt'|_(t)|| |_*d|_+g|_,t-j.dkr|j+d7_+|j,d g7_,nt/j0j1|jd |_2|j+|j27_+|j,|j2g7_,|jrBx?|jD]1\}}|d ks.|d kr t3|_Pq q Wnd S(sA Args: command_runner: CommandRunner (for commands built atop other commands). args: Command-line args (arg0 = actual arg, not command name ala bash). headers: Dictionary containing optional HTTP headers to pass to boto. debug: Debug level to pass in to boto connection (range 0..3). parallel_operations: Should command operations be executed in parallel? gsutil_bin_dir: Bin dir from which gsutil is running. boto_lib_dir: Lib dir where boto runs. config_file_list: Config file list returned by _GetBotoConfigFileList(). gsutil_ver: Version string of currently running gsutil command. bucket_storage_uri_class: Class to instantiate for cloud StorageUris. Settable for testing/mocking. test_method: Optional general purpose method for testing purposes. Application and semantics of this method will vary by command and test type. Implementation note: subclasses shouldn't need to define an __init__ method, and instead depend on the shared initialization that happens here. If you do define an __init__ method in a subclass you'll need to explicitly call super().__init__(). But you're encouraged not to do this, because it will make changing the __init__ interface more painful. s7"%s" command implementation is missing %s specifications%s for "%s" command.s+Wrong number of arguments for "%s" command.sN"%s" command does not support "file://" URIs. Did you mean to use a gs:// URI?s1"%s" command does not support provider-only URIs.R&tWindowsspython tpythontgsutils-rs-RN(4tcommand_runnertargst unparsed_argstheaderstdebugtparallel_operationstgsutil_bin_dirt boto_lib_dirtconfig_file_listt gsutil_vertbucket_storage_uri_classt test_methodRtexclude_symlinkstrecursion_requestedt all_versionstREQUIRED_SPEC_KEYSR'tNoneRRt_default_command_spectupdatet test_stepstgetopttSUPPORTED_SUB_ARGStsub_optsRtmsgtlentMIN_ARGStMAX_ARGSt FILE_URIS_OKt HaveFileUristURIS_START_ARGtPROVIDER_URIS_OKt_HaveProviderUristCONFIG_REQUIREDt_ConfigureNoOpAuthIfNeededRtproj_id_handlerR t suri_buildert gsutil_cmdtgsutil_exec_listtplatformtsystemtostpathtjoint gsutil_pathtTrue(R)R.R/R1R2R3R4R5R6R7R8R9tkttmptetotunused_a((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyt__init__sz               "   #        c Cs4tj||jd|jd|d|jd|jS(s  Helper to instantiate gslib.WildcardIterator. Args are same as gslib.WildcardIterator interface, but this method fills in most of the values from instance state. Args: uri_or_str: StorageUri or URI string naming wildcard objects to iterate. R8R<R1R2(twildcard_iteratorRPR8R1R2(R)t uri_or_strR<((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pytWildcardIterators   cCstd|jdS(s Abstract function in base class. Subclasses must implement this. The return value of this function will be used as the exit status of the process, so subclass commands should return an integer exit code (0 for success, a value in [1,255] for failure). s5Command %s is missing its RunCommand() implementationN(RR(R)((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyt RunCommand'sc Csmd}d}xZ|D]R}tj|d|jdtd|j}|sR|j}q|j|krdSqW|S(sTests whether the uris are all for a single provider. Returns: a StorageUri for one of the uris on success, None on failure. R2tvalidateR8N(R>tbotot storage_uriR2RR8tscheme(R)turi_argstproviderturituri_str((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pytUrisAreForSingleProvider4s   c s jdjd}j|}|sEtdjntjjrtd}|jt j dd|j t _ n4|j}|krtdnt_ t_fd}fd }jr_xtt|D]A}jj||}|jr|jd j||\s*(\S+)\s*s\1sInvalid canned ACL "%s".cs#jjt|t_dS(s9Simple exception handler to allow post-completion status.N(tTHREADED_LOGGERterrortstrRteverything_set_okay(R](R)(s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyt_SetAclExceptionHandlerlscsjj|j}jdks-tjjd|jjrl|j |j t j n|j |j t j dS(Nt setdefaclsSetting ACL on %s...(RQRtGetExpandedUriStrRtAssertionErrorRotinfotexpanded_uri_strtcannedtset_aclt object_nameRR1t set_xml_acl(tname_expansion_resultt exp_src_uri(R)tacl_arg(s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyt _SetAclFuncqs     t*NR<s'ACLs for some objects could not be set.(!R/RmRRRVRWtisfiletopentreadtretsubtcloseRRyt canned_aclsRZRrR;trangeRFRQRt names_buckettclone_replace_nameRkt_RunSingleThreadedSetAclRRPR1R2R8R<tApply( R)RiRgtacl_fileRRsRtiRkRltname_expansion_iterator((RR)s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pytSetAclCommandHelperFsJ                cCst}x|D]}x|j|D]}|jr;q#nt}|j}|jdkrd|GH|jr|j||jt|j q|j |t|j q#d|GH|jr|j ||jt|j q#|j ||jt|j q#Wq W|st dndS(NRts#Setting default object ACL on %s...sSetting ACL on %s...sNo URIs matched(RRct HasPrefixRZtGetUriRRyt set_def_aclR{R1tset_def_xml_aclRzR|R(R)RRit some_matchedRltblrRk((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyRs$       $cCsVt|jds/|jj|jd}nt|j|jdj}t|dkrrtdnt|dkrtd|jd|j fn|d}|j r|j rtd|j n|j dkr|j t |j}n|jt |j}tjjj|jjd}|jdd GHd S( srCommon logic for getting ACLs. Gets the standard ACL or the default object ACL depending on self.command_name.isNo URIs matchedisD%s matched more than one URI, which is not allowed by the %s commands-"%s" command must specify a bucket or object.t getdefaclsutf-8tindents N(R R/RQRtlistRctIterUrisRFRRRt names_objectt get_def_aclRR1tget_acltxmltdomtminidomt parseStringtto_xmltencodet toprettyxml(R)Rkturistaclt parsed_xml((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pytGetAclCommandHelpers"" $cCst|j|j}t|dkr@td|n|d}|j|t|j}tj j j |j d}|j ddGHdS(sPrint an xml subresource, e.g. logging, for a bucket/object. Args: subresource: The subresource name. uri_arg: URI for the bucket/object. Wildcards will be expanded. Raises: CommandException: if errors encountered. is5Wildcards must resolve to exactly one item for get %sisutf-8Rs N(RRcRRFRtget_subresourceRR1RRRRRR(R)t subresourceturi_argRRktxml_strR((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pytGetXmlSubresources   c Cs|jrtjjddtjjj}|dkrItd|ntjjddtjjj}|dkrtd|qn d}d}|j r|j j d||j j d|n|jr|dkrg}d}|r*x6|D]+} |s i}nt jd d || tmultiprocessingtValuetQueueRtProcesst _ApplyThreadstappendtstarttputtsyststderrtwritetreprt_EOF_NAME_EXPANSION_RESULTRXtexitcodetitemstsetattrtvaluet ExceptionR(R)tfuncRtthr_exc_handlert shared_attrst process_countt thread_counttprocst shared_varstnamet work_queuetshardtptlast_name_expansion_resultR}tfailed_process_counttvart plural_str((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyRs                    cCsCx<|D]4}|jjds7|jddkrtSqWtS(sChecks whether args_to_check contain any file URIs. Args: args_to_check: Command-line argument subset to check. Returns: True if args_to_check contains any file URIs. sfile://t:i(tlowert startswithtfindRZR(R)t args_to_checkRl((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyRJbs *cCs+x$|D]}tjd|rtSqWtS(sChecks whether args_to_check contains any provider URIs (like 'gs://'). Args: args_to_check: Command-line argument subset to check. Returns: True if args_to_check contains any provider URIs. s ^[a-z]+://$(RtmatchRZR(R)RRl((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyRMts cCsztj}tjsv|jrc|jddrMt rMtdtj ntd|jqvddl m }ndS(sASets up no-op auth handler if no boto credentials are configured.t Credentialstgs_oauth2_refresh_tokensYour gsutil is configured with OAuth2 authentication credentials. However, OAuth2 is only supported when running under Python 2.6 or later (unless additional dependencies are installed, see README for details); you are running Python %s.sYou have no storage service credentials in any of the following boto config files. Please add your credentials as described in the gsutil README file, or else re-run "gsutil config" to re-create a config file: %si(tno_op_auth_pluginN( RfRRtHasConfiguredCredentialsR6t has_optionR RRtversionRR(R)RR((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyROs   cCstj}|r4x"|D]}||jjqWn|dkrRt||} nzxtr)|j} | tkrzPn|jj| j } |j r|j j dt j|| n|jr| jrt jj| jr|j j d| qX|dkr| j|| qX|| qXW|dkrC| jnWd|dkr`| jnX|rx5|jD]$\} } | jt|| 7_qtWndS(s Perform subset of required requests across a caller specified number of parallel Python threads, which may be one, in which case the requests are processed in the current thread. Args: func: Function to call for each request. work_queue: shared queue of NameExpansionResult to process. shard: Assigned subset (shard number) for this function. num_threads: Number of Python threads to spawn to process this shard. thr_exc_handler: Exception handler for ThreadPool class. shared_vars: Dict of shared memory variables to be managed. (only relevant, and non-None, if this function is run in a separate OS process). is&process %d shard %d is handling uri %ssSkipping symbolic link %s...N(Rt provider_poolt connectionRR RZtgetRRQRuR2RoRwRVtgetpidR:t is_file_uriRWtislinkR{tAddTasktWaitCompletiontShutdownRRtgetattr(R)RRRt num_threadsRRtconnection_poolRt thread_poolR}R~RR((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyRs:            N('t__name__t __module__RRoR(R=R>tCOMMAND_NAME_ALIASESRGR RHRCRRIRLRKRZRNR'R?Rt help_specRAR*tpropertyRR`RcRdRmRRRRRRJRMROR(((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pyR%Us<    0   l   Y    z   (1t__doc__RfRBRRRRVRTRRRatxml.dom.minidomRRtboto.storage_uriRRRtgslib.exceptionRtgslib.help_providerRtgslib.name_expansionRRtgslib.project_idRtgslib.storage_uri_builderR tgslib.thread_poolR t gslib.utilR R tgslib.wildcard_iteratorR RR(RRGRHRCRIRLRKRNRtobjectR%(((s+/tmp/tmp.yUYbTOKr8o/gsutil/gslib/command.pytsJ