ž ¨ÿfu[c@sfdgZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z m Z dZ dZdZejƒZdd„Zd d „ZGd d „d eƒZefed d„ZGdd„deƒZGdd„deƒZeZGdd„deƒZGdd„deƒZGdd„deƒZGdd„deƒZdS(uPooliN(uProcessu cpu_countu TimeoutError(uFinalizeudebugiicCstt|ŒƒS(N(ulistumap(uargs((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyumapstar'sumapstarcCsttj|d|dƒƒS(Nii(ulistu itertoolsustarmap(uargs((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu starmapstar*su starmapstarcsD|EeZdZdZ‡fdd†Zdd„Zdd„Z‡S(uMaybeEncodingErroruVWraps possible unpickleable errors, so they can be safely sent through the socket.csAt|ƒ|_t|ƒ|_tt|ƒj|j|jƒdS(N(urepruexcuvalueusuperuMaybeEncodingErroru__init__(uselfuexcuvalue(u __class__(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__init__5suMaybeEncodingError.__init__cCsd|j|jfS(Nu(Error sending result: '%s'. Reason: '%s'(uvalueuexc(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__str__:s uMaybeEncodingError.__str__cCsdt|ƒS(Nu(ustr(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__repr__>suMaybeEncodingError.__repr__(u__name__u __module__u __qualname__u__doc__u__init__u__str__u__repr__(u __locals__((u __class__u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuMaybeEncodingError1s uMaybeEncodingErrorc+Csö|dks0t|ƒtkr*|dks0t‚|j}|j}t|dƒrn|jjƒ|j jƒn|dk r‡||Œnd}xT|dks®|rã||krãy |ƒ}Wn#t t fk ràt dƒPYnX|dkrût dƒPn|\} } } } } yd| | | Žf}Wn1tk r\}zd |f}WYdd}~XnXy|| | |fƒWn_tk rÕ}z?t||dƒ}t d|ƒ|| | d |ffƒWYdd}~XnX|d7}qWt d|ƒdS( Niu_writeru)worker got EOFError or IOError -- exitinguworker got sentinel -- exitingiu0Possible encoding error while sending result: %suworker exiting after %d tasksTF(uNoneutypeuintuAssertionErroruputugetuhasattru_writerucloseu_readeruEOFErroruIOErrorudebuguTrueu ExceptionuFalseuMaybeEncodingError(uinqueueuoutqueueu initializeruinitargsumaxtasksuputugetu completedutaskujobuiufuncuargsukwdsuresultueuwrapped((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuworkerBs@0     !    ,uworkercBsÁ|EeZdZdZeZd7d7fd7dd„Zdd„Zdd„Zdd „Z d d „Z fid d „Z d7dd„Z d7dd„Z d7d7d7dd„Zddd„Zddd„Zfid7d7dd„Zd7d7d7dd„Zd7d7d7dd„Zedd „ƒZed!d"„ƒZed#d$„ƒZed%d&„ƒZd'd(„Zd)d*„Zd+d,„Zd-d.„Zed/d0„ƒZed1d2„ƒZd3d4„Z d5d6„Z!d7S(8uPooluS Class which supports an async version of applying functions to arguments. cCs-|jƒtjƒ|_i|_t|_||_||_||_ |dkr}y t ƒ}Wq}t k ryd}Yq}Xn|dkr˜t dƒ‚n|dk rÀt|ƒ rÀtdƒ‚n||_g|_|jƒtjdtjd|fƒ|_d|j_t|j_|jjƒtjdtjd|j|j|j|j|jfƒ|_d|j_t|j_|jjƒtjdtjd|j|j |jfƒ|_!d|j!_t|j!_|j!jƒt"||j#d|j|j$|j|j|j|j|j!|jfddƒ|_%dS( Niu&Number of processes must be at least 1uinitializer must be a callableutargetuargsu exitpriorityiT(&u _setup_queuesuqueueuQueueu _taskqueueu_cacheuRUNu_stateu_maxtasksperchildu _initializeru _initargsuNoneu cpu_countuNotImplementedErroru ValueErrorucallableu TypeErroru _processesu_poolu_repopulate_poolu threadinguThreaduPoolu_handle_workersu_worker_handleruTrueudaemonustartu _handle_tasksu _quick_putu _outqueueu _task_handleru_handle_resultsu _quick_getu_result_handleruFinalizeu_terminate_poolu_inqueueu _terminate(uselfu processesu initializeruinitargsumaxtasksperchild((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__init__rsX                              u Pool.__init__cCswd}xjttt|jƒƒƒD]M}|j|}|jdk r"td|ƒ|jƒd}|j|=q"q"W|S(uCleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. ucleaning up worker %dFNT( uFalseureversedurangeulenu_pooluexitcodeuNoneudebugujoinuTrue(uselfucleaneduiuworker((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_join_exited_workers­s"  uPool._join_exited_workersc Cs¦xŸt|jt|jƒƒD]}|jdtd|j|j|j|j |j fƒ}|jj |ƒ|j j ddƒ|_ d|_|jƒtdƒqWdS(u€Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. utargetuargsuProcessu PoolWorkeru added workerNT(urangeu _processesulenu_pooluProcessuworkeru_inqueueu _outqueueu _initializeru _initargsu_maxtasksperchilduappendunameureplaceuTrueudaemonustartudebug(uselfuiuw((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_repopulate_pool¼s#   uPool._repopulate_poolcCs|jƒr|jƒndS(uEClean up any exited workers and start replacements for them. N(u_join_exited_workersu_repopulate_pool(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_maintain_poolÌs uPool._maintain_poolcCsPddlm}|ƒ|_|ƒ|_|jjj|_|jjj|_ dS(Ni(u SimpleQueue( uqueuesu SimpleQueueu_inqueueu _outqueueu_writerusendu _quick_putu_readerurecvu _quick_get(uselfu SimpleQueue((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _setup_queuesÒs   uPool._setup_queuescCs.|jtkst‚|j|||ƒjƒS(u6 Equivalent of `func(*args, **kwds)`. (u_stateuRUNuAssertionErroru apply_asyncuget(uselfufuncuargsukwds((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuapplyÙsu Pool.applycCs|j||t|ƒjƒS(ux Apply `func` to each element in `iterable`, collecting the results in a list that is returned. (u _map_asyncumapstaruget(uselfufuncuiterableu chunksize((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyumapàsuPool.mapcCs|j||t|ƒjƒS(uÌ Like `map()` method but the elements of the `iterable` are expected to be iterables as well and will be unpacked as arguments. Hence `func` and (a, b) becomes func(a, b). (u _map_asyncu starmapstaruget(uselfufuncuiterableu chunksize((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyustarmapçsu Pool.starmapcCs|j||t|||ƒS(u= Asynchronous version of `starmap()` method. (u _map_asyncu starmapstar(uselfufuncuiterableu chunksizeucallbackuerror_callback((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu starmap_asyncïsuPool.starmap_asyncicsï|jtkrtdƒ‚n|dkrrt|jƒ‰|jj‡‡fdd†t|ƒDƒˆjfƒˆS|dks„t ‚t j ˆ||ƒ}t|jƒ‰|jj‡fdd†t|ƒDƒˆjfƒdd„ˆDƒSdS(uP Equivalent of `map()` -- can be MUCH slower than `Pool.map()`. uPool not runningic3s0|]&\}}ˆj|ˆ|fifVqdS(N(u_job(u.0uiux(ufuncuresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu ÿsuPool.imap..c3s0|]&\}}ˆj|t|fifVqdS(N(u_jobumapstar(u.0uiux(uresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu scss"|]}|D] }|Vq qdS(N((u.0uchunkuitem((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu sN( u_stateuRUNu ValueErroru IMapIteratoru_cacheu _taskqueueuputu enumerateu _set_lengthuAssertionErroruPoolu _get_tasks(uselfufuncuiterableu chunksizeu task_batches((ufuncuresultu9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuimap÷s u Pool.imapcsï|jtkrtdƒ‚n|dkrrt|jƒ‰|jj‡‡fdd†t|ƒDƒˆjfƒˆS|dks„t ‚t j ˆ||ƒ}t|jƒ‰|jj‡fdd†t|ƒDƒˆjfƒdd„ˆDƒSdS(uL Like `imap()` method but ordering of results is arbitrary. uPool not runningic3s0|]&\}}ˆj|ˆ|fifVqdS(N(u_job(u.0uiux(ufuncuresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu su&Pool.imap_unordered..c3s0|]&\}}ˆj|t|fifVqdS(N(u_jobumapstar(u.0uiux(uresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu scss"|]}|D] }|Vq qdS(N((u.0uchunkuitem((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu sN( u_stateuRUNu ValueErroruIMapUnorderedIteratoru_cacheu _taskqueueuputu enumerateu _set_lengthuAssertionErroruPoolu _get_tasks(uselfufuncuiterableu chunksizeu task_batches((ufuncuresultu9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuimap_unordered s uPool.imap_unorderedcCsb|jtkrtdƒ‚nt|j||ƒ}|jj|jd|||fgdfƒ|S(u; Asynchronous version of `apply()` method. uPool not runningN( u_stateuRUNu ValueErroru ApplyResultu_cacheu _taskqueueuputu_jobuNone(uselfufuncuargsukwdsucallbackuerror_callbackuresult((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu apply_asyncs +uPool.apply_asynccCs|j||t|||ƒS(u9 Asynchronous version of `map()` method. (u _map_asyncumapstar(uselfufuncuiterableu chunksizeucallbackuerror_callback((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu map_async(suPool.map_asyncc s|jtkrtdƒ‚nt|dƒs<t|ƒ}n|d kr†tt|ƒt|jƒdƒ\}}|r†|d7}q†nt|ƒdkr¡d}nt j |||ƒ}t |j |t|ƒ|d|ƒ‰|j j‡‡fdd†t|ƒDƒd fƒˆS( uY Helper function to implement map, starmap and their async counterparts. uPool not runningu__len__iiiuerror_callbackc3s0|]&\}}ˆj|ˆ|fifVqdS(N(u_job(u.0uiux(umapperuresult(u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu Dsu"Pool._map_async..N(u_stateuRUNu ValueErroruhasattrulistuNoneudivmodulenu_pooluPoolu _get_tasksu MapResultu_cacheu _taskqueueuputu enumerate( uselfufuncuiterableumapperu chunksizeucallbackuerror_callbackuextrau task_batches((umapperuresultu9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _map_async0s  (  uPool._map_asynccCsotjƒ}xB|jtks6|jrP|jtkrP|jƒtjdƒqW|j j dƒt dƒdS(Ngš™™™™™¹?uworker handler exiting( u threadingucurrent_threadu_stateuRUNu_cacheu TERMINATEu_maintain_poolutimeusleepu _taskqueueuputuNoneudebug(upooluthread((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_handle_workersHs  * uPool._handle_workerscCs„tjƒ}xt|jdƒD]ê\}}d }xÔt|ƒD]¢\}} |jratdƒPny|| ƒWq>tk rß} zN| dd…\} } y|| j | d | fƒWnt k rÌYnXWYdd} ~ Xq>Xq>W|rtdƒ||dƒqqPqWtdƒy@tdƒ|j dƒtdƒx|D]} |dƒqBWWnt k rutdƒYnXtd ƒdS( Niu'task handler found thread._state != RUNiudoing set_length()utask handler got sentinelu/task handler sending sentinel to result handleru(task handler sending sentinel to workersu/task handler got IOError when sending sentinelsutask handler exitingiÿÿÿÿF(u threadingucurrent_threaduiterugetuNoneu enumerateu_stateudebugu Exceptionu_setuFalseuKeyErroruputuIOError(u taskqueueuputuoutqueueupoolucacheuthreadutasksequ set_lengthuiutaskueujobuindup((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _handle_tasksUs<           uPool._handle_taskscCsýtjƒ}x»y |ƒ}Wn&ttfk rDtdƒdSYnX|jrq|jtksct‚tdƒPn|dkr‹tdƒPn|\}}}y||j ||ƒWqt k rÆYqXqx§|rs|jtkrsy |ƒ}Wn&ttfk rtdƒdSYnX|dkr4tdƒqÍn|\}}}y||j ||ƒWqÍt k roYqÍXqÍWt |dƒràtdƒy5x.t dƒD] }|j jƒs¶Pn|ƒqWWqàttfk rÜYqàXntdt|ƒ|jƒdS( Nu.result handler got EOFError/IOError -- exitingu,result handler found thread._state=TERMINATEuresult handler got sentinelu&result handler ignoring extra sentinelu_readeru"ensuring that outqueue is not fulli u7result handler exiting: len(cache)=%s, thread._state=%s(u threadingucurrent_threaduIOErroruEOFErrorudebugu_stateu TERMINATEuAssertionErroruNoneu_setuKeyErroruhasattrurangeu_readerupollulen(uoutqueueugetucacheuthreadutaskujobuiuobj((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_handle_resultssX                 uPool._handle_resultsccsCt|ƒ}x0ttj||ƒƒ}|s1dS||fVqdS(N(uiterutupleu itertoolsuislice(ufuncuitusizeux((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu _get_tasks¹s  uPool._get_taskscCstdƒ‚dS(Nu:pool objects cannot be passed between processes or pickled(uNotImplementedError(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __reduce__ÂsuPool.__reduce__cCs5tdƒ|jtkr1t|_t|j_ndS(Nu closing pool(udebugu_stateuRUNuCLOSEu_worker_handler(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyucloseÇs  u Pool.closecCs-tdƒt|_t|j_|jƒdS(Nuterminating pool(udebugu TERMINATEu_stateu_worker_handleru _terminate(uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu terminateÍs   uPool.terminatecCsntdƒ|jttfks%t‚|jjƒ|jjƒ|jjƒx|j D]}|jƒqVWdS(Nu joining pool( udebugu_stateuCLOSEu TERMINATEuAssertionErroru_worker_handlerujoinu _task_handleru_result_handleru_pool(uselfup((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyujoinÓs    u Pool.joincCsWtdƒ|jjƒx9|jƒrR|jjƒrR|jjƒtjdƒqWdS(Nu7removing tasks from inqueue until task handler finishedi( udebugu_rlockuacquireuis_aliveu_readerupollurecvutimeusleep(uinqueueu task_handlerusize((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_help_stuff_finishÜs    uPool._help_stuff_finishc Cs¬tdƒt|_t|_tdƒ|j||t|ƒƒ|jƒsct|ƒdksct‚t|_|jdƒtdƒt j ƒ|k r¢|j ƒn|rõt |ddƒrõtdƒx-|D]"} | j dkrÌ| jƒqÌqÌWntdƒt j ƒ|k r|j ƒntdƒt j ƒ|k rG|j ƒn|r¨t |ddƒr¨td ƒx;|D]0} | jƒrqtd | jƒ| j ƒqqqqWndS( Nufinalizing poolu&helping task handler/workers to finishiujoining worker handleru terminateuterminating workersujoining task handlerujoining result handlerujoining pool workersucleaning up worker %d(udebugu TERMINATEu_stateu_help_stuff_finishulenuis_aliveuAssertionErroruputuNoneu threadingucurrent_threadujoinuhasattruexitcodeu terminateupid( uclsu taskqueueuinqueueuoutqueueupooluworker_handleru task_handleruresult_handlerucacheup((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu_terminate_poolås8    $             uPool._terminate_poolcCs|S(N((uself((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu __enter__suPool.__enter__cCs|jƒdS(N(u terminate(uselfuexc_typeuexc_valuexc_tb((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyu__exit__su Pool.__exit__N("u__name__u __module__u __qualname__u__doc__uProcessuNoneu__init__u_join_exited_workersu_repopulate_poolu_maintain_poolu _setup_queuesuapplyumapustarmapu starmap_asyncuimapuimap_unorderedu apply_asyncu map_asyncu _map_asyncu staticmethodu_handle_workersu _handle_tasksu_handle_resultsu _get_tasksu __reduce__ucloseu terminateujoinu_help_stuff_finishu classmethodu_terminate_poolu __enter__u__exit__(u __locals__((u9/opt/alt/python33/lib64/python3.3/multiprocessing/pool.pyuPoollsB :       *:     . cBsb|EeZdZdd„Zdd„Zdd„Zd dd„Zd d d „Zd d „Z d S(u ApplyResultcCsJtjƒ|_ttƒ|_||_||_||_|||j s.         *ÿ²&%I