:;iKdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZerdd lmZej@jCej@jEe#d Z$dZ%dZ&dZ'dZ(dZ)dZ*GddZ+Gdde+Z,Gdde+Z-GddZ.GddZ/GddZ0y) a Makes it possible to do the compiled analysis in a subprocess. This has two goals: 1. Making it safer - Segfaults and RuntimeErrors as well as stdout/stderr can be ignored and dealt with. 2. Make it possible to handle different Python versions as well as virtualenvs. The architecture here is briefly: - For each Jedi `Environment` there is a corresponding subprocess which operates within the target environment. If the subprocess dies it is replaced at this level. - `CompiledSubprocess` manages exactly one subprocess and handles communication from the parent side. - `Listener` runs within the subprocess, processing each request and yielding results. - `InterpreterEnvironment` provides an API which matches that of `Environment`, but runs functionality inline rather than within a subprocess. It is thus used both directly in places where a subprocess is unnecessary and/or undesirable and also within subprocesses themselves. - `InferenceStateSubprocess` (or `InferenceStateSameProcess`) provide high level access to functionality within the subprocess from within the parent. Each `InterpreterState` has an instance of one of these, provided by its environment. N)partial)Thread)Dict TYPE_CHECKING) pickle_dump pickle_load)debug)memoize_method) functions)DirectObjectAccess AccessPathSignatureParam) InternalErrorInferenceStatez __main__.pyctjdk(r tj}||d<dt j v|d<tj|i|S#t$rd}Y.wrappersJDJ..22(( 3F//7 7r%)r<)rQrr{rzs` @r"rjz$InferenceStateSubprocess.__getattr__sT" 8r%c@t|tr tjt|St|trtfd|DSt|tr|Dcgc]}j|c}St|t r j |j}|St|tr$tj|jS|Scc}w#t$r&|jj|Y|SwxYw)Nc3@K|]}j|ywr:)rx).0orQs r" zCInferenceStateSubprocess._convert_access_handles..sF#Q55a8#s) isinstancerrxtuplelistrXrVrUrWadd_subprocessrYr accesses)rQrZrs` r"rxz0InferenceStateSubprocess._convert_access_handless c> *!4#?#?c #KL L U #F#FF F T "=@ASD003SA A \ * ,,,SVV4  Z (d::3<<HI I B  ,""4(&&s+  ,s*C)C..+DDc|jr=|jjs&|jj|jyyyr:)rprq is_crasheddelete_inference_staterrrQs r"__del__z InferenceStateSubprocess.__del__s9 ::d77BB  % % < z-CompiledSubprocess.__init__..sr%) _executable _env_vars collectionsdeque_inference_state_deletion_queue_cleanup_callable)rQ executableenv_varss r"rSzCompiledSubprocess.__init__s*%!/:/@/@/B,!-r%c tj}d|jjd|jd|j d|d S)N)rgetpidrtrbrr)rQpids r"__repr__zCompiledSubprocess.__repr__s7iik NN # #    OO    r%c tjd|jtjdj }|jt tjjtjj|djdtjddDf}t|tjtjtj|j}t!j"|_t't(|j*|j$fx|_}d|_|j1t3j4|t6|||_|S) NzStart environment subprocess %sparso.c32K|]}t|ywr:)str)r~xs r"rz2CompiledSubprocess._get_process..s:%9SV%9s)rBrCrDenv)targetr T)r dbgrrmodules__file__ _MAIN_PATHrpathdirnamerA version_infor#rPIPErr5Queue _stderr_queuerr,rD_stderr_threaddaemonstartrMfinalizerIr)rQ parso_pathr rFts r" _get_processzCompiledSubprocess._get_processs 3T5E5EF[[)22     GGOOBGGOOJ7 8 HH:S%5%5bq%9: :   $ //????  #[[]"("..$"4"45#  a  ")!1!1$2B2923"5r%rec |jj}|j|d.#t$rYnwxYwt |sJ|j||||Sr:)rpop_send IndexErrorcallable)rQinference_state_idfunctionr r! delete_ids r"rwzCompiledSubprocess.runsm , @@DDF  9d+    !!!zz,hfEEs 0 <<cF|jdtjdiS)Nre)rr get_sys_pathrs r"rzCompiledSubprocess.get_sys_path*szz$ 6 6B??r%c2d|_|jy)NT)rrrs r"_killzCompiledSubprocess._kill-s  r%c |jrtd|jz||||f} t||j j t  t|j j\}}}t!|j"|r |f|_||S#t$r)|jtd|jzwxYw#t$r} |j jjjdd} n#t$r} d| z} Yd} ~ nd} ~ wwxYw|jt!|j"td|jd| d| d d} ~ wwxYw) NzThe subprocess %s has crashed.z2The subprocess %s was killed. Maybe out of memory?r.r/zzThe subprocess z has crashed (z , stderr=z).)rrrrrrBPICKLE_PROTOCOLBrokenPipeErrorrrrCEOFErrorrDreadr2 Exceptionr8rr ) rQrrr r!data is_exception tracebackry eof_errorrDexcs r"rzCompiledSubprocess._send1se ?? @4CSCS ST T!8T69 4 d//177 I  .9$:K:K:M:T:T.U +L)V T//0 %,FKL 9 4 JJL T"&"2"2!34 4 4  <**,3388:AA'9U <5; < JJL !3!3 4$$  sH)B 'C 2C E<8DE7 D0!D+&E7+D00AE77E<c:|jj|y)af Indicate that an inference state (in the subprocess) is no longer needed. The state corresponding to the given id will become inaccessible and the id may safely be re-used to refer to a different context. Note: it is not guaranteed that the corresponding state will actually be deleted immediately. N)rappend)rQrs r"rz)CompiledSubprocess.delete_inference_stateVs& ,,334FGr%r:)rbrcrdrkrrSrr rrwrrrrrer%r"rrsW J.  >68 F@!8:"#JHr%rc(eZdZdZdZdZdZdZy)ListeneraM Main loop for the subprocess which actually does the inference. This class runs within the target environment. It listens to instructions from the parent process, runs inference and returns the results. The subprocess has a long lifetime and is expected to process several requests, including for different `InferenceState` instances in the parent. See `CompiledSubprocess` for the parent half of the system. Communication is via pickled data sent serially over stdin and stdout. Stderr is read only if the child process crashes. The request protocol is a 4-tuple of: * inference_state_id | None: an opaque identifier of the parent's `InferenceState`. An `InferenceState` operating over an `InterpreterEnvironment` is created within this process for each of these, ensuring that each parent context has a corresponding context here. This allows context to be persisted between requests. Unless `None`, the local `InferenceState` will be passed to the given function as the first positional argument. * function | None: the function to run. This is expected to be a member of `.functions`. `None` indicates that the corresponding inference state is no longer needed and should be dropped. * args: positional arguments to the `function`. If any of these are `AccessHandle` instances they will be adapted to the local `InferenceState` before being passed. * kwargs: keyword arguments to the `function`. If any of these are `AccessHandle` instances they will be adapted to the local `InferenceState` before being passed. The result protocol is a 3-tuple of either: * (False, None, function result): if the function returns without error, or * (True, traceback, exception): if the function raises an exception ci|_yr:)_inference_statesrs r"rSzListener.__init__s !#r%cddlm} |j|}|S#t$r(ddlm}|d|}||j|<Y|SwxYw)Nrr)InterpreterEnvironment)project environment)jedi.inferencerrrWjedir)rQrrrrRrs r"_get_inference_statezListener._get_inference_statesi1 I"445GHO I 3,24 O :ID " "#5 6 Is-A  A c|||i|S||j|=y|j||}t|}t|D]>\}}t |t s|j j|j||<@|jD]>\}} t | t s|j j| j||<@||g|i|Sr:) rrr enumeraterrXrsrVrUitems) rQrrr r!rRiargkeyvalues r"_runz Listener._runs  %T,V, ,  &&'9:"77BTUO:D#D/3c<0-AASSTWTZTZ[DG*%lln Ue\2"1"E"E"W"WX]X`X`"aF3K-O=d=f= =r%ctj}ttjdt_tj }|j }|j } t|} dd|jf}t||t1#t$rtdY;wxYw#t$r!}dtj|f}Yd}~Qd}~wwxYw)NwTrF)rrCopenrdevnullrBbufferrrexitrrr format_excrr)rQrCrBpayloadryes r"listenzListener.listens"**c*    %e,  9idii&99  8 Q   9y335q8 9s* B*B(B%$B%( C1C  CN)rbrcrdrkrSrrrrer%r"rrls"H$ >,9r%rcXeZdZdedededdfdZdZdZd Z d Z d Z d Z e d Zy)rXrr\r[raNc.||_||_||_yr:)r\ _subprocessrU)rQrr\r[s r"rSzAccessHandle.__init__s  %r%c||_yr:)r)rQrs r"rzAccessHandle.add_subprocesss %r%c |j}d|jj d|dS#t$rdt|jz}Y@wxYw)N#rz of r)r\rrrUrtrb)rQdetails r"rzAccessHandle.__repr__sK ([[F $~~66?? (3tww<'F (s +!AAc|jSr:rUrs r" __getstate__zAccessHandle.__getstate__s wwr%c||_yr:r)rQstates r" __setstate__zAccessHandle.__setstate__s r%cn|dvs|jdr tdt|j|S)N)rUr\_z$Something went wrong with unpickling) startswithrr _workaroundris r"rjzAccessHandle.__getattr__s7 # #ts'; !GH Ht''..r%c|r>t|dtr+|jj|j|g|i|S|j |g|i|S)z TODO Currently we're passing slice objects around. This should not happen. They are also the only unhashable objects that we're passing around. r)rslicerget_compiled_method_returnrU_cached_resultsrQrr r!s r"rzAccessHandle._workarounds[ JtAw.>4##>>tww^t^W]^ ^#t##D:4:6::r%cX|jj|j|g|i|Sr:)rrrUrs r"rzAccessHandle._cached_resultss,:t::477DZ4ZSYZZr%)rbrcrdrKr intrSrrrrrjrr rrer%r"rXrXsb*#   &@/;[[r%rX)1rkrrrr5rrrM functoolsr threadingrtypingrrjedi._compatibilityrrrr jedi.cacher "jedi.inference.compiled.subprocessr jedi.inference.compiled.accessr r rjedi.api.exceptionsrrrrrArrrrr#r,r8r<rIrKrgrmrrrXrer%r"rs4 &8%8--WW\\"''//(3] C  - $  **,M 6MSW5SWlIHIHXc9c9L.[.[r%