B He:@sdZddlZddlZddlZddlmZmZmZddlm Z ddl m Z ddl m Z mZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZGdddZdS)a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco N)ListOptionalUnion)CharSetGroupProber) CharSetProber) InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MacRomanProber)MBCSGroupProber) ResultDict)SBCSGroupProber) UTF1632Proberc @seZdZdZdZedZedZedZ dddd d d d d dZ dddd ddddZ e j dfe eddddZeedddZeedddZeeedddZddd d!Zeeefdd"d#d$Zedd%d&ZdS)'UniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g?s[-]s(|~{)s[-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9z iso-8859-13z ISO-8859-11ZGB18030CP949zUTF-16)asciiz iso-8859-1ztis-620z iso-8859-9gb2312zeuc-krzutf-16leFN) lang_filtershould_rename_legacyreturncCsdd|_d|_g|_dddd|_d|_d|_tj|_d|_ ||_ t t |_d|_||_|dS)Ng)encoding confidencelanguageF)_esc_charset_prober_utf1632_prober_charset_probersresultdone _got_datar PURE_ASCII _input_state _last_charrlogging getLogger__name__logger_has_win_bytesrreset)selfrrr-y/var/www/downstreamdata.science/rtclock/rtclock-venv/lib/python3.7/site-packages/pip/_vendor/chardet/universaldetector.py__init__ds  zUniversalDetector.__init__)rcCs|jS)N)r$)r,r-r-r. input_state{szUniversalDetector.input_statecCs|jS)N)r*)r,r-r-r. has_win_bytesszUniversalDetector.has_win_bytescCs|jS)N)r)r,r-r-r.charset_proberssz!UniversalDetector.charset_proberscCsjdddd|_d|_d|_d|_tj|_d|_|jr>|j |j rN|j x|j D] }| qVWdS)z Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Ng)rrrFr) r r!r"r*rr#r$r%rr+rr)r,proberr-r-r.r+s   zUniversalDetector.reset)byte_strrcCs|jr dS|sdSt|ts$t|}|js|tjrFdddd|_nv|tjtj frhdddd|_nT|drdddd|_n:|d rd ddd|_n |tj tj frd ddd|_d |_|jd dk rd |_dS|j t jkr(|j|rt j|_ n*|j t jkr(|j|j|r(t j|_ |dd|_|jsFt|_|jjtjkr|j|tjkr|jj|jdd|_d |_dS|j t jkr|jst|j |_|j|tjkr|jj|j|jj!d|_d |_n|j t jkr|j"sBt#|j g|_"|j t$j%@r&|j"&t'|j"&t(|j"&t)x@|j"D]6}||tjkrJ|j||j!d|_d |_PqJW|j*|rd |_+dS)a Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIGg?)rrrzUTF-32szX-ISO-10646-UCS-4-3412szX-ISO-10646-UCS-4-2143zUTF-16Tr),r! isinstance bytearrayr" startswithcodecsBOM_UTF8r BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr$rr#HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr% ESC_ASCIIrrstater DETECTINGfeedFOUND_IT charset_nameget_confidencerr rrrrr NON_CJKappendrr r WIN_BYTE_DETECTORr*)r,r4r3r-r-r.rGs             zUniversalDetector.feedc Cs|jr |jSd|_|js&|jdn|jtjkrBdddd|_n|jtjkrd}d}d}x,|j D]"}|snqd| }||krd|}|}qdW|r||j kr|j }|dk st |}| }|d r|jr|j||}|jr|j|pd|}|||jd|_|jtjkr|jd dkr|jd xn|j D]d}|sLq>t|trxF|jD] }|jd |j |j| q`Wn|jd |j |j| q>W|jS) z Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!rg?r5)rrrNgziso-8859rz no probers hit minimum thresholdz%s %s confidence = %s)r!r r"r)debugr$rr#rBrrJMINIMUM_THRESHOLDrIAssertionErrorlowerr9r* ISO_WIN_MAPgetr LEGACY_MAPrgetEffectiveLevelr&DEBUGr7rprobers) r,prober_confidencemax_prober_confidence max_proberr3rIlower_charset_namer group_proberr-r-r.closesh          zUniversalDetector.close)r( __module__ __qualname____doc__rOrecompiler@rCrMrRrTr ALLboolr/propertyintr0r1rrr2r+rbytesr8rGrr]r-r-r-r.r8sB   r)r`r:r&ratypingrrrcharsetgroupproberr charsetproberrenumsrr r escproberr latin1proberr Zmacromanproberr mbcsgroupproberrZ resultdictrsbcsgroupproberrZ utf1632proberrrr-r-r-r.$s