B žHe,ã@sLddlZddlZddlmZmZddlmZmZe d¡Z Gdd„dƒZ dS)éN)ÚOptionalÚUnioné)ÚLanguageFilterÚ ProbingStates%[a-zA-Z]*[€-ÿ]+[a-zA-Z]*[^a-zA-Z€-ÿ]?c@sìeZdZdZejfeddœdd„Zddœdd„Zee e dœd d „ƒZ ee e dœd d „ƒZ e eefed œdd„Zeedœdd„ƒZedœdd„Zee eefedœdd„ƒZee eefedœdd„ƒZee eefedœdd„ƒZdS)Ú CharSetProbergffffffî?N)Ú lang_filterÚreturncCs$tj|_d|_||_t t¡|_dS)NT) rÚ DETECTINGÚ_stateÚactiverÚloggingÚ getLoggerÚ__name__Úlogger)Úselfr©rúu/var/www/downstreamdata.science/rtclock/rtclock-venv/lib/python3.7/site-packages/pip/_vendor/chardet/charsetprober.pyÚ__init__,szCharSetProber.__init__)r cCs tj|_dS)N)rr r )rrrrÚreset2szCharSetProber.resetcCsdS)Nr)rrrrÚ charset_name5szCharSetProber.charset_namecCst‚dS)N)ÚNotImplementedError)rrrrÚlanguage9szCharSetProber.language)Úbyte_strr cCst‚dS)N)r)rrrrrÚfeed=szCharSetProber.feedcCs|jS)N)r )rrrrÚstate@szCharSetProber.statecCsdS)Ngr)rrrrÚget_confidenceDszCharSetProber.get_confidence)Úbufr cCst dd|¡}|S)Ns([-])+ó )ÚreÚsub)rrrrÚfilter_high_byte_onlyGsz#CharSetProber.filter_high_byte_onlycCs^tƒ}t |¡}xH|D]@}| |dd…¡|dd…}| ¡sL|dkrLd}| |¡qW|S)u7 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [€-ÿ] marker: everything else [^a-zA-Z€-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. Néÿÿÿÿó€r)Ú bytearrayÚINTERNATIONAL_WORDS_PATTERNÚfindallÚextendÚisalpha)rÚfilteredÚwordsÚwordÚ last_charrrrÚfilter_international_wordsLs    z(CharSetProber.filter_international_wordscCs–tƒ}d}d}t|ƒ d¡}x^t|ƒD]R\}}|dkrD|d}d}q&|dkr&||krt|st| |||…¡| d¡d}q&W|s’| ||d …¡|S) a[ Returns a copy of ``buf`` that retains only the sequences of English alphabet and high byte characters that are not between <> characters. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. FrÚcó>rós