Viewing File: /usr/lib/python3.6/site-packages/pip/_vendor/chardet/__pycache__/charsetprober.cpython-36.opt-1.pyc
3
£éajö ã @ s0 d dl Z d dlZddlmZ G dd„ deƒZdS )é Né )Ú ProbingStatec @ sn e Zd ZdZddd„Zdd„ Zedd„ ƒZd d
„ Zed d „ ƒZ d
d„ Z
e dd„ ƒZ e dd„ ƒZ
e dd„ ƒZdS )Ú
CharSetProbergffffffî?Nc C s d | _ || _tjtƒ| _d S )N)Ú_stateÚ lang_filterÚloggingZ getLoggerÚ__name__Zlogger)Úselfr © r
ú#/usr/lib/python3.6/charsetprober.pyÚ__init__' s zCharSetProber.__init__c C s t j| _d S )N)r Z DETECTINGr )r r
r
r Úreset, s zCharSetProber.resetc C s d S )Nr
)r r
r
r Ú charset_name/ s zCharSetProber.charset_namec C s d S )Nr
)r Úbufr
r
r Úfeed3 s zCharSetProber.feedc C s | j S )N)r )r r
r
r Ústate6 s zCharSetProber.statec C s dS )Ng r
)r r
r
r Úget_confidence: s z CharSetProber.get_confidencec C s t jdd| ƒ} | S )Ns ([ -])+ó )ÚreÚsub)r r
r
r Úfilter_high_byte_only= s z#CharSetProber.filter_high_byte_onlyc C sb t ƒ }tjd| ƒ}xJ|D ]B}|j|dd… ƒ |dd… }|jƒ rP|dk rPd}|j|ƒ qW |S )u9
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [€-ÿ]
marker: everything else [^a-zA-Z€-ÿ]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
s% [a-zA-Z]*[€-ÿ]+[a-zA-Z]*[^a-zA-Z€-ÿ]?Nr ó €r éÿÿÿÿr )Ú bytearrayr ÚfindallÚextendÚisalpha)r ÚfilteredZwordsZwordZ last_charr
r
r Úfilter_international_wordsB s
z(CharSetProber.filter_international_wordsc C s¬ t ƒ }d}d}x‚tt| ƒƒD ]r}| ||d … }|dkr>d}n |dkrJd}|dk r |jƒ r ||kr†| r†|j| ||… ƒ |jdƒ |d }q W |s¨|j| |d … ƒ |S )
aÈ
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
Also retains English alphabet and high byte characters immediately
before occurrences of >.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
Fr r ó >ó <Tr r N)r ÚrangeÚlenr r )r r Zin_tagÚprevZcurrZbuf_charr
r
r Úfilter_with_english_lettersg s"
z)CharSetProber.filter_with_english_letters)N)r Ú
__module__Ú __qualname__ZSHORTCUT_THRESHOLDr r
Úpropertyr r r r Ú staticmethodr r r$ r
r
r
r r # s
%r )r r Zenumsr Úobjectr r
r
r
r Ú<module> s
Back to Directory
File Manager