Viewing File: /opt/alt/python35/lib/python3.5/site-packages/chardet/__pycache__/charsetprober.cpython-35.pyc
<_9Yö ã @ sB d d l Z d d l Z d d l m Z Gd d „ d e ƒ Z d S)é Né )Ú ProbingStatec @ s£ e Z d Z d Z d d d „ Z d d „ Z e d d „ ƒ Z d d
„ Z e d d „ ƒ Z d
d „ Z
e d d „ ƒ Z e d d „ ƒ Z
e d d „ ƒ Z d S)Ú
CharSetProbergffffffî?Nc C s( d | _ | | _ t j t ƒ | _ d S)N)Ú_stateÚ lang_filterÚloggingÚ getLoggerÚ__name__Úlogger)Úselfr © r ú/charsetprober.pyÚ__init__' s zCharSetProber.__init__c C s t j | _ d S)N)r Z DETECTINGr )r r r r
Úreset, s zCharSetProber.resetc C s d S)Nr )r r r r
Ú charset_name/ s zCharSetProber.charset_namec C s d S)Nr )r Úbufr r r
Úfeed3 s zCharSetProber.feedc C s | j S)N)r )r r r r
Ústate6 s zCharSetProber.statec C s d S)Ng r )r r r r
Úget_confidence: s z CharSetProber.get_confidencec C s t j d d | ƒ } | S)Ns ([ -])+ó )ÚreÚsub)r r r r
Úfilter_high_byte_only= s z#CharSetProber.filter_high_byte_onlyc C sƒ t ƒ } t j d | ƒ } xa | D]Y } | j | d d … ƒ | d d … } | j ƒ rn | d k rn d } | j | ƒ q" W| S)u9
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [€-ÿ]
marker: everything else [^a-zA-Z€-ÿ]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
s% [a-zA-Z]*[€-ÿ]+[a-zA-Z]*[^a-zA-Z€-ÿ]?Nr s €r éÿÿÿÿr )Ú bytearrayr ÚfindallÚextendÚisalpha)r ÚfilteredZwordsZwordZ last_charr r r
Úfilter_international_wordsB s
z(CharSetProber.filter_international_wordsc C sè t ƒ } d } d } x¯ t t | ƒ ƒ D]› } | | | d … } | d k rW d } n | d k ri d } | d k r( | j ƒ r( | | k r¹ | r¹ | j | | | … ƒ | j d ƒ | d } q( W| sä | j | | d … ƒ | S)
aÈ
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
Also retains English alphabet and high byte characters immediately
before occurrences of >.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
Fr r ó >ó <Ts €r N)r ÚrangeÚlenr r )r r Zin_tagÚprevZcurrZbuf_charr r r
Úfilter_with_english_lettersg s"
z)CharSetProber.filter_with_english_letters)r Ú
__module__Ú __qualname__ZSHORTCUT_THRESHOLDr r Úpropertyr r r r Ú staticmethodr r r% r r r r
r # s %r )r r Zenumsr Úobjectr r r r r
Ú<module> s
Back to Directory
File Manager