Viewing File: /opt/alt/python38/lib/python3.8/site-packages/chardet/__pycache__/universaldetector.cpython-38.pyc

U

<_9Y0@sdZddlZddlZddlZddlmZddlmZmZm	Z	ddl
mZddlm
Z
ddlmZdd	lmZGd
ddeZdS)a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.

:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N)CharSetGroupProber)
InputStateLanguageFilterProbingState)EscCharSetProber)Latin1Prober)MBCSGroupProber)SBCSGroupProberc	@sneZdZdZdZedZedZedZ	dddd	d
ddd
dZ
ejfddZ
ddZddZddZdS)UniversalDetectoraq
    The ``UniversalDetector`` class underlies the ``chardet.detect`` function
    and coordinates all of the different charset probers.

    To get a ``dict`` containing an encoding and its confidence, you can simply
    run:

    .. code::

            u = UniversalDetector()
            u.feed(some_bytes)
            u.close()
            detected = u.result

    g?s[-]s(|~{)s[-]zWindows-1252zWindows-1250zWindows-1251zWindows-1256zWindows-1253zWindows-1255zWindows-1254zWindows-1257)z
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8z
iso-8859-9ziso-8859-13cCsNd|_g|_d|_d|_d|_d|_d|_||_t	t
|_d|_|
dS)N)_esc_charset_prober_charset_probersresultdone	_got_data_input_state
_last_charlang_filterloggingZ	getLogger__name__logger_has_win_bytesreset)selfrrJ/opt/alt/python38/lib/python3.8/site-packages/chardet/universaldetector.py__init__QszUniversalDetector.__init__cCsVdddd|_d|_d|_d|_tj|_d|_|jr>|j	|j
D]}|	qDdS)z
        Reset the UniversalDetector and all of its probers back to their
        initial states.  This is called by ``__init__``, so you only need to
        call this directly in between analyses of different documents.
        Nencoding
confidencelanguageF)rrrrr
PURE_ASCIIrrrrr
)rproberrrrr^s

zUniversalDetector.resetcCs>|jr
dSt|sdSt|ts(t|}|js|tjrJdddd|_nv|tj	tj
frldddd|_nT|drdddd|_n:|d	rd
ddd|_n |tjtjfrdddd|_d|_|jd
dk	rd|_dS|j
tjkr.|j|rtj|_
n*|j
tjkr.|j|j|r.tj|_
|dd|_|j
tjkr|js^t|j|_|j|tjkr:|jj|j|jjd|_d|_n|j
tjkr:|jst |jg|_|jt!j"@r|j#t$|j#t%|jD]:}||tjkr|j||jd|_d|_q&q|j&|r:d|_'dS)a
        Takes a chunk of a document and feeds it through all of the relevant
        charset probers.

        After calling ``feed``, you can check the value of the ``done``
        attribute to see if you need to continue feeding the
        ``UniversalDetector`` more data, or if it has made a prediction
        (in the ``result`` attribute).

        .. note::
           You should always call ``close`` when you're done feeding in your
           document if ``done`` is not already ``True``.
        Nz	UTF-8-SIG?rzUTF-32szX-ISO-10646-UCS-4-3412szX-ISO-10646-UCS-4-2143zUTF-16Tr)(rlen
isinstance	bytearrayr
startswithcodecsBOM_UTF8rBOM_UTF32_LEBOM_UTF32_BEBOM_LEBOM_BErrr#HIGH_BYTE_DETECTORsearch	HIGH_BYTEESC_DETECTORrZ	ESC_ASCIIrrrfeedrZFOUND_ITcharset_nameget_confidencer!r
r	rZNON_CJKappendr
rWIN_BYTE_DETECTORr)rZbyte_strr$rrrr6os









zUniversalDetector.feedc		Cst|jr|jSd|_|js&|jdn|jtjkrBdddd|_n|jtjkrd}d}d}|j	D]"}|sjq`|
}||kr`|}|}q`|r||jkr|j}|j
}|
}|d	r|jr|j||}|||jd|_|jtjkrn|jd
dkrn|jd|j	D]`}|sqt|trP|jD] }|jd|j|j|
q,n|jd|j|j|
q|jS)
z
        Stop analyzing the current document and come up with a final
        prediction.

        :returns:  The ``result`` attribute, a ``dict`` with the keys
                   `encoding`, `confidence`, and `language`.
        Tzno data received!asciir%r&rNrziso-8859rz no probers hit minimum thresholdz%s %s confidence = %s)rrrrdebugrrr#r4r
r8MINIMUM_THRESHOLDr7lowerr+rISO_WIN_MAPgetr!ZgetEffectiveLevelrDEBUGr)rZprobers)	rZprober_confidenceZmax_prober_confidenceZ
max_proberr$r7Zlower_charset_namer Zgroup_proberrrrclosesj	






zUniversalDetector.closeN)r
__module____qualname____doc__r=recompiler2r5r:r?rZALLrrr6rBrrrrr3s$


	
mr)rEr,rrFZcharsetgroupproberrZenumsrrrZ	escproberrZlatin1proberrZmbcsgroupproberr	Zsbcsgroupproberr
objectrrrrr<module>s
Back to Directory File Manager