<_9Y0 @ s d Z d d l Z d d l Z d d l Z d d l m Z d d l m Z m Z m Z d d l
m Z d d l m
Z
d d l m Z d d l m Z Gd
d d e Z d S) a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N )CharSetGroupProber)
InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MBCSGroupProber)SBCSGroupProberc @ s e Z d Z d Z d Z e j d Z e j d Z e j d Z d d d d d
d d d
d d d d d d d d i Z
e j d d Z
d d Z d d Z d d Z d S) UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g?s [-]s (|~{)s [-]z
iso-8859-1z Windows-1252z
iso-8859-2z Windows-1250z
iso-8859-5z Windows-1251z
iso-8859-6z Windows-1256z
iso-8859-7z Windows-1253z
iso-8859-8z Windows-1255z
iso-8859-9z Windows-1254z iso-8859-13z Windows-1257c C sq d | _ g | _ d | _ d | _ d | _ d | _ d | _ | | _ t j t
| _ d | _ | j
d S)N)_esc_charset_prober_charset_probersresultdone _got_data _input_state
_last_char lang_filterlogging getLogger__name__logger_has_win_bytesreset)selfr r /universaldetector.py__init__Q s zUniversalDetector.__init__c C s d d d d d d i | _ d | _ d | _ d | _ t j | _ d | _ | j ra | j j x | j
D] } | j qk Wd S)z
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
encodingN
confidenceg languageF ) r r r r r
PURE_ASCIIr r r r r
)r proberr r r r ^ s
zUniversalDetector.resetc C sF | j r
d St | s d St | t s8 t | } | j sc| j t j rq d d d d d d i | _ n | j t j t j
f r d d d d d d i | _ n | j d r d d
d d d d i | _ nc | j d rd d d d d d i | _ n6 | j t j t j f r:d d
d d d d i | _ d | _ | j d d k rcd | _ d S| j
t j k r| j j | rt j | _
n7 | j
t j k r| j j | j | rt j | _
| d d | _ | j
t j k rd| j s
t | j | _ | j j | t j k rBd | j j d | j j d | j j i | _ d | _ n | j
t j k rB| j st | j g | _ | j t! j" @r| j j# t$ | j j# t% xZ | j D]O } | j | t j k rd | j d | j d | j i | _ d | _ PqW| j& j | rBd | _' d S)a
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nr z UTF-8-SIGr g ?r zUTF-32s zX-ISO-10646-UCS-4-3412s zX-ISO-10646-UCS-4-2143zUTF-16Tr )(r len
isinstance bytearrayr
startswithcodecsBOM_UTF8r BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr r r" HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr Z ESC_ASCIIr r r feedr ZFOUND_IT charset_nameget_confidencer r
r r ZNON_CJKappendr
r WIN_BYTE_DETECTORr )r Zbyte_strr# r r r r4 o s zUniversalDetector.feedc C s | j r | j Sd | _ | j s5 | j j d n| j t j k re d d d d d d i | _ n | j t j k rNd } d
} d } x> | j D]3 } | s q | j
} | | k r | } | } q W| rN| | j k rN| j } | j j
} | j
} | j d r0| j r0| j j | | } d | d | d | j i | _ | j j t j k r| j d d k r| j j d x | j D] } | sqt | t rx^ | j D]+ } | j j d
| j | j | j
qWq| j j d
| j | j | j
qW| j S)z
Stop analyzing the current document and come up with a final
prediction.
:returns: The ``result`` attribute, a ``dict`` with the keys
`encoding`, `confidence`, and `language`.
Tzno data received!r asciir g ?r r$ Ng ziso-8859z no probers hit minimum thresholdz%s %s confidence = %s)r r r r debugr r r" r2 r
r6 MINIMUM_THRESHOLDr5 lowerr) r ISO_WIN_MAPgetr getEffectiveLevelr DEBUGr' r Zprobers) r Zprober_confidenceZmax_prober_confidenceZ
max_proberr# r5 Zlower_charset_namer Z group_proberr r r close s`
zUniversalDetector.closeN)r
__module__ __qualname____doc__r; recompiler0 r3 r8 r= r ZALLr r r4 rA r r r r r 3 s"
mr )rD r* r rE Zcharsetgroupproberr Zenumsr r r Z escproberr Z latin1proberr Zmbcsgroupproberr Zsbcsgroupproberr
objectr r r r r <module>$ s