<_9Y$ @ s* d d l m Z m Z m Z d d l m Z m Z m Z d d l m Z m
Z
m Z d d l m
Z
m Z m Z d d l m Z m Z m Z Gd d d e Z Gd d d e Z Gd
d d e Z Gd d
d
e Z Gd d d e Z Gd d d e Z Gd d d e Z d S) )EUCTW_CHAR_TO_FREQ_ORDEREUCTW_TABLE_SIZE EUCTW_TYPICAL_DISTRIBUTION_RATIO)EUCKR_CHAR_TO_FREQ_ORDEREUCKR_TABLE_SIZE EUCKR_TYPICAL_DISTRIBUTION_RATIO)GB2312_CHAR_TO_FREQ_ORDERGB2312_TABLE_SIZE!GB2312_TYPICAL_DISTRIBUTION_RATIO)BIG5_CHAR_TO_FREQ_ORDERBIG5_TABLE_SIZE BIG5_TYPICAL_DISTRIBUTION_RATIO)JIS_CHAR_TO_FREQ_ORDERJIS_TABLE_SIZE JIS_TYPICAL_DISTRIBUTION_RATIOc @ sp e Z d Z d Z d Z d Z d Z d d Z d d Z d d
Z d d Z
d
d Z d d Z d S)CharDistributionAnalysisi gGz?g{Gz? c C sD d | _ d | _ d | _ d | _ d | _ d | _ | j d S)N)_char_to_freq_order _table_sizetypical_distribution_ratio_done _total_chars _freq_charsreset)self r /chardistribution.py__init__. s z!CharDistributionAnalysis.__init__c C s d | _ d | _ d | _ d S)z reset analyser, clear any stateF N)r r r )r r r r r = s z CharDistributionAnalysis.resetc C st | d k r | j | } n d } | d k rp | j d 7_ | | j k rp d | j | k rp | j d 7_ d S)z"feed a character with known length r r i N) get_orderr r r r )r charZchar_lenorderr r r feedF s z CharDistributionAnalysis.feedc C sr | j d k s! | j | j k r( | j S| j | j k rk | j | j | j | j } | | j k rk | S| j S)z(return confidence based on existing datar )r r MINIMUM_DATA_THRESHOLDSURE_NOr SURE_YES)r rr r r get_confidenceT s ! z'CharDistributionAnalysis.get_confidencec C s | j | j k S)N)r ENOUGH_DATA_THRESHOLD)r r r r got_enough_datad s z(CharDistributionAnalysis.got_enough_datac C s d S)Nr r r )r byte_strr r r r! i s z"CharDistributionAnalysis.get_orderN)
__name__
__module__ __qualname__r* r' r& r% r r r$ r) r+ r! r r r r r ( s r c s. e Z d Z f d d Z d d Z S)EUCTWDistributionAnalysisc s2 t t | j t | _ t | _ t | _ d S)N) superr0 r r r r r r r )r ) __class__r r r r s z"EUCTWDistributionAnalysis.__init__c C s6 | d } | d k r. d | d | d d Sd Sd S)Nr ^ r r r )r r,
first_charr r r r! x s
z#EUCTWDistributionAnalysis.get_order)r- r. r/ r r! r r )r2 r r0 q s r0 c s. e Z d Z f d d Z d d Z S)EUCKRDistributionAnalysisc s2 t t | j t | _ t | _ t | _ d S)N) r1 r7 r r r r r r r )r )r2 r r r s z"EUCKRDistributionAnalysis.__init__c C s6 | d } | d k r. d | d | d d Sd Sd S)Nr r4 r r5 r r )r r, r6 r r r r! s
z#EUCKRDistributionAnalysis.get_order)r- r. r/ r r! r r )r2 r r7 s r7 c s. e Z d Z f d d Z d d Z S)GB2312DistributionAnalysisc s2 t t | j t | _ t | _ t | _ d S)N) r1 r9 r r r r r r
r )r )r2 r r r s z#GB2312DistributionAnalysis.__init__c C sI | d | d } } | d k rA | d k rA d | d | d Sd Sd S)Nr r r8 r5 r4 r r )r r, r6 second_charr r r r! s z$GB2312DistributionAnalysis.get_order)r- r. r/ r r! r r )r2 r r9 s r9 c s. e Z d Z f d d Z d d Z S)Big5DistributionAnalysisc s2 t t | j t | _ t | _ t | _ d S)N) r1 r; r r r r r r
r )r )r2 r r r s z!Big5DistributionAnalysis.__init__c C sd | d | d } } | d k r\ | d k rE d | d | d d Sd | d | d Sn d Sd S) Nr r r5 ? @ r r )r r, r6 r: r r r r! s z"Big5DistributionAnalysis.get_order)r- r. r/ r r! r r )r2 r r; s r; c s. e Z d Z f d d Z d d Z S)SJISDistributionAnalysisc s2 t t | j t | _ t | _ t | _ d S)N) r1 r@ r r r r r r r )r )r2 r r r s z!SJISDistributionAnalysis.__init__c C s | d | d } } | d k r> | d k r> d | d } n1 | d k rk | d k rk d | d d } n d S| | d } | d
k r d } | S)
Nr r r? r r r )r r, r6 r: r# r r r r! s z"SJISDistributionAnalysis.get_order)r- r. r/ r r! r r )r2 r r@ s r@ c s. e Z d Z f d d Z d d Z S)EUCJPDistributionAnalysisc s2 t t | j t | _ t | _ t | _ d S)N) r1 rH r r r r r r r )r )r2 r r r s z"EUCJPDistributionAnalysis.__init__c C s6 | d } | d k r. d | d | d d Sd Sd S)Nr r4 r5 r r r )r r, r" r r r r! s
z#EUCJPDistributionAnalysis.get_order)r- r. r/ r r! r r )r2 r rH s rH N) Z euctwfreqr r r Z euckrfreqr r r Z
gb2312freqr r r
Zbig5freqr r r
Zjisfreqr r r objectr r0 r7 r9 r; r@ rH r r r r <module> s I