]jD+
n d dl Z d dlmZ d dlmZmZ d dlmZ d dlm Z m
Z
m Z m Z ddl
mZ ddlmZmZmZmZ dd lmZ dd
lmZ dd lmZmZmZm Z m Z d e d
e
e fdZ de d
e
e fdZ e d e d
e
e fd Z! e d e d
e
e fd Z" ee de d
e e#e#f fd Z$ d%de
e de#d
e
e fdZ%de de
e d
e&fdZ'd e d
e
e fd Z(d e
e d
efd Z) ed d&d e d"e&d#e e d
efd$ Z*dS )' N)IncrementalDecoder)Counter OrderedDict) lru_cache)DictListOptionalTuple ) FREQUENCIES)KO_NAMESLANGUAGE_SUPPORTED_COUNTTOO_SMALL_SEQUENCEZH_NAMES) is_suspiciously_successive_range)CoherenceMatches)is_accentuatedis_latinis_multi_byte_encodingis_unicode_range_secondary
unicode_range iana_namereturnc t | rt d t j d | j } |d }i dt
dd D ]h}| t |g }|rAt | }|9t | d u r |vrd|< |xx d
z
cc< d
z
it fd D S ) zF
Return associated unicode ranges in a single byte code page.
z.Function not supported on multi-byte code pagez encodings.{}ignore)errorsr @ NFr c 2 g | ]}| z d k |S )g333333? ).0character_rangecharacter_count seen_rangess u/builddir/build/BUILD/imunify360-venv-2.6.3/opt/imunify360/venv/lib/python3.11/site-packages/charset_normalizer/cd.py
<listcomp>z*encoding_unicode_range.<locals>.<listcomp>1 s8
?+o=EE
EEE ) r IOError importlib
import_moduleformatr rangedecodebytesr r sorted)r decoderpichunkr" r# r$ s @@r% encoding_unicode_ranger4 s? i(( HFGGG%n&;&;I&F&FGGZGx AKO
4
%
%s$$
% +E 2 2O &)/::eCC"+5534K0O,,,1,,, 1$
#.
r'
primary_rangec g }t j D ]4\ }}|D ],}t | | k r| | n-5|S )z>
Return inferred languages used with a unicode range.
)r itemsr append)r5 languageslanguage
characters characters r% unicode_range_languagesr= 9 so I + 1 3 3 *# I Y''=88 *** 9 r' c f t | }d}|D ]
}d|vr|} n |dgS t | S )z
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
NLatin Latin Based)r4 r= )r unicode_rangesr5 specified_ranges r% encoding_languagesrC H s] ,I66NM) / ) ) +M E * "= 1 11r' c | d s0| d s| d s| dk rdgS | d s | t v rddgS | d s | t v rd
gS g S ) z
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
shift_
iso2022_jpeuc_jcp932JapanesegbChinesezClassical Chinese
iso2022_krKorean)
startswithr r
)r s r% mb_encoding_languagesrO \ s X&& - - ( ( |D!! 0Y(%:%:.//L)) Y(-B-Bz
Ir' )maxsizer: c d}d}t | D ]*}|st | rd}|rt | du rd}+||fS )zg
Determine main aspects from a supported language if it contains accents and if is pure Latin.
FT)r r r )r: target_have_accentstarget_pure_latinr< s r% get_target_featuresrT q sn
* & & " '~i'@'@ '"& &)!4!4!=!= % 1 11r' Fr; ignore_non_latinc g }t d D }t j D ]q\ }}t | \ }}|r|du r |du r|r%t | }t fd|D } | |z }
|
dk r| ||
f rt
|d d }d |D S ) zE
Return associated languages associated to given characters.
c 3 4 K | ]}t | V d S N)r )r! r< s r% <genexpr>z%alphabet_languages.<locals>.<genexpr> s* T TInY 7 7 T T T T T Tr' Fc g | ]}|v | S r r )r! cr; s r% r& z&alphabet_languages.<locals>.<listcomp> s ? ? ?1qJQr' g?c | d S Nr r xs r% <lambda>z$alphabet_languages.<locals>.<lambda> s
! r' Tkeyreversec g | ]
}|d S )r r )r! compatible_languages r% r& z&alphabet_languages.<locals>.<listcomp> s H H H': " H H Hr' )anyr r7 rT lenr8 r/ ) r; rU r9 source_have_accentsr: language_charactersrR rS r# character_match_countratios ` r% alphabet_languagesrl s I T T T T TTT)4):)<)< 0 0%%1DX1N1N. . 1U : : % ' ',? ' 122 # ? ? ? ?+ ? ? ?!
!
&7 C<< h . / / /ynndCCCI H Hi H H HHr' ordered_charactersc \ | t vr"t d | d}|D ]h}|t | vrt | dt | | }t | t | | d }|d| | || | d fd|D d }fd|D d }t | dk r
|dk r|dz
}t | dk r
|dk r|dz
}5|t | z d k s|t | z d k r|dz
}hj|t | z S )
aN
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
z{} not availabler Nc g | ]}|v S r r )r! echaracters_befores r% r& z1characters_popularity_compare.<locals>.<listcomp> s,
'(A" "
r' Tc g | ]}|v S r r )r! rp characters_afters r% r& z1characters_popularity_compare.<locals>.<listcomp> s,
&'A! !
r' r g?)r
ValueErrorr+ indexcountrg )
r: rm character_approved_countr< characters_before_sourcecharacters_after_sourcebefore_match_countafter_match_countrs rq s
@@r% characters_popularity_comparer} s+ {""+228<<=== ' * * K 1 1 1 #.x#8
H%++I66 6$
#.h"7 ! ' ' 2 2 4 4#
/
"((33 3
. $ $Y / / 1 1
,D
%
+B
%
' ( (A - -2D2I2I $ ) $ & ' '1 , ,1Ba1G1G $ ) $
%=!>!> ># E E 3'>#?#??3FF $ ) $ G
$c*<&=&= ==r' decoded_sequencec z t }| D ]}| du rt | }|+d}|D ]}t || du r|} n||}||vr| ||< h||xx | z
cc< t | S )a
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
One containing the latin letters and the other hebrew.
FN)r isalphar r lowerlistvalues)r~ layersr< r" layer_target_rangediscovered_ranges r% alpha_unicode_splitr s ]]F% 8 8 % ' ' ' 22 " ! & 01A?SS &6" %!0 V + +)2 ):):F% & !"""ioo&7&77""""
r' resultsc t | D ]2}|D ]-}|\ }}|vr|g|< | | .3fdD }t |d d S )z
This function merge results previously given by the function coherence_ratio.
The return type is the same as coherence_ratio.
c
g | ]=}|t t | t | z d f>S )rt )roundsumrg )r! r: per_language_ratioss r% r& z*merge_coherence_ratios.<locals>.<listcomp> sd
'122S9LX9V5W5WW
r' c | d S r] r r^ s r% r` z(merge_coherence_ratios.<locals>.<lambda>" s
qt r' Tra )r r8 r/ )r result
sub_resultr: rk merger s @r% merge_coherence_ratiosr s
&-- 8 8 8 8J (OHe 22216#H- ) 0 0 7 7 7 7 8
,
E %^^T : : ::r' i 皙? threshold lg_inclusionc 0 g }d}d}|| d ng }d|v rd}| d t | D ]}t | }| } t d | D }
|
t k rJd | D } |pt | | D ]Q} t | | }
|
|k r|
d k r|d
z
}| | t |
d f |d k r nRt |d
d S )z
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
A layer = Character extraction by alphabets/ranges.
Fr N,r@ Tc 3 K | ] \ }}|V
d S rX r r! r[ os r% rY z"coherence_ratio.<locals>.<genexpr>< s& 8 8DAqa 8 8 8 8 8 8r' c g | ]\ }}|S r r r s r% r& z#coherence_ratio.<locals>.<listcomp>A s $?$?$?41aQ$?$?$?r' g?r rt c | d S r] r r^ s r% r` z!coherence_ratio.<locals>.<lambda>T s
1 r' ra ) splitremover r most_commonr r rl r} r8 r r/ )r~ r r r rU sufficient_match_countlg_inclusion_listlayersequence_frequenciesr r# popular_character_orderedr: rk s r% coherence_ratior % s~ G 3?3K **3///QS))) ///$%566 &u~~ *6688 8 8K 8 8 888 0 0 0 $?$?;$?$?$?!)
-? %'7.
.
H 23 E y # &!+& NNHeE1oo6 7 7 7%** + '~~t < < <<r' )F)r N)+r) codecsr collectionsr r functoolsr typingr r r r
assetsr constantr
r r r mdr modelsr utilsr r r r r strr4 r= rC rO boolrT rl floatr} r r r r r' r% <module>r s@ % % % % % % , , , , , , , , . . . . . . . . . . . . V V V V V V V V V V V V 0 0 0 0 0 0 $ $ $ $ $ $ "c "d3i " " " "J 3 49 2# 2$s) 2 2 2
2& S T#Y
( +,,,
2#
2%d
*;
2
2
2 -,
2" 5:!I !IS !I-1!I #Y!I !I !I !IH9>9>'+Cy9>
9> 9> 9> 9>x$!# $!$s) $! $! $! $!N;D)9$: ;?O ; ; ; ;8 4QU.= .=.=&+.=AI# .=.= .= .= .= .= .=r'