]jG
d dl mZ d dlmZmZ ddlmZmZ ddlm Z m
Z
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZ G d d Z G d d e Z G d
d e Z G d d
e Z G d de Z G d de Z G d de Z G d de Z G d de Z dee! dee! de"fdZ# ed d%d e!d!e$d"e"de$fd# Z%d$S )& ) lru_cache)ListOptional ) COMMON_SAFE_ASCII_CHARACTERS UNICODE_SECONDARY_RANGE_KEYWORD)is_accentuatedis_asciiis_case_variableis_cjk is_emoticon is_hangul is_hiragana is_katakanais_latinis_punctuation is_separator is_symbolis_thai
remove_accent
unicode_rangec V e Zd ZdZdedefdZdeddfdZd dZe de
fd Z dS )
MessDetectorPluginzy
Base abstract class used for mess detection plugins.
All detectors MUST extend and implement given methods.
characterreturnc t )z@
Determine if given character should be fed in.
NotImplementedErrorselfr s u/builddir/build/BUILD/imunify360-venv-2.6.3/opt/imunify360/venv/lib/python3.11/site-packages/charset_normalizer/md.pyeligiblezMessDetectorPlugin.eligible
"! Nc t )z
The main routine to be executed upon character.
Insert the logic in witch the text would be considered chaotic.
r r s r! feedzMessDetectorPlugin.feed$ s
"!r$ c t )zB
Permit to reset the plugin to the initial state.
r r s r! resetzMessDetectorPlugin.reset+ r# r$ c t )z
Compute the chaos ratio based on what your feed() has seen.
Must NOT be lower than 0.; No restriction gt 0.
r r( s r! ratiozMessDetectorPlugin.ratio1 s
"!r$ r N) __name__
__module__ __qualname____doc__strboolr" r& r) propertyfloatr+ r$ r! r r s
"# "$ " " " " "c "d " " " "" " " " "u " " " X" " "r$ r c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
TooManySymbolOrPunctuationPluginr Nc L d| _ d| _ d| _ d | _ d| _ d S )Nr F)_punctuation_count
_symbol_count_character_count_last_printable_char_frenzy_symbol_in_wordr( s r! __init__z)TooManySymbolOrPunctuationPlugin.__init__; s0 "# ! $(!&+###r$ r c * | S N isprintabler s r! r" z)TooManySymbolOrPunctuationPlugin.eligibleC $$&&&r$ c ( | xj dz
c_ || j k ro|t vrft | r| xj dz
c_ nF| du r0t
| r!t | du r| xj dz
c_ || _ d S )Nr F ) r; r< r r r9 isdigitr r
r: r s r! r& z%TooManySymbolOrPunctuationPlugin.feedF s "
2 2 2!=== i((
(''1,'''!!##u,, i(( - **e33""a'""$-!!!r$ c 0 d| _ d| _ d| _ d S Nr )r9 r; r: r( s r! r) z&TooManySymbolOrPunctuationPlugin.resetX s "# ! r$ c ^ | j dk rdS | j | j z | j z }|dk r|ndS )Nr g333333?)r; r9 r: )r ratio_of_punctuations r! r+ z&TooManySymbolOrPunctuationPlugin.ratio] sK A % %3
#d&8 8 ! " (<s'B'B##Kr$ r, r- r. r/ r> r1 r2 r" r& r) r3 r4 r+ r5 r$ r! r7 r7 : s , , , ,'# '$ ' ' ' '.c .d . . . .$
Lu L L L XL L Lr$ r7 c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
TooManyAccentuatedPluginr Nc " d| _ d| _ d S rH r; _accentuated_countr( s r! r> z!TooManyAccentuatedPlugin.__init__j ! "# r$ r c * | S r@ )isalphar s r! r" z!TooManyAccentuatedPlugin.eligiblen s """r$ c h | xj dz
c_ t | r| xj dz
c_ d S d S Nr )r; r rQ r s r! r& z TooManyAccentuatedPlugin.feedq sJ " ) $ $ ) # #q ( # # # # ) )r$ c " d| _ d| _ d S rH rP r( s r! r) z TooManyAccentuatedPlugin.resetw rR r$ c N | j dk rdS | j | j z }|dk r|ndS )Nr rJ gffffff?rP )r ratio_of_accentuations r! r+ z TooManyAccentuatedPlugin.ratio{ s@ A % %3 #d&; ; )>(E(E$$3Nr$ r, rL r5 r$ r! rN rN i s $ $ $ $## #$ # # # #)c )d ) ) ) ) $ $ $ $ Ou O O O XO O Or$ rN c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
UnprintablePluginr Nc " d| _ d| _ d S rH )_unprintable_countr; r( s r! r> zUnprintablePlugin.__init__ s "# ! r$ r c dS NTr5 r s r! r" zUnprintablePlugin.eligible tr$ c | du r,| du r|dk r| xj dz
c_ | xj dz
c_ d S )NFr )isspacerB r] r; r s r! r& zUnprintablePlugin.feed si 5 ( (%%''500V## # #q ( # # " r$ c d| _ d S rH )r] r( s r! r) zUnprintablePlugin.reset s "# r$ c @ | j dk rdS | j dz | j z S )Nr rJ )r; r] r( s r! r+ zUnprintablePlugin.ratio s+ A % %3'!+t/DDDr$ r, rL r5 r$ r! r[ r[ s " " " "# $ #c #d # # # #$ $ $ $ Eu E E E XE E Er$ r[ c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
SuspiciousDuplicateAccentPluginr Nc 0 d| _ d| _ d | _ d S rH _successive_countr; _last_latin_characterr( s r! r> z(SuspiciousDuplicateAccentPlugin.__init__ s !" ! %)"""r$ r c H | ot | S r@ )rT r r s r! r" z(SuspiciousDuplicateAccentPlugin.eligible s! "":x ':'::r$ c l | xj dz
c_ | j t | rt | j rr| r)| j r| xj dz
c_ t | t | j k r| xj dz
c_ || _ d S rV )r; rl r isupperrk r r s r! r& z$SuspiciousDuplicateAccentPlugin.feed s " & 2 y))
3 t 9::
3 ""
,t'A'I'I'K'K
,&&!+&& Y''=9S+T+TTT&&!+&&%."""r$ c 0 d| _ d| _ d | _ d S rH rj r( s r! r) z%SuspiciousDuplicateAccentPlugin.reset s !" ! %)"""r$ c @ | j dk rdS | j dz | j z S )Nr rJ rE )r; rk r( s r! r+ z%SuspiciousDuplicateAccentPlugin.ratio s+ A % %3&*d.CCCr$ r, rL r5 r$ r! rh rh s * * * * ;# ;$ ; ; ; ; /c /d / / / / * * * *
Du D D D XD D Dr$ rh c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
SuspiciousRanger Nc 0 d| _ d| _ d | _ d S rH )"_suspicious_successive_range_countr; _last_printable_seenr( s r! r> zSuspiciousRange.__init__ s 23/ ! $(!!!r$ r c * | S r@ rA r s r! r" zSuspiciousRange.eligible rC r$ c D | xj dz
c_ | st | s |t v r d | _ d S | j || _ d S t | j }t | }t
|| r| xj dz
c_ || _ d S rV )r; rc r r rv r is_suspiciously_successive_rangeru )r r unicode_range_aunicode_range_bs r! r& zSuspiciousRange.feed s "
i(( 888(,D % F $ ,(1D % F' %
( 22 +O_ M M 9 3 3q 8 3 3$-!!!r$ c 0 d| _ d| _ d | _ d S rH )r; ru rv r( s r! r) zSuspiciousRange.reset s ! 23/$(!!!r$ c T | j dk rdS | j dz | j z }|dk rdS |S )Nr rJ rE g?)r; ru )r ratio_of_suspicious_range_usages r! r+ zSuspiciousRange.ratio sH A % %3
3a 7 !+"' +S 0 03..r$ r, rL r5 r$ r! rs rs s ) ) ) )
'# '$ ' ' ' '.c .d . . . .2) ) ) )
/u / / / X / / /r$ rs c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
SuperWeirdWordPluginr Nc d| _ d| _ d| _ d| _ d| _ d| _ d| _ d| _ d| _ d S )Nr F ) _word_count_bad_word_count_foreign_long_count_is_current_word_bad_foreign_long_watchr; _bad_character_count_buffer_buffer_accent_countr( s r! r> z SuperWeirdWordPlugin.__init__ sO #$ $)!#( ! $%! $%!!!r$ r c dS r_ r5 r s r! r" z SuperWeirdWordPlugin.eligible r` r$ c | rd | j |g | _ t | r| xj dz
c_ | j du r|t
| du st | r\t | du rKt | du r:t | du r)t | du rt | du rd| _ d S | j sd S | s t | st | r"| j r| xj dz
c_ t! | j }| xj |z
c_ |dk re| j |z dk rd| _ t | j d r6| j d r| xj dz
c_ d| _ |dk r | j r| xj dz
c_ d| _ | j r9| xj dz
c_ | xj t! | j z
c_ d| _ d| _ d| _ d | _ d S |d
vr>| du r*t1 | r d| _ | xj |z
c_ d S d S d S d S ) Nr r FT g(\? r > _-<=>|~)rT joinr r r r r r r r r r rc r r r lenr; r ro r r r rF r )r r
buffer_lengths r! r& zSuperWeirdWordPlugin.feed s 77DL)#<==DL i((
/))Q.))(E11 i((E11^I5N5N19%%.. i((E11 **e33 **e33I&&%//+/( F| F " )#<#<" &@LY@W@W" &l" &
! - -M ! !] 2 ! ! !!,}<tCC04D- "$,r"233 5 R8H8P8P8R8R 5,,1,,04D- ""t'?"((A-((,0)(
2$$)$$))S ->->>)),1)',D $ DL()D % % % @ @ @!!##u,,)$$ - )-D % LLI %LLLL
A @,,,,r$ c v d| _ d| _ d| _ d| _ d| _ d| _ d| _ d| _ d S )Nr Fr )r r r r r r; r r r( s r! r) zSuperWeirdWordPlugin.resetD sG $)!#( ! $%!#$ r$ c P | j dk r
| j dk rdS | j | j z S )N
r rJ )r r r r; r( s r! r+ zSuperWeirdWordPlugin.ratioN s3 r ! !d&>!&C&C3(4+@@@r$ r, rL r5 r$ r! r r s & & & & # $ 4&c 4&d 4& 4& 4& 4&l% % % % Au A A A XA A Ar$ r c ^ e Zd ZdZd
dZdedefdZdeddfdZd
dZ e
de fd Z dS ) CjkInvalidStopPluginu
GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and
can be easily detected. Searching for the overuse of '丅' and '丄'.
r Nc " d| _ d| _ d S rH _wrong_stop_count_cjk_character_countr( s r! r> z CjkInvalidStopPlugin.__init__\ !" $%!!!r$ r c dS r_ r5 r s r! r" z CjkInvalidStopPlugin.eligible` r` r$ c t |dv r| xj dz
c_ d S t | r| xj dz
c_ d S d S )N> 丄 丅r )r r r r s r! r& zCjkInvalidStopPlugin.feedc sZ & & " "a ' " " F ) + % % * % % % % + +r$ c " d| _ d| _ d S rH r r( s r! r) zCjkInvalidStopPlugin.resetj r r$ c : | j dk rdS | j | j z S )N rJ )r r r( s r! r+ zCjkInvalidStopPlugin.ration s& $r ) )3%(AAAr$ r, )
r- r. r/ r0 r> r1 r2 r" r& r) r3 r4 r+ r5 r$ r! r r V s
& & & &# $ +c +d + + + +& & & & Bu B B B XB B Br$ r c Z e Zd Zd dZdedefdZdeddfdZd dZe de
fd Z dS )
ArchaicUpperLowerPluginr Nc h d| _ d| _ d| _ d| _ d| _ d | _ d| _ d S )NFr T)_buf _character_count_since_last_sep _successive_upper_lower_count#_successive_upper_lower_count_finalr; _last_alpha_seen_current_ascii_onlyr( s r! r> z ArchaicUpperLowerPlugin.__init__v s? /0,-.*340 ! $ #' r$ r c dS r_ r5 r s r! r" z ArchaicUpperLowerPlugin.eligible r` r$ c | ot | }|du }|r| j dk rt| j dk r4| du r | j du r| xj | j z
c_ d| _ d| _ d | _ d| _ | xj dz
c_ d| _ d S | j du rt | du rd| _ | j | r| j s-| rB| j r)| j du r| xj dz
c_ d| _ nd| _ nd| _ | xj dz
c_ | xj dz
c_ || _ d S )NFr @ r TrE )
rT r r rF r r r r r r; r
ro islower)r r is_concerned chunk_seps r! r& z ArchaicUpperLowerPlugin.feed s ((**J/? /J/J E) =AA4:: %%''500,5588688 23D .34D 0$(D ! DI ! !Q & ! !'+D $ F #t + +0C0Cu0L0L',D $ ,!!##
"(=(E(E(G(G
"!!##
"(,(=(E(E(G(G
" 9$$66!;66 %DII $DII ! " ,,1,, ) r$ c h d| _ d| _ d| _ d| _ d | _ d| _ d| _ d S )Nr FT)r; r r r r r r r( s r! r) z ArchaicUpperLowerPlugin.reset s? ! /0,-.*340 $ #' r$ c : | j dk rdS | j | j z S )Nr rJ )r; r r( s r! r+ z ArchaicUpperLowerPlugin.ratio s& A % %37$:OOOr$ r, rL r5 r$ r! r r u s ( ( ( (# $ (*c (*d (* (* (* (*T( ( ( ( Pu P P P XP P Pr$ r rz r{ r c | |dS | |k rdS d| v rd|v rdS d| v sd|v rdS d| v sd|v r
d| v sd|v rdS | d | d }}|D ]}|t v r ||v r dS | dv |dv }}|s|r
d | v sd |v rdS |r|rdS d
| v sd
|v rd | v sd |v rdS | d k s|d k rdS d | v s d |v s| d v r|d v rd
| v sd
|v rdS d| v sd|v rdS dS )za
Determine if two Unicode range seen next to each other can be considered as suspicious.
NTFLatin Emoticons Combining )HiraganaKatakanaCJKHangulz Basic Latin)r r PunctuationForms)splitr )rz r{ keywords_range_akeywords_range_belrange_a_jp_charsrange_b_jp_charss r! ry ry s /"9t/))u/!!g &@&@uo%%)G)Gu ?""g &@&@&&+ *H*Hu)8)>)> * * S!! '
0 0 0
! ! !55 "
33
' ,
E_$<$<u , u?""h/&A&A O # #u'?'?5 m + +-/O/O5 E_$<$<333 7 7 7 O + +}/O/O5 o % %O)C)C5 4r$ i )maxsize皙?Fdecoded_sequencemaximum_thresholddebugc d t D }t | dz }d}|dk rd}n |dk rd}nd}t | d z t | D ]m\ }}|D ],} | | r| | -|d
k r ||z d
k s ||dz
k r!t d |D }||k r nn|r |D ] }
t |
j |
j
t |d S )
zw
Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.
c " g | ] } |
S r5 r5 ).0md_classs r!
<listcomp>z mess_ratio.<locals>.<listcomp> s+
r$ r rJ i i r
r c 3 $ K | ] }|j V d S r@ )r+ )r dts r! <genexpr>z mess_ratio.<locals>.<genexpr>& s$ !?!?r"(!?!?!?!?!?!?r$ ) r __subclasses__r zipranger" r& sumprint __class__r+ round) r r r detectorslengthmean_mess_ratio!intermediary_mean_mess_ratio_calcr indexdetectorr s r!
mess_ratior sh #5#D#D#F#F I !
"
"Q
&FO
||,.)) 4 ,.)),/) 04 7v G G 5! ) )H ++
)
i(((
AII%"CCqHH
fqj
!!?!?Y!?!?!? ? ?O "333 * * *B ", ) ) ) ) ! $ $$r$ N)r F)& functoolsr typingr r constantr r utilsr r
r r r
r r r r r r r r r r r r7 rN r[ rh rs r r r r1 r2 ry r4 r r5 r$ r! <module>r s[ ! ! ! ! ! ! ! ! S S S S S S S S & " " " " " " " "D,L ,L ,L ,L ,L'9 ,L ,L ,L^O O O O O 1 O O O8E E E E E* E E E8"D "D "D "D "D&8 "D "D "DJ3/ 3/ 3/ 3/ 3/( 3/ 3/ 3/lWA WA WA WA WA- WA WA WAt B B B B B- B B B>IP IP IP IP IP 0 IP IP IPXC c]C5=c]C C C C CL 4IN'% '%'%.3'%BF'%
'% '% '% '% '% '%r$