R6W5 - @ s d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z y d d l
m Z Wn e k
r d Z Yn Xy d d l
m
Z
Wn" e k
r d d l m
Z
Yn Xy d d l m Z Wn e k
rd Z Yn Xd d l m Z d d l m Z y d d l m Z Wn> e k
rd d l m Z d d l m Z d
d Z Yn Xy d d l m Z Wn e k
rd Z Yn Xd
Z d Z d Z d Z d Z d Z i Z! i Z" i Z# Gd d d e$ Z% e% Z& e% Z' d d Z( d d Z) d d Z* d d Z+ d d Z, d d Z- d! d" Z. e* e d d# d$ Z/ e* e d% d& Z0 e* e d' d( d) Z1 e* e d* d+ Z2 e* e d, d- Z3 e j4 d d. k r,e* e d/ d0 d1 Z5 e* e d2 d3 Z6 n3 e* e d/ d4 d1 Z5 e* e d5 d3 Z6 e* e d6 d7 Z7 e* e d d8 d9 Z8 e* e d: d; Z9 e* e d< d= Z: e* e d> d? Z; e* e d d d@ dA Z< e* e dB dC Z= e* e dD dE Z> e* e dF dG Z? e* e dH dI Z@ e* e dJ dK ZA dL dM ZB e* e dN dO ZC e* e dP dQ ZD e* e dR dS ZE e* e dT dU ZF e* e dV dW ZG e* e dX dY ZH GdZ d[ d[ eI ZJ Gd\ d] d] eJ ZK e j4 d d. du k r`d_ d` ZL n da db ZL e( e Gdc dd dd eK ZM e( e Gde df df eK ZN e( e Gdg dh dh eI ZO e( e Gdi dj dj eI ZP e( e Gdk dl dl eJ ZQ e( e Gdm dn dn eJ ZR e( e Gdo dp dp eI ZS e d k re* e e jT ZT e* e e jU ZU e* e e jV ZV e( e e jW ZW e d k r
e) e Gdq dr dr e ZX e) e Gds dt dt e ZY d S)v N)Counter)urlparse)
TableFunction)binary_construct) unicode_type)format_date_time_sqlite)format_date_time)SQLITE_DATETIME_FORMATSc C s
t | t S)N)r r )Z
date_value r
/sqlite_udf.pyr ! s r ) _sqlite_udfZ control_flowdatefileZhelpersmathstringc sX e Z d Z f d d Z f d d Z f d d Z f d d Z S) synchronized_dictc s, t t | j | | t j | _ d S)N)superr __init__ threadingLock_lock)selfargskwargs) __class__r
r r 8 s zsynchronized_dict.__init__c
s+ | j t t | j | SWd QRXd S)N)r r r __getitem__)r key)r r
r r < s
z synchronized_dict.__getitem__c
s. | j t t | j | | SWd QRXd S)N)r r r __setitem__)r r value)r r
r r @ s
z synchronized_dict.__setitem__c
s+ | j t t | j | SWd QRXd S)N)r r r __delitem__)r r )r r
r r D s
z synchronized_dict.__delitem__)__name__
__module__ __qualname__r r r r r
r
)r r r 7 s r c s f d d } | S)Nc s6 x/ D]' } t j | g t | j | q W| S)N)AGGREGATE_COLLECTION
setdefaultappend)klassgroup)groupsr
r decoratorN s
z aggregate.<locals>.decoratorr
)r( r) r
)r( r aggregateM s r* c s f d d } | S)Nc s6 x/ D]' } t j | g t | j | q W| S)N)TABLE_FUNCTION_COLLECTIONr$ r% )r& r' )r( r
r r) V s
z!table_function.<locals>.decoratorr
)r( r) r
)r( r table_functionU s r, c s f d d } | S)Nc s6 x/ D]' } t j | g t | j | q W| S)N)UDF_COLLECTIONr$ r% )fnr' )r( r
r r) ^ s
zudf.<locals>.decoratorr
)r( r) r
)r( r udf] s r/ c G sz t } xj | D]b } t | } xO | D]G } t | d | j } | | k r' | j | | j | d | q' Wq Wd S)Nname )setr# getattrr addZcreate_aggregate)connr( seenr' klassesr& r0 r
r
r register_aggregate_groupsf s
r9 c G se t } xU | D]M } t | } x: | D]2 } | j | k r' | j | j | j | q' Wq Wd S)N)r3 r+ r0 r5 register)r6 r( r7 r' r8 r& r
r
r register_table_function_groupsp s
r; c G sn t } x^ | D]V } t | } xC | D]; } | j } | | k r' | j | | j | d | q' Wq Wd S)Nr1 r2 )r3 r- r r5 Zcreate_function)r6 r( r7 r' Z functionsfunctionr0 r
r
r register_udf_groupsy s
r= c C s+ t | t t | t t | t d S)N)r9 r# r; r+ r= r- )r6 r
r
r register_all s
r> c C s | r
| S| S)Nr
)ZcondZtruthyZfalseyr
r
r if_then_else s r? c C sh | j d d } | j d } | d k r; | d | S| j d } | d k rd | d | S| S)NT +r1 -
r2 )replacefind)Zdate_strZtz_idx1Ztz_idx2r
r
r strip_tz s rG z, c C s d } g } xa | D]Y \ } } t | | \ } } | r | d
k rL d pO d } | j d | | | f q W| sz d S| j | S) NQ im year month weekday hour< minuter1 seconds z%s %s%sz 0 seconds3rV rI ' rX rK : rZ rM rH rN rO rP rQ rR r1 rS )rW rY r[ r\ r] r^ r_ )divmodr% join)ZnsecondsZgluepartsZaccumoffsetr0 valsuffixr
r
r human_delta s rf c C s7 y t j j | } Wn t k
r. d SYn X| d S)Nr1 )ospathsplitext
ValueError)filenameresr
r
r file_ext s
rm c C s3 y$ t | } | j SWd QRXWn Yn Xd S)N)openread)rk Zfhr
r
r file_read s
rp c C s t t j | | S)N)r zlibcompress)data compressionr
r
r gzip s rw c C s
t j | S)N)rs
decompress)ru r
r
r gunzip s ry c C s t j t | | S)N)rs rt r )ru rv r
r
r rw s c C s t j | j d S)Nzutf-8)rs rx decode)ru r
r
r ry s c C s t | } | r | j Sd S)N)r Znetloc)ZurlZ parse_resultr
r
r hostname s r{ c C sA | j } | d k r% | t | <n t j | t | <} | S)N)lowerSTATEget)r Zonr
r
r toggle s
r c G s> | s t j | St | d k r6 | d t | <n d Sd S)Nr1 r F)SETTINGSr~ len)r r r
r
r setting s
r c C s t j d S)N)r clearr
r
r
r clear_settings s r c C s t j d S)N)r} r r
r
r
r
clear_toggles s r c C sA | d k r d | } } n | d k r. d } t j | | | S)Nr r1 )random randrange)startendstepr
r
r randomrange s
r c C s1 y t j | | SWn t k
r, d SYn Xd S)N)r gaussrj )Zmeansigmar
r
r gauss_distribution s
r c C s. y t j | SWn t k
r) d SYn Xd S)N)r sqrtrj )nr
r
r r s
r c C sD y t | SWn/ t k
r? y t | SWn d SYn XYn Xd S)N)intrj float)rT r
r
r tonumber s
r c C s | s | r d S| j | S)Nr )count)haystackZneedler
r
r substr_count s r c C s t | j | S)N)r strip)r charsr
r
r strip_chars s r c G s1 | } x | D] } | j | q W| j S)N)update hexdigest) constructorr Zhash_objargr
r
r _hash s
r c G s
t t j S)N)r hashlibmd5)valsr
r
r r # s r c G s
t t j S)N)r r sha1)r r
r
r r ' s r c G s
t t j S)N)r r sha256)r r
r
r r + s r c G s
t t j S)N)r r sha512)r r
r
r r / s r c C s
t j | S)N)rs adler32)rT r
r
r r 3 s r c C s
t j | S)N)rs crc32)rT r
r
r r 7 s r c @ s4 e Z d Z d d Z d d Z d d Z d S) _heap_aggc C s g | _ d | _ d S)Nr )heapct)r r
r
r r = s z_heap_agg.__init__c C s | S)Nr
)r r r
r
r processA s z_heap_agg.processc C s/ | j d 7_ t j | j | j | d S)Nr1 )r heapqheappushr r )r r r
r
r r D s z_heap_agg.stepN)r r! r" r r r r
r
r
r r < s r c @ s e Z d Z d d Z d S)_datetime_heap_aggc C s
t | S)N)r )r r r
r
r r I s z_datetime_heap_agg.processN)r r! r" r r
r
r
r r H s r c C s | j | j d | j d S)NiQ g $@r g .A)secondsZdaysZ microseconds)tdr
r
r
total_secondsM s r c C s
| j S)N)r )r r
r
r <lambda>R s r c @ s e Z d Z d d Z d S)mintdiffc C s d } } xz | j r | d k rC | d k rC t j | j } q
t j | j } | | } | d k sw | | k r} | } | } q
W| d k r t | Sd S)N)r r heappopr )r dtpmin_diffdtdiffr
r
r finalizeV s
zmintdiff.finalizeN)r r! r" r r
r
r
r r T s r c @ s e Z d Z d d Z d S)avgtdiffc C s | j d k r d S| j d k r& d Sd } } d } xv | j r | d k ro | d k ro t j | j } q9 t j | j } | | } | d 7} | t | 7} | } q9 Wt | | S)Nr1 r )r r r r r r )r totalr r r r r
r
r r g s"
zavgtdiff.finalizeN)r r! r" r r
r
r
r r e s r c @ s4 e Z d Z d d Z d d Z d d Z d S)durationc C s d | _ | _ d S)N)_min_max)r r
r
r r s zduration.__init__c C s^ t | } | j d k s* | | j k r3 | | _ | j d k sQ | | j k rZ | | _ d S)N)r r r )r r r r
r
r r s
z
duration.stepc C s0 | j r, | j r, | j | j } t | Sd S)N)r r r )r r r
r
r r s
zduration.finalizeN)r r! r" r r r r
r
r
r r } s r c @ sa e Z d Z e r9 d d Z d d Z d d Z n$ d d Z d d Z d d Z d
S) modec C s t | _ d S)N)r items)r r
r
r r s z
mode.__init__c G s | j j | d S)N)r r )r r r
r
r r s z mode.stepc C s% | j r! | j j d d d Sd S)Nr1 r )r most_common)r r
r
r r s z
mode.finalizec C s
g | _ d S)N)r )r r
r
r r s c C s | j j | d S)N)r r% )r itemr
r
r r s c C s, | j r( t t | j d | j j Sd S)Nr )r maxr3 r )r r
r
r r s N)r r! r" r r r r r
r
r
r r s r c @ s e Z d Z d d Z d S)minrangec C s | j d k r d S| j d k r& d Sd } } xz | j r | d k ri | d k ri t j | j } q3 t j | j } | | } | d k s | | k r | } | } q3 W| S)Nr r1 )r r r r )r prevr currr r
r
r r s
zminrange.finalizeN)r r! r" r r
r
r
r r s r c @ s e Z d Z d d Z d S)avgrangec C s | j d k r d S| j d k r& d Sd } } d } xp | j r | d k ro | d k ro t j | j } q9 t j | j } | | } | d 7} | | 7} | } q9 Wt | | S)Nr r1 )r r r r r )r r r r r r r
r
r r s"
zavgrange.finalizeN)r r! r" r r
r
r
r r s r c @ s: e Z d Z d Z d d Z d d Z d d Z d S) _rangerangec C s d | _ | _ d S)N)r r )r r
r
r r s z_range.__init__c C sR | j d k s | | j k r' | | _ | j d k sE | | j k rN | | _ d S)N)r r )r r r
r
r r s z _range.stepc C s0 | j d k r, | j d k r, | j | j Sd S)N)r r )r r
r
r r s z_range.finalizeN)r r! r" r0 r r r r
r
r
r r s r c @ sI e Z d Z d d g Z d g Z d Z d d d d Z d d Z d S)
RegexSearchregex
search_stringmatchZ regex_searchNc C s t j | | | _ d S)N)refinditer_iter)r r r r
r
r
initialize s zRegexSearch.initializec C s t | j j d f S)Nr )nextr r' )r idxr
r
r iterate s zRegexSearch.iterate)r r! r" paramscolumnsr0 r r r
r
r
r r s
r c @ sI e Z d Z d d d g Z d g Z d Z d d d Z d d
Z d S)
DateSeriesr stop step_secondsr
Z date_seriesiQ c C s t | | _ t | | _ t | } t j d | | _ | j j d k r | j j d k r | j j d k r | d k r d | _
n | j j d k r| j j d k r| j j
d k r| j j d k r| j j d k r| j j
d k r| d k rd | _
n d | _
d S) Nr r iQ z%Y-%m-%dil r1 z%H:%M:%Sz%Y-%m-%d %H:%M:%S)r r r r datetimeZ timedeltar rP rR rS formatrI rK rN )r r r r r
r
r r s$ zDateSeries.initializec C sF | j | j k r t | j } | j | j 7_ | j | j f S)N)r r
StopIterationr strftimer )r r currentr
r
r r s
zDateSeries.iterateN)r r! r" r r r0 r r r
r
r
r r s
r )rq r )Zr r r r rg r r sysr rs collectionsr ImportErrorr Z urllib.parseZvtfuncr Zpeeweer r Zplayhouse._speedupsr r r Z playhouser Z
cython_udfZ CONTROL_FLOWZDATEFILEZHELPERZMATHSTRINGr# r+ r- dictr r} r r* r, r/ r9 r; r= r> r? rG rf rm rp version_inforw ry r{ r r r r r r r r r r r r r r r r r objectr r r r r r r r r r Zdamerau_levenshtein_distZlevenshtein_distZstr_distZmedianr r r
r
r
r <module> s