Pc @ sW d d l Z d d l m Z d d l m Z d d l Z d d g Z y d d l m Z Wn! e
k
r{ d d l m Z n Xy
e Z
Wn e k
r e Z
n Xy e Wn e k
r e Z n Xd Z e d Z d Z d
Z d Z d Z d
Z d Z d Z d Z e d Z d Z d dk d YZ d dl d YZ d e f d YZ! d Z" d Z# d Z$ d Z% d Z& d Z' d e
f d YZ( d! e( f d" YZ) d# e( f d$ YZ* e+ d% Z, e+ d& Z- e j. d' e j/ e j0 B Z1 e j. d( e j/ e j0 B Z2 e j. d) e j/ e j0 B Z3 d* Z4 e j. d+ Z5 d, Z6 dm Z7 dn Z8 do Z9 e dY Z: dZ Z; e j. d[ Z< d\ Z= d] Z> d^ Z? d_ Z@ d` ZA da ZB e db ZC dc ZD dd ZE de ZF df ZG dg e jH f dh YZI eJ di k rSd dj l mK ZK eK jL n d S(p iN( t etree( t fragment_fromstringt
html_annotatet htmldiff( t escapec C s d t t | d | f S( Ns <span title="%s">%s</span>i ( t html_escapet _unicode( t textt version( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyt default_markup s c C s g | D] \ } } t | | ^ q } | d } x% | d D] } t | | | } q= Wt | } t | | } d j | j S( s
doclist should be ordered from oldest to newest, like::
>>> version1 = 'Hello World'
>>> version2 = 'Goodbye World'
>>> print(html_annotate([(version1, 'version 1'),
... (version2, 'version 2')]))
<span title="version 2">Goodbye</span> <span title="version 1">World</span>
The documents must be *fragments* (str/UTF8 or unicode), not
complete documents
The markup argument is a function to markup the spans of words.
This function is called like markup('Hello', 'version 2'), and
returns HTML. The first argument is text and never includes any
markup. The default uses a span with a title:
>>> print(default_markup('Some Text', 'by Joe'))
<span title="by Joe">Some Text</span>
i i t ( t tokenize_annotatedt html_annotate_merge_annotationst compress_tokenst markup_serialize_tokenst joint strip( t doclistt markupt docR t tokenlistt
cur_tokenst tokenst result( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s %
c C s0 t | d t } x | D] } | | _ q W| S( sF Tokenize a document and add an annotation attribute to each token
t
include_hrefs( t tokenizet Falset
annotation( R R R t tok( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR G s
c C s{ t d | d | } | j } xS | D]K \ } } } } } | d k r( | | | !} | | | !}
t | |
q( q( Wd S( s Merge the annotations from tokens_old into tokens_new, when the
tokens in the new document already existed in the old document.
t at bt equalN( t InsensitiveSequenceMatchert get_opcodest copy_annotations( t
tokens_oldt
tokens_newt st commandst commandt i1t i2t j1t j2t eq_oldt eq_new( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR O s
c C sN t | t | k s t x) t | | D] \ } } | j | _ q. Wd S( sN
Copy annotations from the tokens listed in src to the tokens in dest
N( t lent AssertionErrort zipR ( t srct destt src_tokt dest_tok( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR" \ s c C sq | d g } x] | d D]Q } | d j r\ | j r\ | d j | j k r\ t | | q | j | q W| S( sm
Combine adjacent tokens when there is no HTML between the tokens,
and they share an annotation
i i i( t post_tagst pre_tagsR t compress_merge_backt append( R R R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR
d s
c C s | d } t | t k s. t | t k r> | j | ni t | } | j r` | d 7} n | | 7} t | d | j d | j d | j } | j | _ | | d <d S( sY Merge tok into the last element of tokens (modifying the list of
tokens in-place). it R6 R5 t trailing_whitespaceN( t typet tokenR8 R R: R6 R5 R ( R R t lastR t merged( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR7 s s
$
c c s xy | D]q } x | j D] } | Vq W| j } | | | j } | j rZ | d 7} n | Vx | j D] } | Vqi Wq Wd S( sz
Serialize the list of tokens into a list of text chunks, calling
markup_func around text to add annotations.
R9 N( R6 t htmlR R: R5 ( R t markup_funcR< t preR? t post( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s
c C sF t | } t | } t | | } d j | j } t | S( s Do a diff of the old and new document. The documents are HTML
*fragments* (str/UTF8 or unicode), they are not complete documents
(i.e., no <html> tag).
Returns HTML with <ins> and <del> tags added around the
appropriate text.
Markup is generally ignored, with the markup from new_html
preserved, and possibly some markup from old_html (though it is
considered acceptable to lose some of the old markup). Only the
words in the HTML are diffed. The exception is <img> tags, which
are treated like words, and the href attribute of <a> tags, which
are noted inside the tag itself when there are changes.
R
( R t htmldiff_tokensR R t fixup_ins_del_tags( t old_htmlt new_htmlt old_html_tokenst new_html_tokensR ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s
c C s t d | d | } | j } g } x | D] \ } } } } } | d k ru | j t | | | !d t q. n | d k s | d k r t | | | ! }
t |
| n | d k s | d k r. t | | | ! } t | | q. q. Wt | } | S( s] Does a diff on the tokens themselves, returning a list of text
chunks (not tokens).
R R R t insertt replacet delete( R R! t extendt
expand_tokenst Truet merge_insertt merge_deletet cleanup_delete( t html1_tokenst html2_tokensR% R& R R' R( R) R* R+ t
ins_tokenst
del_tokens( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRC s c c s xz | D]r } x | j D] } | Vq W| s7 | j r` | j rR | j d Vq` | j Vn x | j D] } | Vqj Wq Wd S( se Given a list of tokens, return a generator of the chunks of
text for the data in the tokens.
R9 N( R6 t hide_when_equalR: R? R5 ( R R R< RA RB ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRM s
c C s t | \ } } } | j | | rO | d j d rO | d c d 7<n | j d | r | d j d r | d d | d <n | j | | j d | j | d S( s| doc is the already-handled document (as a list of text chunks);
here we add <ins>ins_chunks</ins> to the end of that. iR9 s <ins>s </ins> N( t split_unbalancedRL t endswithR8 ( t
ins_chunksR t unbalanced_startt balancedt unbalanced_end( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRO s
t DEL_STARTc B s e Z RS( ( t __name__t
__module__( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR] s t DEL_ENDc B s e Z RS( ( R^ R_ ( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR` s t NoDeletesc B s e Z d Z RS( sY Raised when the document no longer contains any pending deletes
(DEL_START/DEL_END) ( R^ R_ t __doc__( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRa s c C s+ | j t | j | | j t d S( s Adds the text chunks in del_chunks to the document doc (another
list of text chunks) with marker to show it is a delete.
cleanup_delete later resolves these markers into <del> tags.N( R8 R] RL R` ( t
del_chunksR ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRP s
c C s xy t | \ } } } Wn t k
r0 Pn Xt | \ } } } t | | | t | | | | } | r | d j d r | d c d 7<n | j d | r | d j d r | d d | d <n | j | | j d | j | | } q | S( s Cleans up any DEL_START/DEL_END markers in the document, replacing
them with <del></del>. To do this while keeping the document
valid, it may need to drop some tags (either start or end tags).
It may also move the del into adjacent tags to try to move it to a
similar location where it was originally located (e.g., moving a
delete into preceding <div> tag, if the del looks like (DEL_START,
'Text</div>', DEL_END)iR9 s <del>s </del> ( t split_deleteRa RW t locate_unbalanced_startt locate_unbalanced_endRX R8 RL ( t chunkst
pre_deleteRK t post_deleteRZ R[ R\ R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRQ s&
c
C s g } g } g } g } xE| D]=} | j d sG | j | q n | d d k } | j d j d } | t k r | j | q n | r3| r | d d | k r | j | | j \ } } } | | | <q\| r#| j g | D] \ } } } | ^ q g } | j | q\| j | q | j | t | | f | j d q W| j g | D] \ } } } | ^ qm g | D] } | d k r| ^ q} | | | f S( s] Return (unbalanced_start, balanced, unbalanced_end), where each is
a list of text and tag chunks.
unbalanced_start is a list of all the tags that are opened, but
not closed in this span. Similarly, unbalanced_end is a list of
tags that are closed but were not opened. Extracting these might
mean some reordering of the chunks.t <i t /i s <>/iN( t
startswithR8 t splitR t
empty_tagst popRL R. t None(
Rg t startt endt tag_stackR[ t chunkt endtagt namet post tag( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRW 5 s:
) #%c C s` y | j t } Wn t k
r, t n X| j t } | | | | d | !| | d f S( s Returns (stuff_before_DEL_START, stuff_inside_DEL_START_END,
stuff_after_DEL_END). Returns the first case found (there may be
more DEL_STARTs in stuff_after_DEL_END). Raises NoDeletes if
there's no DEL_START found. i ( t indexR] t
ValueErrorRa R` ( Rg Rw t pos2( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRd ] s
c C s x | s
Pn | d } | j d j d } | s: Pn | d } | t k s` | j d rd Pn | d d k rx Pn | j d j d } | d k r Pn | d k s t d | | | k r | j d | j | j d q Pq d S(
s pre_delete and post_delete implicitly point to a place in the
document (where the two were split). This moves that point (by
popping items from one and pushing them onto the other). It moves
the point to try to find a place where unbalanced_start applies.
As an example::
>>> unbalanced_start = ['<div>']
>>> doc = ['<p>', 'Text', '</p>', '<div>', 'More Text', '</div>']
>>> pre, post = doc[:3], doc[3:]
>>> pre, post
(['<p>', 'Text', '</p>'], ['<div>', 'More Text', '</div>'])
>>> locate_unbalanced_start(unbalanced_start, pre, post)
>>> pre, post
(['<p>', 'Text', '</p>', '<div>'], ['More Text', '</div>'])
As you can see, we moved the point so that the dangling <div> that
we found will be effectively replaced by the div in the original
document. If this doesn't work out, we just throw away
unbalanced_start without doing anything.
i s <>Rj i Rk t inst dels Unexpected delete tag: %rN( Rm R R] Rl R/ Ro R8 ( RZ Rh Ri t findingt finding_namet nextRv ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRe i s*
c C s x | s
Pn | d } | j d j d } | s: Pn | d } | t k s` | j d rd Pn | j d j d } | d k s | d k r Pn | | k r | j | j d | j q Pq d S( st like locate_unbalanced_start, except handling end tags and
possibly moving the point earlier in the document. ii s <>/s </R| R} N( Rm R R` Rl Ro RI ( R\ Rh Ri R~ R R Rv ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRf s"
R< c B s8 e Z d Z e Z d d e d Z d Z d Z RS( s8 Represents a diffable token, generally a word that is displayed to
the user. Opening tags are attached to this token when they are
adjacent (pre_tags) and closing tags that follow the word
(post_tags). Some exceptions occur when there are empty tags
adjacent to a word, so there may be close tags in pre_tags, or
open tags in post_tags.
We also keep track of whether the word was originally followed by
whitespace, even though we do not want to treat the word as
equivalent to a similar word that does not have a trailing
space.c C sa t j | | } | d k r* | | _ n g | _ | d k rK | | _ n g | _ | | _ | S( N( R t __new__Rp R6 R5 R: ( t clsR R6 R5 R: t obj( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s d t j | | j | j f S( Ns token(%s, %r, %r)( R t __repr__R6 R5 ( t self( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s
t | S( N( R ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR? s N( R^ R_ Rb R RV Rp R R R? ( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR< s
t tag_tokenc B s2 e Z d Z d d e d Z d Z d Z RS( s Represents a token that is actually a tag. Currently this is just
the <img> tag, which takes up visible space just like a word but
is only represented in a document by a tag. c C sM t j | d t | f d | d | d | } | | _ | | _ | | _ | S( Ns %s: %sR6 R5 R: ( R< R R; Rx t datat html_repr( R Rx R R R6 R5 R: R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s, d | j | j | j | j | j | j f S( NsR tag_token(%s, %s, html_repr=%s, post_tags=%r, pre_tags=%r, trailing_whitespace=%s)( Rx R R R6 R5 R: ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s | j S( N( R ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR? s N( R^ R_ Rb Rp R R R R? ( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s
t
href_tokenc B s e Z d Z e Z d Z RS( sh Represents the href in an anchor tag. Unlike other words, we only
show the href when it changes. c C s d | S( Ns Link: %s( ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR? s ( R^ R_ Rb RN RV R? ( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C sL t j | r | } n t | d t } t | d t d | } t | S( sk
Parse the given HTML and returns token objects (words with attached tags).
This parses only the content of a page; anything in the head is
ignored, and the <head> and <body> elements are themselves
optional. The content is then parsed by lxml, which ensures the
validity of the resulting parsed document (though lxml may make
incorrect guesses when the markup is particular bad).
<ins> and <del> tags are also eliminated from the document, as
that gets confusing.
If include_hrefs is true, then the href attribute of <a> tags is
included as a special kind of diffable token.t cleanupt skip_tagR ( R t iselementt
parse_htmlRN t
flatten_elt fixup_chunks( R? R t body_elRg ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s
c C s% | r t | } n t | d t S( s
Parses an HTML fragment, returning an lxml element. Note that the HTML will be
wrapped in a <div> tag that was not in the original document.
If cleanup is true, make sure there's no <head> or <body>, and get
rid of any <ins> and <del> tags.
t
create_parent( t cleanup_htmlR RN ( R? R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s s <body.*?>s
</body.*?>s </?(ins|del).*?>c C sf t j | } | r( | | j } n t j | } | rP | | j } n t j d | } | S( s This 'cleans' the HTML, meaning that any page structure is removed
(only the contents of <body> are used, if there is any <body).
Also <ins> and <del> tags are removed. R
( t _body_ret searchRr t _end_body_reRq t _ins_del_ret sub( R? t match( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR , s s
[ \t\n\r]$c
C s g } d
} g } x| D]} t | t r | d d k r | d } | d } | j d rt | d } t } n t } t d | d | d | d | } g } | j | q | d d
k r | d } t | d | d t } g } | j | q q n t | ra| j d r-| d } t } n t } t
| d | d | } g } | j | q t | r}| j | q t | r| r| j | q| st
d | | | | f | j j | q d s t
q W| st
d d | g S| d j j | | S( sM
This function takes a list of chunks and produces a list of tokens.
i t imgi i R9 iR R6 R: t hrefs4 Weird state, cur_word=%r, result=%r, chunks=%r of %rR
N( Rp t
isinstancet tupleRX RN R R R8 R t is_wordR< t is_start_tagt
is_end_tagR/ R5 RL ( Rg t tag_accumt cur_wordR Rt R1 Rx R: R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR < sZ
t paramR t areat brt basefontt inputt baset metat linkt colt addresst
blockquotet centert dirt divt dlt fieldsett formt h1t h2t h3t h4t h5t h6t hrt isindext menut noframest noscriptt olt pRA t tablet ult ddt dtt framesett lit tbodyt tdt tfoott tht theadt trc c sS | sC | j d k r5 d | j d t | f VqC t | Vn | j t k rw | j rw t | rw | j rw d St | j } x | D] } t | Vq Wx0 | D]( } x t | d | D] } | Vq Wq W| j d k r| j d r| rd | j d f Vn | sOt
| Vt | j } x | D] } t | Vq7Wn d S( s Takes an lxml element el, and generates all the text chunks for
that tag. Each start tag is a chunk, each word is a chunk, and each
end tag is a chunk.
If skip_tag is true, then the outermost container tag is
not returned (just its contents).R R1 NR R R ( Rx t gett start_tagRn R R. t tailt split_wordsR R t end_tag( t elR R t start_wordst wordt childt itemt end_words( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s& 0
$
c C si | s | j r g Sg | j j D] } | d ^ q+ } t j | se | d d | d <n | S( sk Splits some text into words. Includes trailing whitespace (one
space) on each word when appropriate. R9 i( R Rm t end_whitespace_reR ( R t wt words( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s )s
^[ \t\n\r]c
C sO d | j d j g | j j D]% \ } } d | t | t f ^ q f S( s=
The text representation of the start tag for a tag.
s <%s%s>R
s %s="%s"( Rx R t attribt itemsR RN ( R Rv t value( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s; | j r$ t j | j r$ d } n d } d | j | f S( sg The text representation of an end tag for a tag. Includes
trailing whitespace when appropriate. R9 R
s </%s>%s( R t start_whitespace_reR Rx ( R t extra( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s | j d S( NRj ( Rl ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s
| j d S( Ns </( Rl ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s | j d o | j d S( NRj s </( Rl ( R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C s2 t | d t } t | t | d t } | S( s Given an html string, move any <ins> or <del> tags inside of any
block-level elements, e.g. transform <ins><p>word</p></ins> to
<p><ins>word</ins></p> R t
skip_outer( R R t _fixup_ins_del_tagst serialize_html_fragmentRN ( R? R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyRD s
c C s} t | t s t d | t j | d d d t } | ru | | j d d } | | j d } | j S| Sd S( s Serialize a single lxml element as HTML. The serialized form
includes the elements tail.
If skip_outer is true, then don't serialize the outermost tag
s3 You should pass in an element, not a string like %rt methodR? t encodingt >i Rj N( R t
basestringR/ R t tostringR t findt rfindR ( R R R? ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s
c C se x^ d d g D]P } xG | j d | D]2 } t | s? q' n t | d | | j q' Wq
Wd S( s? fixup_ins_del_tags that works on an lxml document in-place
R| R} s descendant-or-self::%sRx N( t xpatht _contains_block_level_tagt _move_el_inside_blockt drop_tag( R Rx R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s c C sG | j t k s | j t k r" t Sx | D] } t | r) t Sq) Wt S( sP True if the element contains any block-level elements, like <p>, <td>, etc.
( Rx t block_level_tagst block_level_container_tagsRN R R ( R R ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s
c C sb xo | D] } t | r Pq q Wd d l } t j | } | j | _ d | _ | j t | | g | (d Sx t | D] } t | r t | | | j rt j | } | j | _ d | _ | j
| j | d | qq t j | } | j | | | j
| q W| j r^t j | } | j | _ d | _ | j
d | n d S( st helper for _fixup_ins_del_tags; actually takes the <ins> etc tags
and moves them inside any block-level tags. iNi i ( R t sysR t ElementR Rp RL t listR R RI Ry RJ R8 ( R Rx R R t children_tagt tail_tagt child_tagt text_tag( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR s4
# c C s& | j } | j p d } | j ry t | s@ | | j 7} qy | d j rf | d j | j 7_ qy | j | d _ n | j | } | r | d k r d } n | | d } | d k r | j r | j | 7_ q| | _ q | j r | j | 7_ q | | _ n | j | | | d +d S( s
Removes an element, but merges its contents into its place, e.g.,
given <p>Hi <i>there!</i></p>, if you remove the <i> element you get
<p>Hi there!</p>
R
ii i N( t getparentR R R. Ry Rp t getchildren( R t parentR Ry t previous( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyt _merge_element_contents7 s*
R c B s e Z d Z d Z d Z RS( st
Acts like SequenceMatcher, but tries not to find very small equal
blocks amidst large spans of changes
i c C s{ t t | j t | j } t | j | d } t j j | } g | D]' } | d | k sq | d rP | ^ qP S( Ni i ( t minR. R t thresholdt difflibt SequenceMatchert get_matching_blocks( R t sizeR t actualR ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR a s !
( R^ R_ Rb R R ( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyR Y s t __main__( t _diffcommand( ( (
R s imgR R R s inputs baseR s linkR ( R R s centers dirR R R R R R R R R R R R R R R R R s pres tableR (
R R R R R R R R R R (M R t lxmlR t lxml.htmlR t ret __all__R? R R t ImportErrort cgit unicodeR t NameErrort strR R R R R R" R
R7 R R RC R RM RO R] R` t ExceptionRa RP RQ RW Rd Re Rf R< R R RN R R t compilet It SR R R R R R Rn R R R R R R R R R R RD R R R R R R R R^ R t main( ( ( s4 /usr/lib64/python2.7/site-packages/lxml/html/diff.pyt <module> s
(
& ' ( 2 (
<
! "