관리-도구
편집 파일: _tokenizer.cpython-310.pyc
o ƚRe�, � @ s� d dl mZmZmZ d dlmZ d dlmZm Z d dl m Z ddl m Z ddl mZ ddl mZmZ dd l mZmZmZ dd l mZmZ dd l mZ dd lmZ dd lmZ ee�Ze dkrae Z ne Z G dd� de �Z dS )� )�absolute_import�division�unicode_literals)�unichr)�deque� OrderedDict)� version_info� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)� tokenTypes� tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Trie)� � c sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d d �Zd d � Zdd� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z d<d=� Z d>d?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Z<dvdw� Z=dxdy� Z>dzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )�� HTMLTokenizera This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. Nc sJ t |fi |��| _|| _d| _g | _| j| _d| _d | _t t | �� � d S �NF) r �stream�parser� escapeFlag� lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs�� __class__� ��/builddir/build/BUILDROOT/alt-python310-pip-21.3.1-3.el8.x86_64/opt/alt/python310/lib/python3.10/site-packages/pip/_vendor/html5lib/_tokenizer.pyr# ( s zHTMLTokenizer.__init__c c sf � t g �| _| �� r1| jjr td | jj�d�d�V | jjs| jr+| j�� V | js"| �� s dS dS )z� This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. � ParseErrorr ��type�dataN)r � tokenQueuer r �errorsr �pop�popleft�r$ r( r( r) �__iter__7 s � � ��zHTMLTokenizer.__iter__c C s� t }d}|r t}d}g }| j�� }||v r+|tur+|�|� | j�� }||v r+|tustd�|�|�}|tv rJt| }| j �t d dd|id�� n�d| krTd ksYn |d krjd }| j �t d dd|id�� nfd | krtd ks�n d| krdks�n d| kr�dks�n d| kr�dks�n |t g d��v r�| j �t d dd|id�� zt |�}W n t y� |d }t d|d? B �t d|d@ B � }Y nw |dkr�| j �t d dd�� | j�|� |S )z�This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. � � � r* z$illegal-codepoint-for-numeric-entity� charAsInt�r, r- �datavarsi � i�� � � �r � � � � � i� i� )#� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� r: i i � i� �;z numeric-entity-without-semicolonr+ )r r r �charr �append�int�joinr r. r � frozenset�chr� ValueError�unget) r$ �isHex�allowed�radix� charStack�cr7 rC �vr( r( r) �consumeNumberEntityG s\ � � � � $� � z!HTMLTokenizer.consumeNumberEntityFc C s� d}| j �� g}|d tv s!|d tddfv s!|d ur+||d kr+| j �|d � �n|d dkr�d}|�| j �� � |d dv rKd}|�| j �� � |rS|d tv s[|si|d tv ri| j �|d � | �|�}n�| j �t d d d �� | j �|� � � dd � |� }n�|d tur�t �d � |��s�n|�| j �� � |d tus�zt �d � |d d� ��}t|�}W n ty� d }Y nw |d u�r|d d kr�| j �t d dd �� |d d kr�|r�|| tv s�|| tv s�|| dkr�| j �|� � � dd � |� }n2t| }| j �|� � � |d � ||d � �7 }n| j �t d dd �� | j �|� � � dd � |� }|�rC| jd d d |7 < d S |tv �rKd}nd}| j �t | |d �� d S )N�&r �<�#F���)�x�XTr* zexpected-numeric-entityr+ r6 rB z named-entity-without-semicolon�=zexpected-named-entityr- r �SpaceCharacters� Characters)r rC r r rJ rD r r rQ r. r r0 rF � entitiesTrie�has_keys_with_prefix�longest_prefix�len�KeyErrorr r r! ) r$ � allowedChar� fromAttribute�outputrN �hex� entityName� entityLength� tokenTyper( r( r) � consumeEntity� st � � � � � � zHTMLTokenizer.consumeEntityc C s | j |dd� dS )zIThis method replaces the need for "entityInAttributeValueState". T)r` ra N)rg )r$ r` r( r( r) �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s� | j }|d tv r^|d �t�|d< |d td kr8|d }t|�}t|�t|�kr4|�|ddd� � ||d< |d td kr^|d rO| j� td d d �� |d r^| j� td d d �� | j� |� | j | _ dS ) z�This method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. r, �name�StartTagr- NrU �EndTagr* zattributes-in-end-tagr+ � selfClosingz self-closing-flag-on-end-tag) r! r � translater r � attributeMapr^ �updater. rD r r )r$ �token�rawr- r( r( r) �emitCurrentToken� s( � � z HTMLTokenizer.emitCurrentTokenc C s� | j �� }|dkr| j| _d S |dkr| j| _d S |dkr5| j�td dd�� | j�td dd�� d S |tu r;dS |t v rS| j�td || j � t d � d�� d S | j � d �}| j�td || d�� d S ) NrR rS � r* �invalid-codepointr+ rZ FrY T�rR rS rs ) r rC �entityDataStater � tagOpenStater. rD r r r � charsUntil�r$ r- �charsr( r( r) r s6 �� � �� � � �zHTMLTokenizer.dataStatec C � | � � | j| _dS �NT)rg r r r2 r( r( r) rv � z HTMLTokenizer.entityDataStatec C s� | j �� }|dkr| j| _d S |dkr| j| _d S |tkr dS |dkr;| j�td dd�� | j�td d d�� d S |t v rS| j�td || j � t d � d�� d S | j � d �}| j�td || d�� d S ) NrR rS Frs r* rt r+ rZ r; rY Tru ) r rC �characterReferenceInRcdatar �rcdataLessThanSignStater r. rD r r rx ry r( r( r) � rcdataState" s6 �� � �� � � �zHTMLTokenizer.rcdataStatec C r{ r| )rg r� r r2 r( r( r) r~ ? r} z(HTMLTokenizer.characterReferenceInRcdatac C � | j �� }|dkr| j| _d S |dkr+| j�td dd�� | j�td dd�� d S |tkr1dS | j �d �}| j�td || d�� d S � NrS rs r* rt r+ rZ r; F)rS rs T) r rC �rawtextLessThanSignStater r. rD r r rx ry r( r( r) � rawtextStateD �& � � � � �zHTMLTokenizer.rawtextStatec C r� r� ) r rC �scriptDataLessThanSignStater r. rD r r rx ry r( r( r) �scriptDataStateV r� z HTMLTokenizer.scriptDataStatec C st | j �� }|tkr dS |dkr'| j�td dd�� | j�td dd�� dS | j�td || j �d� d�� dS ) NFrs r* rt r+ rZ r; T)r rC r r. rD r rx �r$ r- r( r( r) �plaintextStateh s � � ��z HTMLTokenizer.plaintextStatec C s | j �� }|dkr| j| _dS |dkr| j| _dS |tv r.td |g ddd�| _| j| _dS |dkrN| j � td dd �� | j � td d d �� | j | _dS |d kri| j � td d d �� | j � |� | j | _dS | j � td dd �� | j � td dd �� | j � |� | j | _dS )N�!�/rj F)r, ri r- rl �selfClosingAcknowledged�>r* z'expected-tag-name-but-got-right-bracketr+ rZ z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerS T)r rC �markupDeclarationOpenStater �closeTagOpenStater r r! � tagNameStater. rD r rJ �bogusCommentStater� r( r( r) rw w sH � ��� �� � �� zHTMLTokenizer.tagOpenStatec C s� | j �� }|tv rtd |g dd�| _| j| _dS |dkr.| j�td dd�� | j | _dS |t u rN| j�td dd�� | j�td d d�� | j | _dS | j�td d d |id �� | j � |� | j | _dS )Nrk F�r, ri r- rl r� r* z*expected-closing-tag-but-got-right-bracketr+ z expected-closing-tag-but-got-eofrZ �</z!expected-closing-tag-but-got-charr- r8 T) r rC r r r! r� r r. rD r r rJ r� r� r( r( r) r� � s6 �� � � � �� z HTMLTokenizer.closeTagOpenStatec C s� | j �� }|tv r| j| _d S |dkr| �� d S |tu r.| j�t d dd�� | j | _d S |dkr8| j | _d S |dkrR| j�t d dd�� | j d d 7 < d S | j d |7 < d S ) Nr� r* zeof-in-tag-namer+ r� rs rt ri r; T) r rC r �beforeAttributeNameStater rr r r. rD r r �selfClosingStartTagStater! r� r( r( r) r� � s0 �� � � � ��zHTMLTokenizer.tagNameStatec C �R | j �� }|dkrd| _| j| _dS | j�td dd�� | j �|� | j | _dS �Nr� r6 rZ rS r+ T) r rC �temporaryBuffer�rcdataEndTagOpenStater r. rD r rJ r� r� r( r( r) r � � � z%HTMLTokenizer.rcdataLessThanSignStatec C �Z | j �� }|tv r| j|7 _| j| _dS | j�td dd�� | j � |� | j | _dS �NrZ r� r+ T) r rC r r� �rcdataEndTagNameStater r. rD r rJ r� r� r( r( r) r� � � � z#HTMLTokenizer.rcdataEndTagOpenStatec C � | j o| j d �� | j�� k}| j�� }|tv r+|r+td | jg dd�| _ | j| _d S |dkrB|rBtd | jg dd�| _ | j | _d S |dkr]|r]td | jg dd�| _ | � � | j | _d S |t v rj| j|7 _d S | j �td d| j d �� | j�|� | j| _d S � Nri rk Fr� r� r� rZ r� r+ T)r! �lowerr� r rC r r r� r r� rr r r r. rD rJ r� �r$ � appropriater- r( r( r) r� � �B � �� ��� �� z#HTMLTokenizer.rcdataEndTagNameStatec C r� r� ) r rC r� �rawtextEndTagOpenStater r. rD r rJ r� r� r( r( r) r� � r� z&HTMLTokenizer.rawtextLessThanSignStatec C r� r� ) r rC r r� �rawtextEndTagNameStater r. rD r rJ r� r� r( r( r) r� r� z$HTMLTokenizer.rawtextEndTagOpenStatec C r� r� )r! r� r� r rC r r r� r r� rr r r r. rD rJ r� r� r( r( r) r� r� z$HTMLTokenizer.rawtextEndTagNameStatec C s| | j �� }|dkrd| _| j| _dS |dkr'| j�td dd�� | j| _dS | j�td dd�� | j � |� | j | _dS ) Nr� r6 r� rZ z<!r+ rS T) r rC r� �scriptDataEndTagOpenStater r. rD r �scriptDataEscapeStartStaterJ r� r� r( r( r) r� , s �� z)HTMLTokenizer.scriptDataLessThanSignStatec C r� r� ) r rC r r� �scriptDataEndTagNameStater r. rD r rJ r� r� r( r( r) r� : r� z'HTMLTokenizer.scriptDataEndTagOpenStatec C r� r� )r! r� r� r rC r r r� r r� rr r r r. rD rJ r� r� r( r( r) r� E r� z'HTMLTokenizer.scriptDataEndTagNameStatec C �L | j �� }|dkr| j�td dd�� | j| _dS | j �|� | j| _dS �N�-rZ r+ T) r rC r. rD r � scriptDataEscapeStartDashStater rJ r� r� r( r( r) r� a � �z(HTMLTokenizer.scriptDataEscapeStartStatec C r� r� ) r rC r. rD r � scriptDataEscapedDashDashStater rJ r� r� r( r( r) r� k r� z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s� | j �� }|dkr| j�td dd�� | j| _d S |dkr$| j| _d S |dkr@| j�td dd�� | j�td dd�� d S |tkrJ| j | _d S | j � d �}| j�td || d�� d S ) Nr� rZ r+ rS rs r* rt r; )rS r� rs T) r rC r. rD r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r rx ry r( r( r) �scriptDataEscapedStateu s0 � � � �� � �z$HTMLTokenizer.scriptDataEscapedStatec C s� | j �� }|dkr| j�td dd�� | j| _d S |dkr$| j| _d S |dkrD| j�td dd�� | j�td dd�� | j| _d S |t krN| j | _d S | j�td |d�� | j| _d S ) Nr� rZ r+ rS rs r* rt r; T) r rC r. rD r r� r r� r� r r r� r( r( r) r� � s. � � � ���z(HTMLTokenizer.scriptDataEscapedDashStatec C s� | j �� }|dkr| j�td dd�� d S |dkr | j| _d S |dkr5| j�td dd�� | j| _d S |dkrU| j�td dd�� | j�td d d�� | j| _d S |t kr_| j | _d S | j�td |d�� | j| _d S ) Nr� rZ r+ rS r� rs r* rt r; T) r rC r. rD r r� r r� r� r r r� r( r( r) r� � s4 �� � � ���z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j �� }|dkrd| _| j| _dS |tv r,| j�td d| d�� || _| j | _dS | j�td dd�� | j � |� | j | _dS r� ) r rC r� � scriptDataEscapedEndTagOpenStater r r. rD r � scriptDataDoubleEscapeStartStaterJ r� r� r( r( r) r� � s �� z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C sR | j �� }|tv r|| _| j| _dS | j�td dd�� | j � |� | j | _dS r� ) r rC r r� � scriptDataEscapedEndTagNameStater r. rD r rJ r� r� r( r( r) r� � r� z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C r� r� )r! r� r� r rC r r r� r r� rr r r r. rD rJ r� r� r( r( r) r� � r� z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C � | j �� }|ttd�B v r+| j�td |d�� | j�� dkr%| j | _ dS | j | _ dS |t v rC| j�td |d�� | j|7 _dS | j � |� | j | _ dS �N)r� r� rZ r+ �scriptT)r rC r rG r. rD r r� r� � scriptDataDoubleEscapedStater r� r rJ r� r( r( r) r� � � �� �z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j �� }|dkr| j�td dd�� | j| _d S |dkr/| j�td dd�� | j| _d S |dkrK| j�td dd�� | j�td dd�� d S |tkr`| j�td d d�� | j | _d S | j�td |d�� d S � Nr� rZ r+ rS rs r* rt r; �eof-in-script-in-scriptT) r rC r. rD r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r r� r( r( r) r� � s2 � � � �� ��z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j �� }|dkr| j�td dd�� | j| _d S |dkr/| j�td dd�� | j| _d S |dkrO| j�td dd�� | j�td dd�� | j| _d S |t krd| j�td d d�� | j | _d S | j�td |d�� | j| _d S r� ) r rC r. rD r �$scriptDataDoubleEscapedDashDashStater r� r� r r r� r( r( r) r� s6 �� � �� ��z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j �� }|dkr| j�td dd�� d S |dkr+| j�td dd�� | j| _d S |dkr@| j�td dd�� | j| _d S |dkr`| j�td dd�� | j�td d d�� | j| _d S |t kru| j�td d d�� | j | _d S | j�td |d�� | j| _d S ) Nr� rZ r+ rS r� rs r* rt r; r� T) r rC r. rD r r� r r� r� r r r� r( r( r) r� % s< ��� � �� ��z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sR | j �� }|dkr | j�td dd�� d| _| j| _dS | j �|� | j | _dS )Nr� rZ r+ r6 T) r rC r. rD r r� � scriptDataDoubleEscapeEndStater rJ r� r� r( r( r) r� > s �z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C r� r� )r rC r rG r. rD r r� r� r� r r� r rJ r� r( r( r) r� I r� z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s: | j �� }|tv r| j �td� dS |tv r&| jd �|dg� | j| _dS |dkr0| � � dS |dkr:| j | _dS |dv rY| j �t d dd �� | jd �|dg� | j| _dS |d krx| j �t d d d �� | jd �d dg� | j| _dS |t u r�| j �t d d d �� | j| _dS | jd �|dg� | j| _dS )NTr- r6 r� r� )�'�"rX rS r* �#invalid-character-in-attribute-namer+ rs rt r; z#expected-attribute-name-but-got-eof)r rC r rx r r! rD �attributeNameStater rr r� r. r r r r� r( r( r) r� Y sJ ���� � � �� ��z&HTMLTokenizer.beforeAttributeNameStatec C s� | j �� }d}d}|dkr| j| _n�|tv r-| jd d d || j �td� 7 < d}nz|dkr4d}ns|tv r=| j| _nj|dkrF| j | _na|d kre| j � t d d d �� | jd d d d 7 < d}nB|dv r�| j � t d dd �� | jd d d |7 < d}n#|t u r�| j � t d dd �� | j| _n| jd d d |7 < d}|r�| jd d d �t�| jd d d< | jd d d� D ] \}}| jd d d |kr�| j � t d dd �� nq�|r�| �� dS )NTFrX r- rU r r� r� rs r* rt r+ r; �r� r� rS r� zeof-in-attribute-namezduplicate-attribute)r rC �beforeAttributeValueStater r r! rx r �afterAttributeNameStater� r. rD r r r rm r rr )r$ r- �leavingThisState� emitTokenri �_r( r( r) r� w s` � � � � � ��z HTMLTokenizer.attributeNameStatec C sN | j �� }|tv r| j �td� dS |dkr | j| _dS |dkr&| �� dS |tv r:| jd � |dg� | j | _dS |dkrD| j | _dS |dkrc| j � t d d d �� | jd � d dg� | j | _dS |d v r�| j � t d d d �� | jd � |dg� | j | _dS |tu r�| j � t d dd �� | j| _dS | jd � |dg� | j | _dS )NTrX r� r- r6 r� rs r* rt r+ r; r� z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r rC r rx r� r rr r r! rD r� r� r. r r r r� r( r( r) r� � sP ����� � � �� ��z%HTMLTokenizer.afterAttributeNameStatec C sn | j �� }|tv r| j �td� dS |dkr | j| _dS |dkr,| j| _| j �|� dS |dkr6| j| _dS |dkrK| j � t d dd�� | � � dS |d krm| j � t d d d�� | j d d d d7 < | j| _dS |dv r�| j � t d dd�� | j d d d |7 < | j| _dS |tu r�| j � t d dd�� | j| _dS | j d d d |7 < | j| _dS )NTr� rR r� r� r* z.expected-attribute-value-but-got-right-bracketr+ rs rt r- rU r r; )rX rS �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)r rC r rx � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterJ � attributeValueSingleQuotedStater. rD r rr r! r r r� r( r( r) r� � sV �� �� �� � � �� ��z'HTMLTokenizer.beforeAttributeValueStatec C �� | j �� }|dkr| j| _d S |dkr| �d� d S |dkr8| j�td dd�� | jd d d d 7 < d S |t u rM| j�td d d�� | j | _d S | jd d d || j � d � 7 < d S )Nr� rR rs r* rt r+ r- rU r r; z#eof-in-attribute-value-double-quote)r� rR rs T� r rC �afterAttributeValueStater rh r. rD r r! r r rx r� r( r( r) r� � �. � � �� �� �z-HTMLTokenizer.attributeValueDoubleQuotedStatec C r� )Nr� rR rs r* rt r+ r- rU r r; z#eof-in-attribute-value-single-quote)r� rR rs Tr� r� r( r( r) r� r� z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j �� }|tv r| j| _dS |dkr| �d� dS |dkr$| �� dS |dv rB| j�t d dd�� | j d d d |7 < dS |d kr`| j�t d d d�� | j d d d d 7 < dS |t u ru| j�t d d d�� | j | _dS | j d d d || j � td�tB � 7 < dS )NrR r� )r� r� rX rS r� r* z0unexpected-character-in-unquoted-attribute-valuer+ r- rU r rs rt r; z eof-in-attribute-value-no-quotes)rR r� r� r� rX rS r� rs T)r rC r r� r rh rr r. rD r r! r r rx rG r� r( r( r) r� s@ � �� � � �� �� �z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j �� }|tv r| j| _dS |dkr| �� dS |dkr#| j| _dS |tu r>| j� t d dd�� | j � |� | j | _dS | j� t d dd�� | j � |� | j| _dS )Nr� r� r* z$unexpected-EOF-after-attribute-valuer+ z*unexpected-character-after-attribute-valueT) r rC r r� r rr r� r r. rD r rJ r r� r( r( r) r� . s. � � � � �� z&HTMLTokenizer.afterAttributeValueStatec C s� | j �� }|dkrd| jd< | �� dS |tu r/| j�td dd�� | j �|� | j | _ dS | j�td dd�� | j �|� | j | _ dS )Nr� Trl r* z#unexpected-EOF-after-solidus-in-tagr+ z)unexpected-character-after-solidus-in-tag) r rC r! rr r r. rD r rJ r r r� r� r( r( r) r� B s$ � � �� z&HTMLTokenizer.selfClosingStartTagStatec C sD | j �d�}|�dd�}| j�td |d�� | j �� | j| _dS )Nr� rs r; �Commentr+ T) r rx �replacer. rD r rC r r r� r( r( r) r� T s � z HTMLTokenizer.bogusCommentStatec C sn | j �� g}|d dkr)|�| j �� � |d dkr(td dd�| _| j| _dS nq|d dv r\d}dD ]}|�| j �� � |d |vrGd } nq3|r[td dd d dd �| _| j| _dS n>|d d kr�| jd ur�| jj j r�| jj j d j | jj j kr�d}d D ]}|�| j �� � |d |kr�d } nq}|r�| j | _dS | j�td dd�� |r�| j �|�� � |s�| j| _dS )NrU r� r� r6 r+ T)�d�D))�o�O�rO �C��t�T��y�Y��p�P��e�EF�Doctype)r, ri �publicId�systemId�correct�[)r� r� �Ar� r� r� r* zexpected-dashes-or-doctype)r rC rD r r! �commentStartStater � doctypeStater �tree� openElements� namespace�defaultNamespace�cdataSectionStater. rJ r0 r� )r$ rN �matched�expectedr( r( r) r� c s` � ��� � � ��z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j �� }|dkr| j| _d S |dkr)| j�td dd�� | jd d7 < d S |dkrE| j�td d d�� | j�| j� | j| _d S |t u ra| j�td d d�� | j�| j� | j| _d S | jd |7 < | j | _d S ) Nr� rs r* rt r+ r- r; r� �incorrect-comment�eof-in-commentT) r rC �commentStartDashStater r. rD r r! r r � commentStater� r( r( r) r� � s6 � �� � � ��z HTMLTokenizer.commentStartStatec C s� | j �� }|dkr| j| _d S |dkr)| j�td dd�� | jd d7 < d S |dkrE| j�td d d�� | j�| j� | j| _d S |t u ra| j�td d d�� | j�| j� | j| _d S | jd d| 7 < | j | _d S ) Nr� rs r* rt r+ r- � -�r� r� r� T) r rC �commentEndStater r. rD r r! r r r� r� r( r( r) r� � s6 � �� � � ��z#HTMLTokenizer.commentStartDashStatec C s� | j �� }|dkr| j| _d S |dkr)| j�td dd�� | jd d7 < d S |tu rE| j�td dd�� | j�| j� | j | _d S | jd || j � d � 7 < d S ) Nr� rs r* rt r+ r- r; r� )r� rs T) r rC �commentEndDashStater r. rD r r! r r rx r� r( r( r) r� � s* � � � � � �zHTMLTokenizer.commentStatec C s� | j �� }|dkr| j| _d S |dkr-| j�td dd�� | jd d7 < | j| _d S |t u rI| j�td dd�� | j�| j� | j | _d S | jd d| 7 < | j| _d S ) Nr� rs r* rt r+ r- r� zeof-in-comment-end-dashT) r rC r� r r. rD r r! r� r r r� r( r( r) r� � s* � � � ��z!HTMLTokenizer.commentEndDashStatec C s6 | j �� }|dkr| j�| j� | j| _dS |dkr4| j�td dd�� | jd d7 < | j| _dS |dkrI| j�td d d�� | j | _dS |d krc| j�td d d�� | jd |7 < dS |t u r| j�td d d�� | j�| j� | j| _dS | j�td d d�� | jd d| 7 < | j| _dS )Nr� rs r* rt r+ r- u --�r� z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r rC r. rD r! r r r r� �commentEndBangStater r� r( r( r) r� � sJ � �� �� � � � ��z HTMLTokenizer.commentEndStatec C s� | j �� }|dkr| j�| j� | j| _d S |dkr)| jd d7 < | j| _d S |dkrG| j�td dd�� | jd d 7 < | j | _d S |t u rc| j�td d d�� | j�| j� | j| _d S | jd d| 7 < | j | _d S ) Nr� r� r- z--!rs r* rt r+ u --!�z eof-in-comment-end-bang-stateT) r rC r. rD r! r r r� r r� r r� r( r( r) r� s4 �� � � ��z!HTMLTokenizer.commentEndBangStatec C s� | j �� }|tv r| j| _dS |tu r0| j�td dd�� d| j d< | j�| j � | j | _dS | j�td dd�� | j � |� | j| _dS )Nr* �!expected-doctype-name-but-got-eofr+ Fr� zneed-space-after-doctypeT) r rC r �beforeDoctypeNameStater r r. rD r r! r rJ r� r( r( r) r� s$ � � �� zHTMLTokenizer.doctypeStatec C s� | j �� }|tv r d S |dkr-| j�td dd�� d| jd< | j�| j� | j| _d S |dkrG| j�td dd�� d | jd < | j | _d S |t u rh| j�td d d�� d| jd< | j�| j� | j| _d S || jd < | j | _d S ) Nr� r* z+expected-doctype-name-but-got-right-bracketr+ Fr� rs rt r; ri r� T) r rC r r. rD r r! r r �doctypeNameStater r� r( r( r) r * s< � � � � � � �z$HTMLTokenizer.beforeDoctypeNameStatec C s | j �� }|tv r| jd �t�| jd< | j| _d S |dkr6| jd �t�| jd< | j� | j� | j | _d S |dkrT| j� t d dd�� | jd d7 < | j | _d S |t u r�| j� t d dd�� d | jd < | jd �t�| jd< | j� | j� | j | _d S | jd |7 < d S ) Nri r� rs r* rt r+ r; zeof-in-doctype-nameFr� T)r rC r r! rm r �afterDoctypeNameStater r. rD r r r r r� r( r( r) r D s8 �� � � � �z HTMLTokenizer.doctypeNameStatec C sH | j �� }|tv r dS |dkr | j�| j� | j| _dS |tu rDd| jd< | j � |� | j�t d dd�� | j�| j� | j| _dS |dv red}d D ]}| j �� }||vr[d} nqL|rd| j | _dS n |d v r�d}d D ]}| j �� }||vr|d} nqm|r�| j | _dS | j � |� | j�t d d d |id�� d| jd< | j | _dS )Nr� Fr� r* �eof-in-doctyper+ r� T))�u�U)�b�B)�l�L)�i�Ir� ��s�S)r� r r� r� )�m�Mz*expected-space-or-right-bracket-in-doctyper- r8 )r rC r r. rD r! r r r rJ r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r$ r- r� r� r( r( r) r ] s` .�+� �$� �� � � z#HTMLTokenizer.afterDoctypeNameStatec C � | j �� }|tv r| j| _dS |dv r*| j�td dd�� | j �|� | j| _dS |t u rK| j�td dd�� d| j d< | j�| j � | j | _dS | j �|� | j| _dS � N)r� r� r* �unexpected-char-in-doctyper+ r Fr� T) r rC r �"beforeDoctypePublicIdentifierStater r. rD r rJ r r! r r� r( r( r) r � �, � � � � �z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s | j �� }|tv r d S |dkrd| jd< | j| _d S |dkr*d| jd< | j| _d S |dkrK| j�t d dd�� d | jd < | j�| j� | j | _d S |t u rl| j�t d d d�� d | jd < | j�| j� | j | _d S | j�t d d d�� d | jd < | j | _d S )Nr� r6 r� r� r� r* �unexpected-end-of-doctyper+ Fr� r r T) r rC r r! �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater. rD r r r r r� r( r( r) r � �D � � � � � � �� z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C �� | j �� }|dkr| j| _d S |dkr)| j�td dd�� | jd d7 < d S |dkrJ| j�td d d�� d | jd < | j�| j� | j| _d S |t u rk| j�td d d�� d | jd < | j�| j� | j| _d S | jd |7 < d S )Nr� rs r* rt r+ r� r; r� r Fr� r T� r rC �!afterDoctypePublicIdentifierStater r. rD r r! r r r� r( r( r) r � �8 � �� � � � �z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C r )Nr� rs r* rt r+ r� r; r� r Fr� r Tr r� r( r( r) r � r z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j �� }|tv r| j| _d S |dkr | j�| j� | j| _d S |dkr:| j�t d dd�� d| jd< | j | _d S |dkrT| j�t d dd�� d| jd< | j | _d S |t u ru| j�t d d d�� d | jd < | j�| j� | j| _d S | j�t d dd�� d | jd < | j | _d S ) Nr� r� r* r r+ r6 r� r� r Fr� T)r rC r �-betweenDoctypePublicAndSystemIdentifiersStater r. rD r! r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r r� r( r( r) r � sH �� � � � � � �� z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j �� }|tv r d S |dkr | j�| j� | j| _d S |dkr,d| jd< | j| _d S |dkr;d| jd< | j | _d S |t kr\| j�t d dd�� d | jd < | j�| j� | j| _d S | j�t d d d�� d | jd < | j | _d S ) Nr� r� r6 r� r� r* r r+ Fr� r T) r rC r r. rD r! r r r" r# r r r r� r( r( r) r! s< �� � � � �� z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C r r ) r rC r �"beforeDoctypeSystemIdentifierStater r. rD r rJ r r! r r� r( r( r) r ) r z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s | j �� }|tv r d S |dkrd| jd< | j| _d S |dkr*d| jd< | j| _d S |dkrK| j�t d dd�� d | jd < | j�| j� | j | _d S |t u rl| j�t d d d�� d | jd < | j�| j� | j | _d S | j�t d dd�� d | jd < | j | _d S ) Nr� r6 r� r� r� r* r r+ Fr� r T) r rC r r! r" r r# r. rD r r r r r� r( r( r) r$ = r z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C r )Nr� rs r* rt r+ r� r; r� r Fr� r T� r rC �!afterDoctypeSystemIdentifierStater r. rD r r! r r r� r( r( r) r" Z r z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C r )Nr� rs r* rt r+ r� r; r� r Fr� r Tr% r� r( r( r) r# r r z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j �� }|tv r dS |dkr | j�| j� | j| _dS |tu r>| j�t d dd�� d| jd< | j�| j� | j| _dS | j�t d dd�� | j | _dS ) Nr� r* r r+ Fr� r T) r rC r r. rD r! r r r r r r� r( r( r) r& � s* � � � ��z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C s` | j �� }|dkr| j�| j� | j| _dS |tu r-| j �|� | j�| j� | j| _dS dS )Nr� T) r rC r. rD r! r r r rJ r� r( r( r) r � s � �z HTMLTokenizer.bogusDoctypeStatec C s� g } |� | j�d�� |� | j�d�� | j�� }|tkr n!|dks%J �|d dd � dkr:|d d d� |d< n|� |� qd�|�}|�d�}|d krft|�D ] }| j� t d d d �� qR|� dd �}|rs| j� t d |d �� | j | _ dS )NT�]r� rU ���z]]r6 rs r r* rt r+ r; rZ ) rD r rx rC r rF �count�ranger. r r� r r )r$ r- rC � nullCountr� r( r( r) r� � s6 � � �z HTMLTokenizer.cdataSectionState)Nr )N�__name__� __module__� __qualname__�__doc__r# r3 rQ rg rh rr r rv r� r~ r� r� r� rw r� r� r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r r r r r! r r$ r" r# r& r r� � __classcell__r( r( r& r) r s� HP # 6 "- 3 r N) � __future__r r r Zpip._vendor.sixr rH � collectionsr r �sysr � constantsr r r r r r r r r r � _inputstreamr �_trier r[ �dictrn �objectr r( r( r( r) �<module> s"