관리-도구
편집 파일: collector.cpython-38.pyc
U ʗReMO � @ sL d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl ZddlmZ ddlmZ ddlmZmZmZmZmZmZmZmZm Z m Z m Z ddl m Z ddl!m"Z" ddl#m$Z$m%Z% dd l&m'Z' dd l(m)Z) dd l*m+Z+ dd l,m-Z- dd l.m/Z/ ddl0m1Z1 ddl2m3Z3m4Z4 ddl5m6Z6 ddl7m8Z8m9Z9m:Z: e�r\ddlm;Z; ne<Z;e�=e>�Z?ej@jAjBZCeeDeDf ZEeDeeD d�dd�ZFG dd� deG�ZHe"dd�dd�ZIG d d � d eG�ZJeDe-dd �d d �ZKeDe-e"d �d!d"�ZLeEeeD d#�d$d%�ZMeDeDd&�d'd(�ZNeDeDd&�d)d*�ZOe �Pd+e jQ�ZReDeSeDd,�d-d.�ZTeDeDd�d/d0�ZUeeDeeD f eDeDee) d1�d2d3�ZVG d4d5� d5�ZWG d6d7� d7e;�ZXeXeXd8�d9d:�ZYeYd;ee) d<�d=d>��ZZG d?d;� d;�Z[G d@dA� dAe�Z\dQe)e eDeGf eedB ddC�dDdE�Z]dRe"eSe[dG�dHdI�Z^dSe)ee- ed; dJ�dKdL�Z_G dMdN� dNe�Z`G dOdP� dP�ZadS )TzO The main purpose of this module is to expose LinkCollector.collect_sources(). � N)� HTMLParser)�Values) � TYPE_CHECKING�Callable�Dict�Iterable�List�MutableMapping� NamedTuple�Optional�Sequence�Tuple�Union)�requests)�Response)� RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)� PipSession)�raise_for_status)�is_archive_file)�pairwise�redact_auth_from_url)�vcs� )�CandidatesFromPage� LinkSource� build_source)�Protocol��url�returnc C s6 t jD ]*}| �� �|�r| t|� dkr| S qdS )zgLook for VCS schemes in the URL. Returns the matched VCS scheme, or None if there's no match. z+:N)r �schemes�lower� startswith�len)r" �scheme� r) ��/builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_internal/index/collector.py�_match_vcs_scheme: s r+ c s& e Zd Zeedd�� fdd� Z� ZS )�_NotAPIContentN)� content_type� request_descr# c s t � �||� || _|| _d S �N)�super�__init__r- r. )�selfr- r. �� __class__r) r* r1 F s z_NotAPIContent.__init__)�__name__� __module__� __qualname__�strr1 � __classcell__r) r) r3 r* r, E s r, )�responser# c C s6 | j �dd�}|�� }|�d�r$dS t|| jj��dS )z� Check the Content-Type header to ensure the response contains a Simple API Response. Raises `_NotAPIContent` if the content type is not a valid content-type. � Content-Type�Unknown)z text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr% r& r, �request�method)r: r- �content_type_lr) r) r* �_ensure_api_headerL s �rC c @ s e Zd ZdS )�_NotHTTPN)r5 r6 r7 r) r) r) r* rD b s rD )r" �sessionr# c C sF t j�| �\}}}}}|dkr$t� �|j| dd�}t|� t|� dS )z� Send a HEAD request to the URL, and ensure the response contains a simple API Response. Raises `_NotHTTP` if the URL is not available for a HEAD request, or `_NotAPIContent` if the content type is not a valid content type. > �http�httpsT)�allow_redirectsN)�urllib�parse�urlsplitrD �headr rC )r" rE r( �netloc�path�query�fragment�respr) r) r* �_ensure_api_responsef s rR c C sz t t| �j�rt| |d� t�dt| �� |j| d�dddg�dd�d �}t |� t |� t�d t| �|j �d d �� |S ) aY Access an Simple API response with GET, and return the response. This consists of three parts: 1. If the URL looks suspiciously like an archive, send a HEAD first to check the Content-Type is HTML or Simple API, to avoid downloading a large file. Raise `_NotHTTP` if the content type cannot be determined, or `_NotAPIContent` if it is not HTML or a Simple API. 2. Actually perform the request. Raise HTTP exceptions on network failures. 3. Check the Content-Type header to make sure we got a Simple API response, and raise `_NotAPIContent` otherwise. �rE zGetting page %sz, r= z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)�Acceptz Cache-Control)r> zFetched page %s as %sr; r< ) r r �filenamerR �logger�debugr r? �joinr rC r> )r" rE rQ r) r) r* �_get_simple_responsex s, ���� �rY )r> r# c C s< | r8d| kr8t j�� }| d |d< |�d�}|r8t|�S dS )z=Determine if we have any encoding information in our headers.r; z content-type�charsetN)�email�message�Message� get_paramr8 )r> �mrZ r) r) r* �_get_encoding_from_headers� s r` )�partr# c C s t j�t j�| ��S )zP Clean a "part" of a URL path (i.e. after splitting on "@" characters). )rI rJ �quote�unquote�ra r) r) r* �_clean_url_path_part� s re c C s t j�t j�| ��S )z� Clean the first part of a URL path that corresponds to a local filesystem path (i.e. the first part after splitting on "@" characters). )rI r@ � pathname2url� url2pathnamerd r) r) r* �_clean_file_url_path� s rh z(@|%2F))rN � is_local_pathr# c C s^ |r t }nt}t�| �}g }tt�|dg��D ]$\}}|�||�� |�|�� � q.d� |�S )z* Clean the path portion of a URL. � ) rh re �_reserved_chars_re�splitr � itertools�chain�append�upperrX )rN ri � clean_func�parts� cleaned_parts�to_clean�reservedr) r) r* �_clean_url_path� s rv c C s6 t j�| �}|j }t|j|d�}t j�|j|d��S )z� Make sure a link is fully quoted. For example, if ' ' occurs in the URL, it will be replaced with "%20", and without double-quoting other characters. )ri )rN )rI rJ �urlparserM rv rN � urlunparse�_replace)r" �resultri rN r) r) r* � _clean_link� s r{ )�element_attribs�page_url�base_urlr# c C sL | � d�}|sdS ttj�||��}| � d�}| � d�}t||||d�}|S )zW Convert an anchor element's attributes in a simple repository page to a Link. �hrefNzdata-requires-pythonz data-yanked)� comes_from�requires_python� yanked_reason)r? r{ rI rJ �urljoinr )r| r} r~ r r" � pyrequirer� �linkr) r) r* �_create_link_from_element s �r� c @ s: e Zd Zddd�dd�Zeed�dd�Zed �d d �ZdS ) �CacheablePageContent� IndexContentN��pager# c C s |j s t�|| _d S r/ )�cache_link_parsing�AssertionErrorr� �r2 r� r) r) r* r1 s z CacheablePageContent.__init__)�otherr# c C s t |t| ��o | jj|jjkS r/ )� isinstance�typer� r" )r2 r� r) r) r* �__eq__ s zCacheablePageContent.__eq__�r# c C s t | jj�S r/ )�hashr� r" �r2 r) r) r* �__hash__" s z CacheablePageContent.__hash__) r5 r6 r7 r1 �object�boolr� �intr� r) r) r) r* r� s r� c @ s e Zd Zdee d�dd�ZdS )� ParseLinksr� r� c C s d S r/ r) r� r) r) r* �__call__' s zParseLinks.__call__N)r5 r6 r7 r r r� r) r) r) r* r� & s r� )�fnr# c sL t jdd�ttt d�� fdd� ��t �� �dtt d�� �fdd � �}|S ) z� Given a function that parses an Iterable[Link] from an IndexContent, cache the function's result (keyed by CacheablePageContent), unless the IndexContent `page` has `page.cache_link_parsing == False`. N)�maxsize)�cacheable_pager# c s t � | j��S r/ )�listr� )r� )r� r) r* �wrapper2 s z*with_cached_index_content.<locals>.wrapperr� r� c s | j r�t| ��S t� | ��S r/ )r� r� r� )r� �r� r� r) r* �wrapper_wrapper6 s z2with_cached_index_content.<locals>.wrapper_wrapper)� functools� lru_cacher� r r �wraps)r� r� r) r� r* �with_cached_index_content+ s r� r� r� c c s | j �� }|�d�r�t�| j�}|�dg �D ]r}|�d�}|dkrDq,|�d�}|rbt|t�sbd}n|sjd}t t t j � | j|��| j|�d�||�di �d �V q,t| j�}| jp�d }|�| j�|�� | j}|jp�|} |jD ]"} t| || d �} | dkr�q�| V q�dS ) z\ Parse a Simple API's Index Content, and yield its anchor elements as Link objects. r= �filesr" N�yankedrj zrequires-python�hashes)r� r� r� r� zutf-8)r} r~ )r- r% r& �json�loads�contentr? r� r8 r r{ rI rJ r� r" �HTMLLinkParser�encoding�feed�decoder~ �anchorsr� ) r� rB �data�file�file_urlr� �parserr� r" r~ �anchorr� r) r) r* � parse_links? sB � �r� c @ s< e Zd ZdZd eeee eedd�dd�Zed�dd �Z dS ) r� z5Represents one response (or page), along with its URLTN)r� r- r� r" r� r# c C s" || _ || _|| _|| _|| _dS )am :param encoding: the encoding to decode the given content. :param url: the URL from which the HTML was downloaded. :param cache_link_parsing: whether links parsed from this page's url should be cached. PyPI index urls should have this set to False, for example. N)r� r- r� r" r� )r2 r� r- r� r" r� r) r) r* r1 q s zIndexContent.__init__r� c C s t | j�S r/ )r r" r� r) r) r* �__str__� s zIndexContent.__str__)T) r5 r6 r7 �__doc__�bytesr8 r r� r1 r� r) r) r) r* r� n s �� c sn e Zd ZdZedd�� fdd� Zeeeeee f dd�dd�Z eeeee f ee d �d d �Z � Z S ) r� zf HTMLParser that keeps the first base HREF and a list of all anchor elements' attributes. Nr! c s$ t � jdd� || _d | _g | _d S )NT)�convert_charrefs)r0 r1 r" r~ r� )r2 r" r3 r) r* r1 � s zHTMLLinkParser.__init__)�tag�attrsr# c C sH |dkr,| j d kr,| �|�}|d k rD|| _ n|dkrD| j�t|�� d S )N�base�a)r~ �get_hrefr� ro �dict)r2 r� r� r r) r) r* �handle_starttag� s z HTMLLinkParser.handle_starttag)r� r# c C s" |D ]\}}|dkr| S qd S )Nr r) )r2 r� �name�valuer) r) r* r� � s zHTMLLinkParser.get_href) r5 r6 r7 r� r8 r1 r r r r� r� r9 r) r) r3 r* r� � s "r� ).N)r� �reason�methr# c C s |d krt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rV rW )r� r� r� r) r) r* �_handle_get_simple_fail� s r� T)r: r� r# c C s& t | j�}t| j| jd || j|d�S )Nr; )r� r" r� )r` r> r� r� r" )r: r� r� r) r) r* �_make_index_content� s �r� )r� rE r# c C s |d krt d��| j�dd�d }t|�}|r@t�d|| � d S tj�|�\}}}}}}|dkr�t j � tj � |��r�|�d�s�|d7 }tj�|d�}t�d |� zt||d �}W �nF tk r� t�d | � Y �n4 tk �r } zt�d | |j|j� W 5 d }~X Y n� tk �r6 } zt| |� W 5 d }~X Y n� tk �rb } zt| |� W 5 d }~X Y n� tk �r� } z$d } | t|�7 } t| | tjd� W 5 d }~X Y nd t j k �r� } zt| d|� �� W 5 d }~X Y n0 t j k �r� t| d� Y nX t || j d�S d S )Nz?_get_html_page() missing 1 required keyword argument: 'session'�#r r zICannot look at %s URL %s because it does not support lookup as web pages.r� �/z index.htmlz# file: URL is directory, getting %srS z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)r� )!� TypeErrorr" rl r+ rV �warningrI rJ rw �osrN �isdirr@ rg �endswithr� rW rY rD r, r. r- r r� r r r8 �infor �ConnectionError�Timeoutr� r� ) r� rE r" � vcs_schemer( �_rN rQ �excr� r) r) r* �_get_index_content� s^ �� � � r� c @ s. e Zd ZU eee ed< eee ed< dS )�CollectedSources� find_links� index_urlsN)r5 r6 r7 r r r �__annotations__r) r) r) r* r� s r� c @ sx e Zd ZdZeedd�dd�Zedeee d d�dd ��Z e e e d �d d ��Zeee d �dd�Ze eed�dd�ZdS )� LinkCollectorz� Responsible for collecting Link objects from all configured locations, making network requests as needed. The class's main method is its collect_sources() method. N)rE � search_scoper# c C s || _ || _d S r/ )r� rE )r2 rE r� r) r) r* r1 s zLinkCollector.__init__F)rE �options�suppress_no_indexr# c C s` |j g|j }|jr8|s8t�dd�dd� |D ��� g }|jp@g }tj||d�}t ||d�}|S )z� :param session: The Session to use to make requests. :param suppress_no_index: Whether to ignore the --no-index option when constructing the SearchScope object. zIgnoring indexes: %s�,c s s | ]}t |�V qd S r/ )r )�.0r" r) r) r* � <genexpr>' s z'LinkCollector.create.<locals>.<genexpr>�r� r� )rE r� ) � index_url�extra_index_urls�no_indexrV rW rX r� r �creater� )�clsrE r� r� r� r� r� �link_collectorr) r) r* r� s" � ��zLinkCollector.creater� c C s | j jS r/ )r� r� r� r) r) r* r� 8 s zLinkCollector.find_links)�locationr# c C s t || jd�S )z> Fetch an HTML page containing package links. rS )r� rE )r2 r� r) r) r* �fetch_response< s z LinkCollector.fetch_response)� project_name�candidates_from_pager# c s� t �� �fdd��j�|�D ���� }t �� �fdd��jD ���� }t�tj �r�dd� t � ||�D �}t |�� d|� d�g| }t� d�|�� tt|�t|�d �S ) Nc 3 s$ | ] }t |� �jjd d d�V qdS )F�r� �page_validator� expand_dirr� N�r rE �is_secure_origin�r� �loc�r� r2 r) r* r� H s ��z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 s$ | ] }t |� �jjd d d�V qdS )Tr� Nr� r� r� r) r* r� R s ��c S s* g | ]"}|d k r|j d k rd|j � ��qS )Nz* )r� )r� �sr) r) r* � <listcomp>^ s �z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:� r� )� collections� OrderedDictr� �get_index_urls_locations�valuesr� rV � isEnabledFor�logging�DEBUGrm rn r' rW rX r� r� )r2 r� r� �index_url_sources�find_links_sources�linesr) r� r* �collect_sourcesB s& � � ����z LinkCollector.collect_sources)F)r5 r6 r7 r� r r r1 � classmethodr r� r� �propertyr r8 r� r r r� r� r r� r r) r) r) r* r� s( � �� �r� )N)T)N)br� r� � email.messager[ r� rm r� r� r� �re� urllib.parserI �urllib.request�xml.etree.ElementTree�xml� html.parserr �optparser �typingr r r r r r r r r r r � pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r �pip._internal.exceptionsr �pip._internal.models.linkr �!pip._internal.models.search_scoper � pip._internal.network.sessionr �pip._internal.network.utilsr � pip._internal.utils.filetypesr �pip._internal.utils.miscr r �pip._internal.vcsr �sourcesr r r r r� � getLoggerr5 rV �etree� ElementTree�Element� HTMLElementr8 �ResponseHeadersr+ � Exceptionr, rC rD rR rY r` re rh �compile� IGNORECASErk r� rv r{ r� r� r� r� r� r� r� r� r� r� r� r� r) r) r) r* �<module> s� 4 ? � . � � � � � � D