[robbrad]

This is a regression from 0.173.5, and the most important item in this release

#2294: "This has been working for me for over a year until updating last night to 0.173.5" (Cheshire East), with a second user reporting the same for Liverpool.

Cause

Eight scrapers accept the UPRN either as uprn or inside a URL stored by an old Home Assistant config (those configs have no uprn field at all):

user_uprn = kwargs.get("uprn")
check_uprn(user_uprn)                    # <- first
url = f"...?uprn={user_uprn}"
if not user_uprn:
    url = kwargs.get("url")              # <- legacy fallback, after

Until 0.173.5, check_uprn() only printed a warning for a missing UPRN and returned, so the fallback ran. #2291 made it raise (the right call for scrapers that genuinely need a UPRN), but these eight called it before the fallback. So every legacy install fails at setup with exactly the message in the issue: Error getting identifier: Invalid UPRN. Please check the provided UPRN.

Affected: Cheshire East, Dartford, Eastbourne, Glasgow, Herefordshire, Lewes, Liverpool, Newham.

How I found the full set

Not just the two reported. I scanned all 190 check_uprn/check_usrn call sites for the "check, then fall back to the URL" shape and cross-checked with every scraper that reads kwargs["url"]. That gave these 8 (plus Charnwood, which has the same bug and is fixed by contributor PR #2323 in the November release PR). The other hits were false positives: variables computed internally, or url used as the page to open rather than a stored UPRN link. Huntingdon already had the correct structure and is the model I followed.

Fix

Use the UPRN when given, else the stored URL, and raise the clear Invalid UPRN error only when neither exists:

if user_uprn:
    url = f"...?uprn={user_uprn}"
elif kwargs.get("url"):
    url = kwargs.get("url")          # backwards compatibility
else:
    check_uprn(user_uprn)            # clear "Invalid UPRN" error

check_uprn() itself stays strict, as #2291 intended, for the ~180 scrapers where a UPRN really is required.

This touches 8 scraper files plus one test file. That's over the usual 3-file guideline, but it's one regression and the same 5-line change in each, so splitting it would only multiply releases.

Tests

test_legacy_url_fallback.py is parametrized over all eight and covers three cases each: only a legacy URL (also with uprn=None and uprn=""), a UPRN winning over a stored URL, and neither (a clear error, with no request made).

I checked the tests detect the bug: run against the unfixed scrapers from master, exactly the 8 legacy-URL cases fail (one per council) and the other 16 pass.

Test plan

• make unit-tests on Python 3.14.10: 519 passed. The 7 failures are the same as master's: 3 sandbox-only network tests, plus 4 already fixed by test: stop the Orkney and Broxbourne tests failing on the calendar and the network #2341.
• The new tests fail on master's scrapers (8) and pass with the fix (24).
• Commit passes commitlint with the same rules as lint.yml.
• This PR's CI live-tests all eight councils with their uprn. That exercises the normal path but not the legacy path, which needs a real stored-URL config. Whoever can reproduce (the Error getting identifier: Invalid UPRN. Please check the provided UPRN. #2294 reporters) should confirm after release.

This is a fix:, so merging it triggers a release.

Targets #2294. I'm deliberately not using Fixes: the logic is covered by tests, but only a real legacy-URL install can confirm it, so the issue should stay open until a reporter does.

---

---

Generated by Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 45 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: a80a6a9b-44ee-484c-a308-54eef9f9a3f6

Commits

Reviewing files that changed from the base of the PR and between 0d1b26a and 7693cc6.

Files selected for processing: 9

• uk_bin_collection/tests/test_legacy_url_fallback.py
• uk_bin_collection/uk_bin_collection/councils/CheshireEastCouncil.py
• uk_bin_collection/uk_bin_collection/councils/DartfordBoroughCouncil.py
• uk_bin_collection/uk_bin_collection/councils/EastbourneBoroughCouncil.py
• uk_bin_collection/uk_bin_collection/councils/GlasgowCityCouncil.py
• uk_bin_collection/uk_bin_collection/councils/HerefordshireCouncil.py
• uk_bin_collection/uk_bin_collection/councils/LewesDistrictCouncil.py
• uk_bin_collection/uk_bin_collection/councils/LiverpoolCityCouncil.py
• uk_bin_collection/uk_bin_collection/councils/NewhamCouncil.py

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
8	1	7	0

View the top 1 failed test(s) by shortest run time

uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[DartfordBoroughCouncil]

Stack Traces | 138s run time

self = <HTTPSConnection(host='windmz.dartford.gov.uk', port=443) at 0x7777583e4410>

    def _new_conn(self) -> socket.socket:
        """Establish a socket connection and set nodelay settings on it.
    
        :return: New socket connection.
        """
        try:
>           sock = connection.create_connection(
                (self._dns_host, self.port),
                self.timeout,
                source_address=self.source_address,
                socket_options=self.socket_options,
            )

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14........./site-packages/urllib3/connection.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.../urllib3/util/connection.py:85: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

address = ('windmz.dartford.gov.uk', 443), timeout = None, source_address = None
socket_options = [(6, 1, 1)]

    def create_connection(
        address: tuple[str, int],
        timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
        source_address: tuple[str, int] | None = None,
        socket_options: _TYPE_SOCKET_OPTIONS | None = None,
    ) -> socket.socket:
        """Connect to *address* and return the socket object.
    
        Convenience function.  Connect to *address* (a 2-tuple ``(host,
        port)``) and return the socket object.  Passing the optional
        *timeout* parameter will set the timeout on the socket instance
        before attempting to connect.  If no *timeout* is supplied, the
        global default timeout setting returned by :func:`socket.getdefaulttimeout`
        is used.  If *source_address* is set it must be a tuple of (host, port)
        for the socket to bind as a source address before making the connection.
        An host of '' or port 0 tells the OS to use the default.
        """
    
        host, port = address
        if host.startswith("["):
            host = host.strip("[]")
        err = None
    
        # Using the value from allowed_gai_family() in the context of getaddrinfo lets
        # us select whether to work with IPv4 DNS records, IPv6 records, or both.
        # The original create_connection function always returns all records.
        family = allowed_gai_family()
    
        try:
            host.encode("idna")
        except UnicodeError:
            raise LocationParseError(f"'{host}', label empty or too long") from None
    
        for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
            af, socktype, proto, canonname, sa = res
            sock = None
            try:
                sock = socket.socket(af, socktype, proto)
    
                # If provided, set socket level options before connecting.
                _set_socket_options(sock, socket_options)
    
                if timeout is not _DEFAULT_TIMEOUT:
                    sock.settimeout(timeout)
                if source_address:
                    sock.bind(source_address)
>               sock.connect(sa)
E               TimeoutError: [Errno 110] Connection timed out

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.../urllib3/util/connection.py:73: TimeoutError

The above exception was the direct cause of the following exception:

self = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7777583e4190>
method = 'GET', url = '/ufs/WS_CHECK_COLLECTIONS.eb?UPRN=100060861698'
body = None
headers = {'User-Agent': 'uk-bin-collection/0.173.5 (+https://github.com/robbrad/UKBinCollectionData)', 'Accept-Encoding': 'gzip, deflate, zstd', 'Accept': '*/*', 'Connection': 'keep-alive'}
retries = Retry(total=0, connect=None, read=False, redirect=None, status=None)
redirect = False, assert_same_host = False
timeout = Timeout(connect=None, read=None, total=None), pool_timeout = None
release_conn = False, chunked = False, body_pos = None, preload_content = False
decode_content = False, response_kw = {}

    def urlopen(  # type: ignore[override]
        self,
        method: str,
        url: str,
        body: _TYPE_BODY | None = None,
        headers: typing.Mapping[str, str] | None = None,
        retries: Retry | bool | int | None = None,
        redirect: bool = True,
        assert_same_host: bool = True,
        timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
        pool_timeout: int | None = None,
        release_conn: bool | None = None,
        chunked: bool = False,
        body_pos: _TYPE_BODY_POSITION | None = None,
        preload_content: bool = True,
        decode_content: bool = True,
        **response_kw: typing.Any,
    ) -> BaseHTTPResponse:
        """
        Get a connection from the pool and perform an HTTP request. This is the
        lowest level call for making a request, so you'll need to specify all
        the raw details.
    
        .. note::
    
           More commonly, it's appropriate to use a convenience method
           such as :meth:`request`.
    
        .. note::
    
           `release_conn` will only behave as expected if
           `preload_content=False` because we want to make
           `preload_content=False` the default behaviour someday soon without
           breaking backwards compatibility.
    
        :param method:
            HTTP request method (such as GET, POST, PUT, etc.)
    
        :param url:
            The URL to perform the request on.
    
        :param body:
            Data to send in the request body, either :class:`str`, :class:`bytes`,
            an iterable of :class:`str`/:class:`bytes`, or a file-like object.
    
        :param headers:
            Dictionary of custom headers to send, such as User-Agent,
            If-None-Match, etc. If None, pool headers are used. If provided,
            these headers completely replace any pool-specific headers.
    
        :param retries:
            Configure the number of retries to allow before raising a
            :class:`~urllib3.exceptions.MaxRetryError` exception.
    
            If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a
            :class:`~urllib3.util.retry.Retry` object for fine-grained control
            over different types of retries.
            Pass an integer number to retry connection errors that many times,
            but no other types of errors. Pass zero to never retry.
    
            If ``False``, then retries are disabled and any exception is raised
            immediately. Also, instead of raising a MaxRetryError on redirects,
            the redirect response will be returned.
    
        :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.
    
        :param redirect:
            If True, automatically handle redirects (status codes 301, 302,
            303, 307, 308). Each redirect counts as a retry. Disabling retries
            will disable redirect, too.
    
        :param assert_same_host:
            If ``True``, will make sure that the host of the pool requests is
            consistent else will raise HostChangedError. When ``False``, you can
            use the pool on an HTTP proxy and request foreign hosts.
    
        :param timeout:
            If specified, overrides the default timeout for this one
            request. It may be a float (in seconds) or an instance of
            :class:`urllib3.util.Timeout`.
    
        :param pool_timeout:
            If set and the pool is set to block=True, then this method will
            block for ``pool_timeout`` seconds and raise EmptyPoolError if no
            connection is available within the time period.
    
        :param bool preload_content:
            If True, the response's body will be preloaded into memory.
    
        :param bool decode_content:
            If True, will attempt to decode the body based on the
            'content-encoding' header.
    
        :param release_conn:
            If False, then the urlopen call will not release the connection
            back into the pool once a response is received (but will release if
            you read the entire contents of the response such as when
            `preload_content=True`). This is useful if you're not preloading
            the response's content immediately. You will need to call
            ``r.release_conn()`` on the response ``r`` to return the connection
            back into the pool. If None, it takes the value of ``preload_content``
            which defaults to ``True``.
    
        :param bool chunked:
            If True, urllib3 will send the body using chunked transfer
            encoding. Otherwise, urllib3 will send the body using the standard
            content-length form. Defaults to False.
    
        :param int body_pos:
            Position to seek to in file-like body in the event of a retry or
            redirect. Typically this won't need to be set because urllib3 will
            auto-populate the value when needed.
        """
        # Ensure that the URL we're connecting to is properly encoded
        if url.startswith("/"):
            # URLs starting with / are inherently schemeless.
            url = to_str(_encode_target(url))
            destination_scheme = None
        else:
            parsed_url = parse_url(url)
            destination_scheme = parsed_url.scheme
            url = to_str(parsed_url._replace(fragment=None).url)
    
        if headers is None:
            headers = self.headers
    
        if not isinstance(retries, Retry):
            retries = Retry.from_int(retries, redirect=redirect, default=self.retries)
    
        if release_conn is None:
            release_conn = preload_content
    
        # Check host
        if assert_same_host and not self.is_same_host(url):
            raise HostChangedError(self, url, retries)
    
        conn = None
    
        # Track whether `conn` needs to be released before
        # returning/raising/recursing. Update this variable if necessary, and
        # leave `release_conn` constant throughout the function. That way, if
        # the function recurses, the original value of `release_conn` will be
        # passed down into the recursive call, and its value will be respected.
        #
        # See issue #651 [1] for details.
        #
        # [1] <https://github..../urllib3/issues/651>
        release_this_conn = release_conn
    
        http_tunnel_required = connection_requires_http_tunnel(
            self.proxy, self.proxy_config, destination_scheme
        )
    
        # Merge the proxy headers. Only done when not using HTTP CONNECT. We
        # have to copy the headers dict so we can safely change it without those
        # changes being reflected in anyone else's copy.
        if not http_tunnel_required:
            headers = headers.copy()  # type: ignore[attr-defined]
            headers.update(self.proxy_headers)  # type: ignore[union-attr]
    
        # Must keep the exception bound to a separate variable or else Python 3
        # complains about UnboundLocalError.
        err = None
    
        # Keep track of whether we cleanly exited the except block. This
        # ensures we do proper cleanup in finally.
        clean_exit = False
    
        # Rewind body position, if needed. Record current position
        # for future rewinds in the event of a redirect/retry.
        body_pos = set_file_position(body, body_pos)
    
        timeout_obj = self._get_timeout(timeout)
        try:
            # Request a connection from the queue.
            conn = self._get_conn(timeout=pool_timeout)
            conn.timeout = timeout_obj.connect_timeout  # type: ignore[assignment]
    
            # Is this a closed/new connection that requires CONNECT tunnelling?
            if self.proxy is not None and http_tunnel_required and conn.is_closed:
                try:
                    self._prepare_proxy(conn)
                except (BaseSSLError, OSError, SocketTimeout) as e:
                    self._raise_timeout(
                        err=e, url=self.proxy.url, timeout_value=conn.timeout
                    )
                    raise
    
            # If we're going to release the connection in ``finally:``, then
            # the response doesn't need to know about the connection. Otherwise
            # it will also try to release it and we'll have a double-release
            # mess.
            response_conn = conn if not release_conn else None
    
            # Make the request on the HTTPConnection object
>           response = self._make_request(
                conn,
                method,
                url,
                timeout=timeout_obj,
                body=body,
                headers=headers,
                chunked=chunked,
                retries=retries,
                response_conn=response_conn,
                preload_content=preload_content,
                decode_content=decode_content,
                **response_kw,
            )

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.............../site-packages/urllib3/connectionpool.py:793: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.............../site-packages/urllib3/connectionpool.py:494: in _make_request
    raise new_e
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.............../site-packages/urllib3/connectionpool.py:470: in _make_request
    self._validate_conn(conn)
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.............../site-packages/urllib3/connectionpool.py:1125: in _validate_conn
    conn.connect()
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14........./site-packages/urllib3/connection.py:827: in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <HTTPSConnection(host='windmz.dartford.gov.uk', port=443) at 0x7777583e4410>

    def _new_conn(self) -> socket.socket:
        """Establish a socket connection and set nodelay settings on it.
    
        :return: New socket connection.
        """
        try:
            sock = connection.create_connection(
                (self._dns_host, self.port),
                self.timeout,
                source_address=self.source_address,
                socket_options=self.socket_options,
            )
        except socket.gaierror as e:
            raise NameResolutionError(self.host, self, e) from e
        except SocketTimeout as e:
>           raise ConnectTimeoutError(
                self,
                f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
            ) from e
E           urllib3.exceptions.ConnectTimeoutError: (<HTTPSConnection(host='windmz.dartford.gov.uk', port=443) at 0x7777583e4410>, 'Connection to windmz.dartford.gov.uk timed out. (connect timeout=None)')

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14........./site-packages/urllib3/connection.py:248: ConnectTimeoutError

The above exception was the direct cause of the following exception:

self = <requests.adapters.HTTPAdapter object at 0x7777585bfd90>
request = <PreparedRequest [GET]>, stream = False, timeout = None, verify = True
cert = None, proxies = OrderedDict()

    def send(
        self,
        request: PreparedRequest,
        stream: bool = False,
        timeout: _t.TimeoutType = None,
        verify: _t.VerifyType = True,
        cert: _t.CertType = None,
        proxies: dict[str, str] | None = None,
    ) -> Response:
        """Sends PreparedRequest object. Returns Response object.
    
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
            data before giving up, as a float, or a :ref:`(connect timeout,
            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple or urllib3 Timeout object
        :param verify: (optional) Either a boolean, in which case it controls whether
            we verify the server's TLS certificate, or a string, in which case it
            must be a path to a CA bundle to use
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
        :param proxies: (optional) The proxies dictionary to apply to the request.
        :rtype: requests.Response
        """
    
        assert _is_prepared(request)
    
        try:
            conn = self.get_connection_with_tls_context(
                request, verify, proxies=proxies, cert=cert
            )
        except LocationValueError as e:
            raise InvalidURL(e, request=request)
    
        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(
            request,
            stream=stream,
            timeout=timeout,
            verify=verify,
            cert=cert,
            proxies=proxies,
        )
    
        chunked = not (request.body is None or "Content-Length" in request.headers)
    
        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                resolved_timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError:
                raise ValueError(
                    f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                    f"or a single float to set both timeouts to the same value."
                )
        elif isinstance(timeout, TimeoutSauce):
            resolved_timeout = timeout
        else:
            resolved_timeout = TimeoutSauce(connect=timeout, read=timeout)
    
        try:
>           resp = conn.urlopen(
                method=request.method,
                url=url,
                body=request.body,  # type: ignore[arg-type]  # urllib3 stubs don't accept Iterable[bytes | str]
                headers=request.headers,  # type: ignore[arg-type]  # urllib3#3072
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=resolved_timeout,
                chunked=chunked,
            )

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14....../site-packages/requests/adapters.py:696: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.............../site-packages/urllib3/connectionpool.py:847: in urlopen
    retries = retries.increment(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Retry(total=0, connect=None, read=False, redirect=None, status=None)
method = 'GET', url = '/ufs/WS_CHECK_COLLECTIONS.eb?UPRN=100060861698'
response = None
error = ConnectTimeoutError(<HTTPSConnection(host='windmz.dartford.gov.uk', port=443) at 0x7777583e4410>, 'Connection to windmz.dartford.gov.uk timed out. (connect timeout=None)')
_pool = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7777583e4190>
_stacktrace = <traceback object at 0x777758a61180>

    def increment(
        self,
        method: str | None = None,
        url: str | None = None,
        response: BaseHTTPResponse | None = None,
        error: Exception | None = None,
        _pool: ConnectionPool | None = None,
        _stacktrace: TracebackType | None = None,
    ) -> Self:
        """Return a new Retry object with incremented retry counters.
    
        :param response: A response object, or None, if the server did not
            return a response.
        :type response: :class:`~urllib3.response.BaseHTTPResponse`
        :param Exception error: An error encountered during the request, or
            None if the response was received successfully.
    
        :return: A new ``Retry`` object.
        """
        if self.total is False and error:
            # Disabled, indicate to re-raise the error.
            raise reraise(type(error), error, _stacktrace)
    
        total = self.total
        if total is not None:
            total -= 1
    
        connect = self.connect
        read = self.read
        redirect = self.redirect
        status_count = self.status
        other = self.other
        cause = "unknown"
        status = None
        redirect_location = None
    
        if error and self._is_connection_error(error):
            # Connect retry?
            if connect is False:
                raise reraise(type(error), error, _stacktrace)
            elif connect is not None:
                connect -= 1
    
        elif error and self._is_read_error(error):
            # Read retry?
            if read is False or method is None or not self._is_method_retryable(method):
                raise reraise(type(error), error, _stacktrace)
            elif read is not None:
                read -= 1
    
        elif error:
            # Other retry?
            if other is not None:
                other -= 1
    
        elif response and response.get_redirect_location():
            # Redirect retry?
            if redirect is not None:
                redirect -= 1
            cause = "too many redirects"
            response_redirect_location = response.get_redirect_location()
            if response_redirect_location:
                redirect_location = response_redirect_location
            status = response.status
    
        else:
            # Incrementing because of a server error like a 500 in
            # status_forcelist and the given method is in the allowed_methods
            cause = ResponseError.GENERIC_ERROR
            if response and response.status:
                if status_count is not None:
                    status_count -= 1
                cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status)
                status = response.status
    
        history = self.history + (
            RequestHistory(method, url, error, status, redirect_location),
        )
    
        new_retry = self.new(
            total=total,
            connect=connect,
            read=read,
            redirect=redirect,
            status=status_count,
            other=other,
            history=history,
        )
    
        if new_retry.is_exhausted():
            reason = error or ResponseError(cause)
>           raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E           urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='windmz.dartford.gov.uk', port=443): Max retries exceeded with url: /ufs/WS_CHECK_COLLECTIONS.eb?UPRN=100060861698 (Caused by ConnectTimeoutError(<HTTPSConnection(host='windmz.dartford.gov.uk', port=443) at 0x7777583e4410>, 'Connection to windmz.dartford.gov.uk timed out. (connect timeout=None)'))

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.../urllib3/util/retry.py:555: MaxRetryError

During handling of the above exception, another exception occurred:

fixturefunc = <function scrape_step at 0x7777586d2090>
request = <FixtureRequest for <Function test_scenario_outline[DartfordBoroughCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x77775876eba0>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if inspect.isgeneratorfunction(fixturefunc):
            fixturefunc = cast(Callable[..., Generator[FixtureValue]], fixturefunc)
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)
                             ^^^^^^^^^^^^^^^^^^^^^

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14.../site-packages/_pytest/fixtures.py:1005: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
                           ^^^^^^^^^^^^^^^^^
uk_bin_collection/uk_bin_collection/collect_data.py:109: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:130: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
uk_bin_collection/uk_bin_collection/get_bin_data.py:71: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
uk_bin_collection/uk_bin_collection/get_bin_data.py:94: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../uk_bin_collection/councils/DartfordBoroughCouncil.py:32: in parse_data
    page = requests.get(url, headers=headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14....../site-packages/requests/api.py:87: in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14....../site-packages/requests/api.py:71: in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14....../site-packages/requests/sessions.py:651: in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14....../site-packages/requests/sessions.py:784: in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <requests.adapters.HTTPAdapter object at 0x7777585bfd90>
request = <PreparedRequest [GET]>, stream = False, timeout = None, verify = True
cert = None, proxies = OrderedDict()

    def send(
        self,
        request: PreparedRequest,
        stream: bool = False,
        timeout: _t.TimeoutType = None,
        verify: _t.VerifyType = True,
        cert: _t.CertType = None,
        proxies: dict[str, str] | None = None,
    ) -> Response:
        """Sends PreparedRequest object. Returns Response object.
    
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param stream: (optional) Whether to stream the request content.
        :param timeout: (optional) How long to wait for the server to send
            data before giving up, as a float, or a :ref:`(connect timeout,
            read timeout) <timeouts>` tuple.
        :type timeout: float or tuple or urllib3 Timeout object
        :param verify: (optional) Either a boolean, in which case it controls whether
            we verify the server's TLS certificate, or a string, in which case it
            must be a path to a CA bundle to use
        :param cert: (optional) Any user-provided SSL certificate to be trusted.
        :param proxies: (optional) The proxies dictionary to apply to the request.
        :rtype: requests.Response
        """
    
        assert _is_prepared(request)
    
        try:
            conn = self.get_connection_with_tls_context(
                request, verify, proxies=proxies, cert=cert
            )
        except LocationValueError as e:
            raise InvalidURL(e, request=request)
    
        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(
            request,
            stream=stream,
            timeout=timeout,
            verify=verify,
            cert=cert,
            proxies=proxies,
        )
    
        chunked = not (request.body is None or "Content-Length" in request.headers)
    
        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                resolved_timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError:
                raise ValueError(
                    f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                    f"or a single float to set both timeouts to the same value."
                )
        elif isinstance(timeout, TimeoutSauce):
            resolved_timeout = timeout
        else:
            resolved_timeout = TimeoutSauce(connect=timeout, read=timeout)
    
        try:
            resp = conn.urlopen(
                method=request.method,
                url=url,
                body=request.body,  # type: ignore[arg-type]  # urllib3 stubs don't accept Iterable[bytes | str]
                headers=request.headers,  # type: ignore[arg-type]  # urllib3#3072
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=resolved_timeout,
                chunked=chunked,
            )
    
        except (ProtocolError, OSError) as err:
            raise ConnectionError(err, request=request)
    
        except MaxRetryError as e:
            if isinstance(e.reason, ConnectTimeoutError):
                # TODO: Remove this in 3.0.0: see #2811
                if not isinstance(e.reason, NewConnectionError):
>                   raise ConnectTimeout(e, request=request)
E                   requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='windmz.dartford.gov.uk', port=443): Max retries exceeded with url: /ufs/WS_CHECK_COLLECTIONS.eb?UPRN=100060861698 (Caused by ConnectTimeoutError(<HTTPSConnection(host='windmz.dartford.gov.uk', port=443) at 0x7777583e4410>, 'Connection to windmz.dartford.gov.uk timed out. (connect timeout=None)'))

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.14/lib/python3.14....../site-packages/requests/adapters.py:717: ConnectTimeout

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}