Keep the HTTP listing cache independent of the first caller's detail

[feiiiiii5]

Summary

With use_listings_cache=True, the first caller's detail argument decided the type that every later caller on that URL received. One ls(url, detail=False) was enough to leave a list of strings in the cache, after which info, find and walk — which all go through ls(detail=True) — got strings where dicts are required.

The bug

Both HTTP files cache the listing under the URL alone, storing whichever shape the caller happened to ask for:

The current cache decision

The fix

if self.use_listings_cache and url in self.dircache:
    out = self.dircache[url]
else:
    out = self._ls_real(url, detail=detail, **kwargs)
    self.dircache[url] = out
return out

detail is in neither the cache key nor the cached value, so it is decided once and frozen. This is not an exotic call order — info, find, walk, expand_path and cat all funnel through ls(detail=True) (spec.py:792, spec.py:528), while ordinary user code calls ls(detail=False) on the same URL. Whichever ran first decided for both.

Measured against master at 778f956, with _ls_real stubbed so no network is involved:

Measured, both orders

detail=True first, then detail=False:
  ls(url)               -> [{'name': ..., 'size': 1, 'type': 'file'}, ...]
  ls(url, detail=False) -> [{'name': ...}, ...]     <- asked for names, got dicts

detail=False first, then the detailed consumers:
  find(url)  -> TypeError: string indices must be integers, not 'str'
  info(a.txt)-> FileNotFoundError

With use_listings_cache=False the same sequence is correct, so the trigger is exactly the documented cache option.

The same three lines are in http_sync.py:340 — the requests-based HTTPFileSystem had the identical defect, so this is one bug written twice. Both are fixed here.

The fix

Store the detailed shape and project down for a detail=False caller:

if self.use_listings_cache and url in self.dircache:
    out = self.dircache[url]
else:
    out = self._ls_real(url, detail=True, **kwargs)
    self.dircache[url] = out
if detail:
    return out
return sorted(entry["name"] for entry in out)

One cache entry per URL as before, no extra requests, and the cached shape can no longer depend on call order.

detail=True is safe to make canonical because _ls_real always builds a list of dicts with a "name" key on that path — including the trailing-slash redirect, where it recurses with detail=False and then re-wraps the names into dicts before returning.

Validation

6 new tests — the same three cases against each implementation:

Red/green

pytest fsspec/implementations/tests/test_http.py fsspec/implementations/tests/test_http_sync.py -k list_cache
  without this change   6 failed, 12 passed
      test_list_cache_detail_shape_is_independent_of_the_first_caller[True]      FAILED
      test_list_cache_detail_shape_is_independent_of_the_first_caller[False]     FAILED
      test_list_cache_detail_false_does_not_poison_the_detailed_consumers        FAILED
      (and the same three in test_http_sync.py)
  with this change    18 passed

All 6 are red without the change. The 12 that pass either way are the pre-existing listing-cache tests, and they matter here: test_list_cache, test_list_cache_with_expiry_time_cached, ..._purged, ..._max_paths and ..._reuse only ever drive h.glob(...), which reaches ls exclusively with detail=True. Nothing in the suite exercised a mixed sequence, which is why this survived.

The parametrised test asserts the type of each call rather than a snapshot, in both orders, so a future change cannot reintroduce order-dependence in either direction. A second test checks that the projected names are exactly the ones the detailed listing carries, and that info/find/glob still work after a detail=False listing.

Full suite

pytest fsspec -q --ignore=fsspec/implementations/tests/test_github_paths.py
  base          1976 passed, 249 skipped, 2 xfailed
  with change   1982 passed, 249 skipped, 2 xfailed

The +6 is exactly the new tests; no failures either side. I skipped test_github_paths.py because it needs network access for the GitHub API. The tests use the repo's existing local server fixture, so nothing here reaches the internet.

ruff check and ruff format --check are clean on all four files. One pre-existing FURB110 finding in http_sync.py:149 is unrelated and untouched.

A separate issue I did not touch

self.dircache[url] = out also runs when use_listings_cache is False, and HTTPFileSystem.use_listings_cache (default False) and DirCache.use_listings_cache (default True) are different objects set from different places — so with default options every ls() writes an entry into an unbounded dict that is never consulted. That is a memory-growth issue rather than a correctness one, it would widen this diff, and picking a policy for it is your call, so I have left it alone.