Repository navigation
Close the catalog data gap with AllTrails and Strava #1002
Description
Problem
Measured against AllTrails and Strava, our gap is not the feature list — it is that the catalog holds 596,556 trails and almost none of them carry the data someone opens a trail page to read.
What was measured
400 trails sampled live from production on 25 Sep 2026, at offsets 0, 60k, 200k and 400k of the catalog — a straight cut through the middle, not a worst case.
| Field | Coverage |
|---|---|
| Elevation gain | 0 / 400 |
| Star rating | 0 / 400 |
| Any review | 0 / 400 |
| Non-stock photo | 0 / 400 |
| Dogs policy | 1 / 400 |
| Surface | 233 / 400 (58%) |
Difficulty across the sample: 310 easy, 21 moderate, 69 hard — graded from length alone, because ascent is missing.
A second sample of the first 200 in the default ordering was barely better: 8/200 with elevation, 5/200 with a real photo.
What that looks like on a page
/trail/105902 (Theodore Solomons Trail) today: "Illustrative photo", elevation gain "Not recorded", a generated description ending "Mapped by the OpenStreetMap community", 0 reviews, 0 records. /trails opens sorted longest-first, so the first screen is the North Country Trail at 1,161 miles, the Grand Enchantment at 1,036 and the Iditarod at 996 — every one of them graded HARD.
Sources for the comparison
- Strava's own feature and subscription page.
- AllTrails' help centre (Plus and Peak benefit articles, feature index) — alltrails.com refuses automated browsers, so their support site stands in for the product tour.
- Wildloop from this repository and from the live API.
The work
Four fixes, in order. The first two are the same problem, and three of the four run against machinery that already exists here.
- Backfill elevation gain for the whole trail catalog (0 of 400 sampled have it) #1003 — Backfill elevation gain for the whole catalog
- Grade trail difficulty on ascent, not distance alone #1004 — Grade difficulty on ascent, not distance alone
- Open the trail catalog on what is near and well rated, not the longest trails #1005 — Open the catalog on what is near and well rated, not the longest trails
- Put real photos on the trails people actually open (0 of 400 sampled have one) #1006 — Put real photos on the trails people actually open
Backlog behind that
Not filed separately yet — say if any of these should be.
What AllTrails has and we do not: conditions forecast (temperature, precipitation, snow depth, air quality) beside our community reports; offline download of a park or a drawn area rather than one trail; waypoints and off-route alerts while recording; park pages; points of interest; custom and shareable lists; sending routes out to Garmin and COROS; printed maps. Peak adds 3D maps, a multi-day route assistant, smart re-routing, plant/insect identification and community heatmaps.
What Strava has and we do not: goals of any kind — the concept does not exist in the codebase; progress and fitness over time; AI activity summaries; finish-time predictions; routes generated from recorded activity volume; Local Legend.
What neither of them has: the territory game, and a trail catalog sitting next to clubs, events and challenges. That is the reason to exist — and it only works on top of a catalog people trust.
Activity
feat(trails): open the catalog on day hikes, not thousand-mile thru-h…
glennmichael123 commentedon Sep 29, 2026
The catalog is mostly not trails
Measured while working #1005. This reframes the epic: enriching 596,556 rows is the wrong target, because most of them are not things anybody would set out to walk.
1,200 trails, six offsets spanning the catalog (0 → 500,000), default ordering, 30 Sep 2026.
Median trail length: 0.54 miles.
The catalog is imported from OpenStreetMap, where a way is whatever a mapper drew between two junctions. So two thirds of the rows are path segments, not trails — and one real trail is often several rows. In the first four rows of the default listing today,
Gwt Francis Peak To Morgan Co.appears twice (13.88 mi and 1.35 mi) andCanjilon Mountainappears twice (6.72 mi and 0.45 mi).23.2% of rows share a name with another row in the sample — 1,029 distinct names across 1,200 rows.
Continental Divide NST,Link TrailandRed Traileach appear 6 times.Why this changes the plan
The four fixes here assume the catalog is a list of trails that needs better data. It is closer to a list of path geometry that needs assembling into trails first.
Red Trailrows gets it is unanswerable. Merging first makes the target perhaps a quarter the size and each row worth a photo.Same for the third-party comparison: AllTrails does not list six
Red Trailentries. Their catalog is curated routes, ours is raw OSM ways. That is the difference behind "they look higher quality than us", more than any single missing field.Suggested addition to the work list
Partial mitigation already landed
d5ec7c1 (#1005) adds a
browse_bandgenerated column so the catalog opens on day hikes rather than on thru-hikes or fragments. That fixes the first screen without fixing the data — fragments are demoted, not merged.Separate bug found while sampling
sort=namereturns HTTP 520 past offset ~55,000. Cloudflare 520 with the origin timing out at ~12s.Not reproducible on the default sort,
sort=distanceorsort=longestat the same offset — so ordering 596k rows bynamehas no usable index. The Name chip in the catalog breaks for anyone paging deep. Worth its own issue; say the word and I will file it.Method
Live production API,
limit=200at offsets 0 / 70,000 / 150,000 / 260,000 / 380,000 / 500,000, de-duplicated by id. Default ordering, which at the time of sampling was stillnational_trail DESCand therefore length-neutral — d5ec7c1 had not deployed. Elevation 0.1% and rating 0.0% in the same sample, consistent with the 25 Sep numbers above.feat(trails): work out which catalog rows are pieces of one trail
feat(trails): make the catalog data gap measurable more than once
feat(trails): decide which way fragments fold into which trail
feat(catalog): list a trail once, and send its pieces' URLs to it
chrisbbreuer commentedon Oct 4, 2026
Merge-or-suppress fragments is in (48a2caf, 263cca7, a6fbd8c). Pieces are recorded in a new
trail_partstable and hidden from the catalog, search, suggestions, the sitemap and coverage counts. Their pages 301 to the trail they belong to, and nothing that points at a trail id changes.A row folds when it shares an exact name and country with a row it physically joins (the existing
clusterRunsgeometry check) and starts within 3 miles of it. The kept row is the NPS/USFS record first, then the longest. Rows with reviews or photos, and hand-entered rows, are never folded.On 8,368 production rows around LA, Boulder and Garmisch, 35% fold: LA 30.5%, Boulder 36.0%, Garmisch 37.6%. Rows listed under 0.4 mi drop from 3,922 to 1,981. Temescal Ridge Trail goes from 10 rows to 1, Mesa Trail from 17 to 3.
trails:fold-fragmentsruns nightly from 10:40 UTC, and the ingest folds each shard as it writes. Kept trails still carry their own distance and line, so merging the pieces' lines into the kept trail is the natural follow-up.feat(trails): list a folded trail at the length of the whole trail
glennmichael123 commentedon Oct 6, 2026
Measured again, 6 Oct 2026 — the first run comparable to the baseline
400 trails, sampled evenly across the id space rather than by offset, read-only through the public API. Ids come from the sitemaps, so every sampled id is a trail that exists. Same method as
buddy trails:catalog-coverage(#1009), which samples by id for the reason the top of this issue could not be re-run: the default ordering changed under #1005, so an offset no longer selects the same trails.Sample spans ids 21 → 1,430,869. Catalog total reported by the API: 427,023.
Difficulty across the sample: 326 easy, 60 moderate, 14 hard — against 310 / 21 / 69 in September. Neither the difficulty split nor the surface figure is a trend: the two samples are different trails, which is the whole reason the measurement moved to ids.
The elevation backfill is still running
elevation_checked_aton the sampled rows carries timestamps from 25 Sep through today, at roughly 11,000–17,000 trails a day with some idle days. That is about 79,000 of 427,023 checked so far, so on this rate the catalog is covered around early November.Of the trails it has reached, 66% came back with a real value (49 of 74). The rest were checked and had nothing to give. So the honest projection for #1003 when it completes is elevation on about two thirds of the catalog, from zero in September — and 82% of the catalog has not been looked at yet, which is why the sampled figure is 12% rather than 66%.
What has not moved, and why
Two things found while measuring
glennmichael123 commentedon Oct 6, 2026
Correction to the comment above: nothing was removed. The 596,556 → 427,023 drop is fragment folding (263cca7), not deletion.
trail_partsrecords each row that is a piece of another trail. A piece is hidden rather than deleted — its row stays, so saved trails, activities, reviews, photos, efforts, segments, page views and plans keep pointing at it, and deleting itstrail_partsrow lists it again. The catalog filters withid NOT IN (SELECT trail_id FROM trail_parts), and the API'smeta.totalis literallyCOUNT(trails) − COUNT(trail_parts), so the ~170,000 "missing" rows are OSM way fragments now presented as parts of the trail they belong to.buddy trails:fold-fragments --dry-runreports the real figure.That makes the September baseline worse to compare against than the comment above allows for, in a way that matters for reading it: that sample was taken by offset over the unfolded catalog, so some of its 400 trails were way fragments — short, flat, unnamed pieces. They are the rows most likely to be graded easy and least likely to carry an ascent, which flatters the 310-easy split and part of the 0/400 elevation figure. The 6 Oct sample is drawn from the sitemaps, which exclude pieces, so it measures what a visitor can actually browse.
The elevation reading stands on its own either way: it comes from
elevation_checked_aton the sampled rows, not from the comparison.