A production-grade skill for querying, traversing, caching, and presenting
Hacker News public data through the Firebase v0 API.
Validation, retries, timeouts, safe HTML rendering, and bounded comment-tree traversal.
- Overview
- Why This Skill
- Architecture
- Features
- Installation
- Quick Start
- API Reference
- Bounded Comment-Tree Traversal
- Robustness & Error Handling
- Security & HTML Sanitization
- Caching Strategy
- Usage Examples
- Project Structure
- Best Practices
- Limitations
- Contributing
- License
Hacker News Advanced Insight And Analytics Engine is a technical skill that wraps the public Hacker News Firebase API v0 into a clean, resilient, and well-documented toolkit.
It is designed for:
- Agents and LLM tools that need reliable access to HN data.
- Data engineers building story/comment analytics.
- Dashboard developers that need bounded, cached, and safe fetches.
- Researchers who want reproducible access to HN snapshots.
The base URL is:
https://hacker-news.firebaseio.com/v0
There is currently no documented rate limit, but this skill still enforces retries, exponential backoff, timeouts, and caching to remain a good citizen of the API.
The raw HN API is a direct dump of in-memory data structures. It is awkward by design:
- Counting comments on a story means traversing the whole tree.
- Listing children means loading the parent, then loading each child.
- No pagination, no projections, no filtering.
- HTML fields are raw and must be sanitized before rendering.
This skill solves those problems by providing:
- Validated, typed wrappers around every endpoint.
- A bounded comment-tree traversal (no runaway recursion).
- A clear HTML sanitization policy.
- A documented caching strategy.
- A retry + backoff + timeout network layer.
- Deterministic error types for upstream failures.
The engine follows a layered architecture:
| Layer | Responsibility |
|---|---|
| HN Firebase API v0 | The upstream data source — direct in-memory dump. |
| HNClient | HTTP layer with retries (4×), exponential backoff, 10s timeout, and a structured error hierarchy. |
| Modules | items (fetch + tree traversal), users (profiles), lists (story IDs + updates + maxitem), sanitize (HTML allow-list), cache (TTL with graceful degradation). |
| Output | Typed dictionaries, safe HTML, structured exceptions, and ready-to-use reports. |
- ✅ Full coverage of the v0 endpoints:
item,user,maxitem,topstories,newstories,beststories,askstories,showstories,jobstories,updates. - ✅ Strict input validation (
intvsbool, positive IDs, non-empty usernames). - ✅ Case-sensitive username handling with URL encoding.
- ✅ Graceful handling of
nullresponses (deleted or unknown items/users). - ✅ Bounded comment-tree traversal (
max_depth,max_nodes). - ✅ HTML sanitization with an explicit allow-list.
- ✅ Retry with exponential backoff for 429/5xx and transient network errors.
- ✅ Structured
HNErrorexception hierarchy with clear context. - ✅ Optional TTL caching with
cachetools(graceful no-op fallback). - ✅ Ready for integration into agents, pipelines, and dashboards.
pip install requests bleachOptional, for caching:
pip install cachetoolsOr install the full development bundle:
pip install -e ".[dev,cache]"from hn_engine import get_item, get_story_ids, get_comment_tree
# Fetch a single story
story = get_item(8863)
print(story["title"])
print(story["score"], "points by", story["by"])
# Fetch the current top story IDs (up to 500)
top_ids = get_story_ids("top")
print(len(top_ids), "top stories available")
# Bounded comment-tree traversal
tree = get_comment_tree(8863, max_depth=2, max_nodes=50)
print(len(tree), "nodes fetched")Stories, comments, jobs, Ask HNs and polls are all items, identified by unique integer IDs.
Endpoint: /v0/item/<id>.json
| Field | Description |
|---|---|
id |
Required. The item's unique id. |
deleted |
true if the item is deleted. |
type |
One of job, story, comment, poll, pollopt. |
by |
Username of the author. |
time |
Creation date, Unix time. |
text |
Comment, story or poll text. HTML. |
dead |
true if the item is dead. |
parent |
Parent comment or story ID. |
poll |
Associated poll ID (for pollopts). |
kids |
Comment IDs, in ranked display order. |
url |
Story URL. |
score |
Story score, or pollopt votes. |
title |
Story, poll or job title. HTML. |
parts |
Related pollopt IDs, in display order. |
descendants |
Total comment count for stories or polls (not for comments). |
Example — story 8863:
{
"by": "dhouston",
"descendants": 71,
"id": 8863,
"kids": [8952, 9224, 8917, 8884, 8887, 8943, 8869, 8958, 9005, 9671],
"score": 111,
"time": 1175714200,
"title": "My YC app: Dropbox - Throw away your USB drive",
"type": "story",
"url": "http://www.getdropbox.com/u/2/screencast.html"
}Example — comment 2921983:
{
"by": "norvig",
"id": 2921983,
"kids": [2922097, 2922429, 2924562],
"parent": 2921506,
"text": "Aw shucks, guys ... you make me blush with your compliments.",
"time": 1314211127,
"type": "comment"
}Item types at a glance:
| Type | Has title | Has text | Has kids | Has url | Has parts |
|---|---|---|---|---|---|
| story | ✅ | ✅ | ✅ | ✅ | ❌ |
| ask | ✅ | ✅ | ✅ | ❌ | ❌ |
| job | ✅ | ✅ | ❌ | ✅ | ❌ |
| comment | ❌ | ✅ | ✅ | ❌ | ❌ |
| poll | ✅ | ✅ | ✅ | ❌ | ✅ |
| pollopt | ❌ | ✅ | ❌ | ❌ | ❌ |
Users are identified by case-sensitive usernames and live under /v0/user/. Only users with public activity are exposed.
Endpoint: /v0/user/<username>.json
| Field | Description |
|---|---|
id |
Required. Case-sensitive username. |
created |
Creation date, Unix time. |
karma |
User karma. |
about |
Optional self-description. HTML. |
submitted |
List of story, poll and comment IDs. |
Example — user jl:
{
"about": "This is a test",
"created": 1173923446,
"id": "jl",
"karma": 2937,
"submitted": [8265435, 8168423, 8090946, 8090326, 7699907]
}
⚠️ submittedcan be very large. Never fetch every submitted item blindly — apply pagination or filtering in your own code.
Endpoint: /v0/maxitem.json
Returns the largest item ID currently known. Useful for tailing new items backward.
9130260| Kind | Endpoint | Limit |
|---|---|---|
| top | /v0/topstories.json |
500 |
| new | /v0/newstories.json |
500 |
| best | /v0/beststories.json |
500 |
| ask | /v0/askstories.json |
200 |
| show | /v0/showstories.json |
200 |
| job | /v0/jobstories.json |
200 |
Each endpoint returns an array of IDs (not full items):
[9129911, 9129199, 9127761, 9128141, 9128264]Endpoint: /v0/updates.json
{
"items": [8423305, 8420805, 8423379],
"profiles": ["thefox", "mdda", "plinkplonk"]
}This is the recommended way to build an incremental sync loop:
- Poll
/v0/updates.jsonevery 30–60 seconds. - For each changed item ID, refresh the cached item.
- For each changed profile, refresh the cached user.
HN comment trees can be enormous — deeply nested, with hundreds or thousands of nodes. This engine provides get_comment_tree() with two hard bounds:
max_depth— Stop descending after N levels (0 = root only).max_nodes— Stop fetching after N total nodes.
Every returned node includes a _depth key so you know exactly where it sits in the tree.
from hn_engine import get_comment_tree, sanitize_html
tree = get_comment_tree(8863, max_depth=2, max_nodes=50)
for node_id, node in tree.items():
if node.get("type") != "comment":
continue
depth = node["_depth"]
text = sanitize_html(node.get("text", ""))
print(f"{' ' * depth}#{node_id}: {text[:80]}...")The engine enforces the following rules:
- Input validation —
item_idmust be a positiveint(booleans are explicitly rejected). - Retries — up to 4 attempts with exponential backoff on 429, 500, 502, 503, 504, and transient network errors.
- Timeouts — default 10 seconds per request.
- Deterministic errors — every failure raises
HNErrorwith a clear message including the failing URL and HTTP status. - Graceful null — missing or deleted items/users return
None, never throw. - Bounded traversal —
get_comment_treerequiresmax_depthandmax_nodes.
HNError (base)
├── ValidationError → bad input, not retryable
├── NetworkError → connection/timeout, retryable
├── RateLimitError → HTTP 429, retryable, carries retry_after
└── NotFoundError → HTTP 404 or null response, not retryable
Example:
from hn_engine.errors import HNError, RateLimitError
try:
story = get_item(8863)
except RateLimitError as exc:
print(f"Rate limited. Retry after {exc.retry_after}s")
except HNError as exc:
print(f"Failed: {exc}")text, title, and about fields contain raw HTML. Always sanitize them before display, logging, or embedding in a UI.
| Element | Attributes |
|---|---|
p, br |
— |
i, em |
— |
b, strong |
— |
a |
href, rel |
pre, code |
— |
ul, ol, li |
— |
blockquote |
— |
Protocols: http, https, mailto only.
from hn_engine import sanitize_html
clean = sanitize_html('<a href="http://example.com">link</a><script>evil()</script>')
# → '<a href="http://example.com">link</a>'If bleach is not installed, the sanitizer degrades gracefully to a tag-stripping fallback — no breaking changes.
| Resource | Suggested TTL |
|---|---|
| Items | 5–10 min |
| Story lists | 1–2 min |
updates.json |
30–60 s |
| Users | 10–30 min |
maxitem.json |
1 min |
The HNCache class wraps cachetools.TTLCache when available, or degrades to a no-op passthrough otherwise.
from hn_engine import HNCache, get_item
cache = HNCache(maxsize=1024, ttl=300.0)
@cache.cached(ttl=120)
def fetch_story(story_id: int):
return get_item(story_id)
story = fetch_story(8863) # cache miss → API call
story = fetch_story(8863) # cache hit → instant return
print(cache.stats)
# → {'hits': 1, 'misses': 1, 'total': 2, 'hit_rate': 0.5}
⚠️ Never cachenullresponses indefinitely — the resource may appear later.
from hn_engine import get_story_ids, get_item, sanitize_html, StoryKind
top_ids = get_story_ids(StoryKind.TOP, limit=10)
for item_id in top_ids:
item = get_item(item_id)
if item is None:
continue
title = sanitize_html(item.get("title", ""))
print(f"[{item.get('score', 0):>4}] {title}")from hn_engine import get_comment_tree, sanitize_html
tree = get_comment_tree(8863, max_depth=2, max_nodes=100)
for node_id, node in tree.items():
if node.get("type") != "comment":
continue
text = sanitize_html(node.get("text", ""))
print(f"#{node_id} by {node.get('by')}: {text[:80]}...")import time
from hn_engine import get_updates, get_item
seen_items: set[int] = set()
while True:
updates = get_updates()
for item_id in updates["items"]:
if item_id in seen_items:
continue
item = get_item(item_id)
if item is None:
continue
seen_items.add(item_id)
print("Refreshed item", item_id, "type=", item.get("type"))
time.sleep(45)from hn_engine import get_maxitem
last_seen = get_maxitem()
# ... later ...
current = get_maxitem()
new_ids = list(range(current, last_seen, -1))
print(f"{len(new_ids)} new items appeared")from hn_engine import get_user
user = get_user("jl")
if user:
print(f"@{user['id']} — karma: {user['karma']}")More examples in the examples/ directory:
| Example | Description |
|---|---|
01_top_stories.py |
Fetch and display top stories with metadata. |
02_comment_tree.py |
Traverse a bounded comment tree. |
03_incremental_sync.py |
Poll updates.json in a loop. |
04_maxitem_and_user.py |
Detect new items and fetch user profiles. |
05_caching.py |
Use the TTL cache decorator. |
Hacker-News-Advanced-Insight-And-Analytics-Engine/
├── hn_engine/
│ ├── __init__.py # Public API exports
│ ├── client.py # HNClient: retries, backoff, timeout
│ ├── items.py # get_item, get_comment_tree, item_type_traits
│ ├── users.py # get_user
│ ├── lists.py # get_story_ids, get_updates, get_maxitem
│ ├── sanitize.py # sanitize_html with allow-list
│ ├── cache.py # HNCache: TTL with graceful degradation
│ └── errors.py # Exception hierarchy
├── examples/
│ ├── 01_top_stories.py
│ ├── 02_comment_tree.py
│ ├── 03_incremental_sync.py
│ ├── 04_maxitem_and_user.py
│ └── 05_caching.py
├── tests/
│ ├── test_errors.py
│ ├── test_client.py
│ ├── test_items_validation.py
│ ├── test_lists.py
│ ├── test_sanitize.py
│ ├── test_cache.py
│ └── test_integration.py
├── assets/
│ ├── banner.svg
│ ├── architecture.svg
│ ├── comment-tree.svg
│ └── caching.svg
├── .github/workflows/ci.yml
├── pyproject.toml
├── README.md
└── LICENSE
- Do bound every traversal (
max_depth,max_nodes). - Do sanitize every HTML field before rendering.
- Do cache aggressively — the API is read-heavy.
- Do ignore unknown fields — the API may add fields without warning.
- Do URL-encode usernames — they are case-sensitive.
- Don't rely on
descendantsfor comment-level counts. - Don't recurse unboundedly over
kids. - Don't assume
nullmeans "deleted forever" — the item may reappear.
- The API has no pagination and no filtering.
- Comment counts require manual traversal.
- No guaranteed rate limit means a well-behaved client should self-throttle.
- Some items may return
nulleven when they exist (deleted / dead). submittedon active users can contain thousands of IDs.
Issues and pull requests are welcome. Please:
- Follow PEP 8 and
ruffformatting. - Add tests for any new endpoint wrapper.
- Keep the sanitization allow-list conservative.
- Document any new retry behavior.
- Render and inspect any new SVG assets before committing.
Run the test suite:
# Unit tests (no network)
pytest -m "not integration" -v
# Integration tests (live API)
pytest -m integration -v
# Lint and type-check
ruff check hn_engine tests
mypy hn_engineMIT — see LICENSE.