feat: full-text search and schema discovery tools for LangChain agents

CHANGELOG.md

@@ -7,7 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Full-text search tool backed by the engine's BM25 index. `make_hotdata_tools` grows
+  `search_table`/`search_column`/`search_columns`/`search_k`/`search_tool_name` and appends a
+  `hotdata_search_text` tool when a table and column are given; `make_hotdata_search_tool`
+  builds one directly, so several searchable corpora can be registered side by side. The
+  corpus is pinned at construction rather than chosen by the model, because nothing in the
+  tool surface lets an agent discover which columns carry a BM25 index.
+- `bm25_search_sql` and `bm25_search_json` for building and running a ranked search without
+  going through a tool. `bm25_search_json` returns the same `{"metadata", "rows"}` envelope as
+  `execute_sql_json`.
+- `hotdata_describe_tables` tool (registered by default, `describe_tables=False` to omit) and
+  `describe_tables_json`/`make_hotdata_describe_tables_tool`. With no argument it lists the
+  scoped database's tables and their column counts; with a table name it returns that table's
+  columns and types, capped so one wide table cannot flood the model's context. Reads
+  `information_schema`, so it needs no extra permissions. Without it an agent guesses column
+  names off the shape of the data it has already seen.
+- `demo/` — an end-to-end script that creates a managed database, loads the public SF Airbnb
+  fixture, builds a BM25 index, invokes the search tool, and then hands both the search and
+  SQL tools to a LangChain agent.
+
+### Changed
 
+- Tool descriptions now state the engine's actual contract instead of a one-line summary.
+  `hotdata_execute_sql` names the dialect and the supported constructs, points at the search
+  tool for text relevance when one is registered, and warns that an aggregate query must
+  reference a column (`COUNT(*)` alone is rejected). The database tools steer callers towards
+  ids, since names are non-unique display labels. Every claim was verified against a live
+  engine and is pinned by `tests/test_descriptions.py`. Without this an agent reaches for
+  `to_tsvector` and the query fails; with it, the correct search-then-SQL path is taken with
+  no system-prompt guidance at all.
+- Require `hotdata-framework>=0.9.0` / `hotdata>=0.8.0`. The pinned 0.4.1 uploaded through
+  `POST /v1/files`, which the API no longer serves, so `load_managed_table` failed with a bare
+  `Not Found`; 0.9.0 uses the session/finalize upload flow.
+
+### Fixed
+
+- README quickstart used `create_tool_calling_agent`/`AgentExecutor`, neither of which exists
+  in LangChain v1; replaced with `create_agent`.
+- README and `examples/langchain_basic.py` ran SQL without a database scope, which the API
+  now rejects with `a database is required`. Both now scope their queries.
 
 ## [0.2.2] - 2026-06-27
 

README.md

@@ -1,6 +1,6 @@
 # hotdata-langchain
 
-Give your [LangChain](https://python.langchain.com/) agents access to [Hotdata](https://hotdata.dev) — run SQL against your workspace connections and work with managed databases.
+Give your [LangChain](https://python.langchain.com/) agents access to [Hotdata](https://hotdata.dev) — run SQL against your workspace connections, full-text search indexed columns, and work with managed databases.
 
 ## Install
 
@@ -14,18 +14,27 @@ Set `HOTDATA_API_KEY` in your environment. Optionally set `HOTDATA_WORKSPACE` to
 
 ## Quickstart
 
+The package itself depends only on `langchain-core`, and works with any tool-calling model.
+Running an agent additionally needs the `langchain` package and the integration for whichever
+model provider you use.
+
 ```python
-from langchain.agents import AgentExecutor, create_tool_calling_agent
+from langchain.agents import create_agent
 import hotdata_langchain as hl
 
 client = hl.from_env()
-tools = hl.make_hotdata_tools(client)
+tools = hl.make_hotdata_tools(client, database="sales")
 
-agent = create_tool_calling_agent(llm=your_llm, tools=tools, prompt=your_prompt)
-executor = AgentExecutor(agent=agent, tools=tools)
-result = executor.invoke({"input": "How many rows are in the orders table?"})
+agent = create_agent(model=your_model, tools=tools)
+result = agent.invoke(
+    {"messages": [{"role": "user", "content": "Which product categories have the most orders?"}]}
+)
+print(result["messages"][-1].content)
 ```
 
+Queries run against a database scope, so pass `database=` (a managed database name or id).
+`hl.from_env().list_managed_databases()` shows what is available in the workspace.
+
 ## Tools
 
 `make_hotdata_tools(client)` returns a list of LangChain `StructuredTool` objects ready to pass to any agent:
@@ -36,13 +45,33 @@ result = executor.invoke({"input": "How many rows are in the orders table?"})
 | `hotdata_list_managed_databases` | List available managed databases |
 | `hotdata_create_managed_database` | Create a new managed database |
 | `hotdata_load_managed_table` | Load a parquet file into a managed table |
+| `hotdata_describe_tables` | List tables, or one table's columns and types |
+| `hotdata_search_text` | Full-text search an indexed column, ranked by relevance (opt-in — see below) |
+
+The descriptions carry the engine's contract — dialect, what SQL can and cannot do, and where
+to look things up — so an agent does not need a system prompt explaining the query engine.
+
+## Letting the agent discover the schema
+
+`hotdata_describe_tables` is registered by default. Called with no arguments it lists every
+table with its column count; called with a table name it returns that table's columns and
+types. Without it an agent has to guess column names, and a guess that misses fails the query.
+
+```python
+tools = hl.make_hotdata_tools(client, database="sales")            # included
+tools = hl.make_hotdata_tools(client, database="sales", describe_tables=False)  # omitted
+```
+
+It reads `information_schema` in whichever database the tools are scoped to, so it needs no
+extra permissions. With it turned off, the SQL tool's description tells the agent to query
+`information_schema` directly instead.
 
 ## Calling tools directly
 
 You can also invoke tools outside of an agent loop:
 
 ```python
-tools = {t.name: t for t in hl.make_hotdata_tools(client)}
+tools = {t.name: t for t in hl.make_hotdata_tools(client, database="sales")}
 
 result = tools["hotdata_execute_sql"].invoke({"sql": "SELECT * FROM orders LIMIT 10"})
 print(result)  # JSON rows
@@ -60,9 +89,64 @@ tools["hotdata_load_managed_table"].invoke({
 })
 ```
 
+## Full-text search
+
+Point the agent at a text column carrying a BM25 index and it gets a search tool alongside SQL:
+
+```python
+tools = hl.make_hotdata_tools(
+    client,
+    database="sf_airbnb",
+    search_table="default.public.listings",   # catalog.schema.table
+    search_column="description",              # must have a BM25 index
+    search_columns=["id", "name", "price", "description"],  # what each hit returns
+    search_k=5,
+)
+
+hits = {t.name: t for t in tools}["hotdata_search_text"].invoke(
+    {"query": "cozy apartment with a view"}
+)
+```
+
+Rows come back ranked, each with a `score`. The agent supplies only `query` and an optional
+`k`; the table and column are fixed when you build the tool. That is deliberate — nothing in
+the tool surface lets an agent discover which columns are indexed, and the engine errors
+outright rather than falling back to a scan when a column has no BM25 index.
+
+Inside a managed database the built-in catalog is always `default`, so a managed table reads
+as `default.<schema>.<table>` when `database=` scopes the query to it.
+
+For more than one searchable corpus, build the tools yourself and give each a distinct name
+and description — the agent then routes on the descriptions:
+
+```python
+tools = [
+    # Configure the first corpus here, so the SQL tool's description still names a search
+    # tool to defer text matching to. Passing no search_table/search_column drops that,
+    # and the agent goes back to trying to match text in SQL.
+    *hl.make_hotdata_tools(
+        client,
+        database="sf_airbnb",
+        search_table="default.public.listings",
+        search_column="description",
+        search_tool_name="search_listings",
+    ),
+    hl.make_hotdata_search_tool(
+        client, table="default.public.reviews", column="comments",
+        name="search_reviews", database="sf_airbnb",
+    ),
+]
+```
+
+Provisioning the index itself is not yet part of this package; create it through the Hotdata
+API or CLI. `demo/` has a script that does the whole flow — managed database, data load, BM25
+index, then an agent that picks between search and SQL.
+
 ## Scoping queries to a managed database
 
-Pass `database=` so all SQL the agent runs resolves against a specific managed database:
+`database=` resolves all SQL the agent runs against a specific managed database, by name or
+id. The API requires a database scope, so queries fail with `a database is required` without
+it:
 
 ```python
 tools = hl.make_hotdata_tools(client, database="sales")
@@ -83,6 +167,9 @@ uv run python examples/langchain_basic.py
 uv run python examples/langchain_managed_db.py
 ```
 
+For a full end-to-end run against a real workspace — data load, BM25 index build, then an
+agent choosing between search and SQL — see [`demo/`](demo/README.md).
+
 ## Development
 
 ```bash

demo/README.md

@@ -0,0 +1,127 @@
+# BM25 search demo
+
+Takes a Hotdata workspace from empty to a LangChain agent that full-text searches real
+data, in one script. Uses the public San Francisco Airbnb listings fixture (7,535 rows)
+and builds a BM25 index over the free-text `description` column.
+
+## Run it
+
+Steps 1–5 need only a Hotdata key and exercise the tool directly, no model involved:
+
+```bash
+uv run --group demo --env-file .env python demo/bm25_search_demo.py
+```
+
+The agent step (6) runs when you name a tool-calling model. Any provider works — install
+its LangChain integration, set its key, and pass `<provider>:<model>`:
+
+```bash
+uv run --group demo --env-file .env python demo/bm25_search_demo.py \
+    --model '<provider>:<model>'
+```
+
+The script is safe to re-run: it reuses the managed database, skips the load when the
+table already has rows, and reuses an existing index.
+
+```bash
+# different search text, more hits
+uv run --group demo --env-file .env python demo/bm25_search_demo.py \
+    --query "quiet garden studio near the park" --k 10
+
+# stop before the agent step even with a model set
+uv run --group demo --env-file .env python demo/bm25_search_demo.py --skip-agent
+
+# tear down the managed database it created
+uv run --group demo --env-file .env python demo/bm25_search_demo.py --cleanup
+
+# trace the run into a named LangSmith project
+LANGSMITH_TRACING=true LANGSMITH_PROJECT=hotdata-langchain-bm25 \
+    uv run --group demo --env-file .env python demo/bm25_search_demo.py
+```
+
+## Credentials
+
+| Variable | Needed for | Notes |
+|---|---|---|
+| `HOTDATA_API_KEY` | steps 1–5 | everything except the agent run |
+| `HOTDATA_WORKSPACE` | optional | pins a workspace; first available otherwise |
+| your model provider's key | step 6 | skipped without it, or without `--model` |
+| `LANGSMITH_API_KEY` + `LANGSMITH_TRACING=true` | optional | traces the run to LangSmith |
+| `LANGSMITH_PROJECT` | optional | project the traces land in (default: `default`) |
+
+BM25 needs no embedding provider — just a string column — so there is no extra
+credential for the index itself, unlike a vector index. LangSmith is observability
+only; the agent still needs a model provider key of its own.
+
+With tracing on, every tool call becomes a run in the LangSmith project — the search
+tool shows up as a `tool` run named `hotdata_search_text` carrying its query and the
+ranked JSON it returned, so the generated SQL and the agent's choice between search
+and SQL are both inspectable after the fact.
+
+## What each step does
+
+1. **Managed database** — creates `langchain_bm25_demo` with `public.listings` declared
+   up front, so the load materialises into it directly.
+2. **Listings data** — downloads the fixture parquet (cached in the system temp dir) and
+   loads it into the managed table.
+3. **BM25 index** — creates a `bm25` index on `description` through `IndexesApi` and
+   polls until it reports ready. This step is a hard prerequisite: `bm25_search` has no
+   brute-force fallback and errors outright when no index exists.
+4. **Tools** — reads the real table schema and narrows the projection to a handful of
+   useful columns. The listings table is 85 columns wide; returning all of them would
+   flood the agent's context.
+5. **Direct tool invocation** — prints the generated SQL, then the ranked hits with
+   their BM25 scores. Proves the tool against the live engine without an LLM in the loop.
+6. **Agent run** — gives a LangChain agent both `hotdata_search_text` and
+   `hotdata_execute_sql` and asks a question neither answers alone: find listings whose
+   descriptions mention a quiet garden studio, then report listing counts and average
+   review scores across *all* listings in those neighbourhoods. The aggregate spans the
+   whole dataset, not the handful search returned, so the agent has to use search to
+   identify the neighbourhoods and SQL to aggregate over them. The printed tool calls
+   show which pathway it picked for which part.
+
+   Set the model with `--model` or `DEMO_MODEL`; any tool-calling model works, and the
+   step is skipped when its provider key is absent.
+
+   **What this step demonstrates is the routing.** Across five runs of the current
+   configuration the agent called the search tool and the schema tool every time, and the
+   final table matched the whole-neighbourhood figures every time. Two of those runs also
+   recovered from a bad query after the error was handed back to them.
+
+   Do not read five runs as a guarantee. Answering a compound question correctly is a
+   property of the model rather than of these tools, and an earlier configuration — which
+   shared one row budget between the search and SQL tools, silently truncating the
+   aggregate — got it right in only four of five. The printed tool calls are the reliable
+   part; treat the prose answer as illustrative.
+
+## What makes the agent run work
+
+**The tool descriptions, not the system prompt.** The demo's system prompt says only who
+the agent is — it deliberately says nothing about which tool to use or how the engine
+behaves. That guidance lives in the tool descriptions, so any application gets it without
+having to teach the model about the query engine.
+
+It matters. An earlier version of this demo had a one-line SQL tool description and a
+system prompt spelling out the rules; the model still reached for Postgres full-text
+idioms (`to_tsvector`/`plainto_tsquery`), which the engine rejects with
+`Invalid function 'to_tsvector'`. With the constraint stated in the SQL tool's own
+description instead, the model uses the search tool and feeds its results into SQL —
+even with no system-prompt guidance at all.
+
+**Tool errors have to reach the model.** The tools raise on failure, and an exception out
+of a tool aborts the whole graph — so the demo wraps them (`with_error_feedback`) to
+return the error as a message instead. The wrapper also digs the engine's real message
+out of the exception chain: the framework raises `RuntimeError("Bad Request")` while the
+useful text sits in the underlying API response body. Descriptions lower the failure
+rate; readable errors are what let the model recover from what slips through.
+
+## Why the generated SQL looks the way it does
+
+Step 5 prints the query. Two details in it are load-bearing:
+
+- **`ORDER BY score DESC`** — the engine returns hits in rowid order, not ranked, so
+  ranking has to be asked for explicitly.
+- **`k` appearing twice** — once as `bm25_search(...)`'s fourth argument and once as a
+  trailing `LIMIT`. The fourth argument is what actually bounds the search, because
+  `ORDER BY` blocks limit pushdown; relying on the trailing `LIMIT` alone would let the
+  scan fall back to the engine's much larger default bound.