diff --git a/docs/entities/entity-api.md b/docs/entities/entity-api.md index 01c5cfbf..c78d1a46 100644 --- a/docs/entities/entity-api.md +++ b/docs/entities/entity-api.md @@ -136,10 +136,16 @@ The CRUD API provides create, read, update, and delete operations for entity doc 204 No Content ``` +Reads by ID always return the latest stored state. Pass `?async=true` on create, update, patch and upsert to return without waiting for the entity to become searchable — see [Write Guarantees](/docs/entities/indexing-guarantees#write-guarantees). + ## Search API The Search API provides list and search functionality for entity documents. +:::info +Search and list read from a search index, not from the primary store. By default a write is searchable as soon as the write call returns; with `?async=true` it becomes searchable shortly after. See [Search Indexing Guarantees](/docs/entities/indexing-guarantees) for the full consistency model. +::: + ### Listing Entities [[API Docs](/api/entity#operation/listEntities)] @@ -197,6 +203,8 @@ Keep the `stable_for` TTL short. Stable windows consume server-side resources an - Fetching a single page (up to 1,000 items, the max `size`) - The next page request depends on user input (the stable window may expire) +See [Pagination](/docs/entities/indexing-guarantees#pagination) for the guarantees of each pagination mode. + ## Relations API [[API Docs](/api/entity/#tag/Relations)] diff --git a/docs/entities/entity-tables.md b/docs/entities/entity-tables.md index a2957900..0d103989 100644 --- a/docs/entities/entity-tables.md +++ b/docs/entities/entity-tables.md @@ -46,6 +46,17 @@ Each filter also supports an **(empty)** option to find entities where the attri Filter suggestions load progressively (up to 50 results). Type a keyword to search within the filter values if you need to find a specific option. ::: +## Data Freshness + +Entity tables read from the search index through the [Search API](/api/entity/#tag/Entities/operation/searchEntities). + +- Changes made in epilot or through the API with default settings show up the next time the table loads. +- Entities written with `?async=true` (bulk imports, migrations) appear shortly after the write. +- Columns and filters based on **related** entities update in the background. After renaming a contact, filtering orders by the contact's name can match the old value for a short time. +- Only attributes defined in the schema can be filtered and sorted. + +See [Search Indexing Guarantees](/docs/entities/indexing-guarantees) for details. + ## Saved Views [[API Docs](/api/entity#tag/Saved-Views)] diff --git a/docs/entities/indexing-guarantees.md b/docs/entities/indexing-guarantees.md new file mode 100644 index 00000000..15fdad90 --- /dev/null +++ b/docs/entities/indexing-guarantees.md @@ -0,0 +1,204 @@ +--- +sidebar_position: 5.5 +title: Search Indexing Guarantees +description: "Consistency and freshness guarantees of the entity search and list APIs: when writes become searchable, what is indexed, and how to paginate reliably." +--- + +# Search Indexing Guarantees + +[[API Docs](/api/entity/#tag/Entities)] +[[SDK](https://www.npmjs.com/package/@epilot/entity-client)] + +Entities live in two places: a **primary store**, which is the source of truth, and a **search index**, which powers `searchEntities`, `listEntities` and with them every [entity table](/docs/entities/entity-tables) in epilot. + +Every write goes to the primary store first and is indexed afterwards. This page describes what you can rely on between those two steps. + +## At a Glance + +| You do | You can rely on | +|--------|-----------------| +| Write an entity (default) | It is searchable **when the write call returns** | +| Write with `?async=true` | It becomes searchable shortly after — **eventually consistent**, no upper bound | +| Read by ID (`getEntity`) | **Always** the latest stored state | +| Search or list | Matching, sorting and `hits` reflect the index; returned entities are loaded fresh from the primary store | +| Change an entity that other entities relate to | Related entities are re-indexed **asynchronously** | +| Paginate with `from` | Up to `from + size = 25,000`, results may shift between pages | +| Paginate with `stable_for` + `search_after` | A consistent snapshot, no missing or duplicate items | + +## Which API Reads From Where + +| Operation | Reads from | Consistency | +|-----------|-----------|-------------| +| `getEntity`, `getEntityV2` | Primary store | Strong — read-your-writes | +| `getRelations`, `getRelationsV3` | Primary store | Strong — read-your-writes | +| `searchEntities`, `listEntities` | Search index | See [write guarantees](#write-guarantees) | +| `getRelationsV2` | Search index | Same as search | +| `autocomplete` | Search index | Same as search | +| `upsertEntity` (matching by `unique_key`) | Search index | Same as search — see [upserts](#upserts-and-unique-keys) | +| `exportEntities` | Search index | Same as search | + +If you know the entity ID, read by ID. Use search to find entities, not to confirm a write. + +## Write Guarantees + +### Synchronous writes (default) + +`createEntity`, `updateEntity`, `patchEntity`, `upsertEntity` and the relation operations (`addRelations`, `updateRelation`, `removeRelations`, `deleteRelation`) index the entity **before they respond**. Once you receive the success response: + +- The entity is stored and readable by ID. +- The entity is visible in `searchEntities` and `listEntities` with its new values. +- Webhooks and automations triggered by the write can already find the entity through search. + +`deleteEntity` is always synchronous. A deleted entity leaves the default search results before the call returns. Soft-deleted entities stay findable with `include_deleted: "true"` or `"only"`. + +### Asynchronous writes (`?async=true`) + +Pass `async=true` to skip waiting for the index. The call returns as soon as the entity is stored, and indexing happens in the background. + +- The entity is immediately readable by ID. +- It appears in search **eventually** — usually within a few seconds. There is no guaranteed upper bound: during bulk imports or large migrations the delay grows with the backlog. +- Multiple quick writes to the same entity are coalesced. The index always converges to the **latest stored state**, never to an intermediate or older one. +- Webhooks and automations triggered by the write may run **before** the entity is searchable. + +Use `async=true` for migrations and bulk loads where throughput matters more than immediate searchability. Keep the default for anything interactive. + +```json title="createEntity — POST /v1/entity/contact?async=true" +{ + "first_name": "Erika", + "last_name": "Mustermann" +} +``` + +:::caution +Do not write with `async=true` and then search for the entity right away. Use the `_id` from the write response and read by ID instead. +::: + +### Failed writes + +A write that returns a `5xx` error may already be stored but not yet indexed. Read the entity by ID to check, and call [`reindexEntity`](#repairing-a-stale-entity) or repeat the update to make it searchable. + +## Related Entity Data + +Search documents contain a copy of the data of directly related entities (one level deep). This is what makes free-text search and relation filters work across relations — for example finding an order by its customer's name. + +That copy is refreshed **asynchronously**, even for synchronous writes: + +- When you update a contact, the contact itself is searchable with the new values right away. +- Orders, contracts and other entities that relate to this contact are re-indexed in the background. Until then, searching orders by the contact's **old** name still matches. +- Up to 500 related entities are refreshed right after each change. Entities with more relations than that are caught up by a nightly job. +- Changes that cannot affect related search documents are skipped. + +The entities returned in the response are not affected: hydrated relations (`hydrate: true`) are always loaded fresh from the primary store. Only **matching, filtering and sorting** on related data can lag. + +## What Is Indexed + +Only data described by the entity schema is searchable: + +- Attributes defined in the schema, including attributes added by [capabilities](/docs/entities/capabilities) +- System fields such as `_id`, `_schema`, `_title`, `_tags`, `_purpose`, `_created_at`, `_updated_at`, `_acl` +- Direct relations and a copy of the related entities' data, one level deep + +Properties that are **not** part of the schema are stored and returned by `getEntity`, but you cannot search, filter or sort by them. Add the attribute to the schema first. + +Invalid values are skipped rather than rejected: a date attribute holding a non-date string or an address attribute without address fields is stored, but not indexed. + +:::info +After adding an attribute to a schema, existing entities become searchable by it the next time they are written or re-indexed. +::: + +## Search Results + +### Matching vs. returned data + +`searchEntities` and `listEntities` evaluate the query, sort order, aggregations and `hits` total against the index. The entities in `results` are then loaded from the primary store. + +This means a result can be **newer** than the state that matched. If an entity was changed with `async=true` a moment ago, it may still be returned for a filter it no longer satisfies — carrying its current values. + +### Totals + +`hits` is the exact number of matching entities, not an estimate, including for result sets beyond the pagination window. + +### Permissions + +Access restrictions are part of the index and applied at query time. A change to an entity's access is effective in search as soon as that entity is re-indexed, following the same rules as any other write. + +## Pagination + +### Offset pagination (`from` / `size`) + +- `size` is capped at **1,000** per page. +- `from + size` is limited to **25,000**. A larger `from` is not rejected — it is clamped to the last available window, so paging past the limit returns the same items again. +- Each page is an independent query. If entities are created, changed or deleted between requests, items can shift across pages and appear twice or not at all. + +Offset pagination is the right choice for interactive tables where a person pages through results. + +### Deep pagination (`search_after`) + +Pass the `sort_end` array of the previous response as `search_after` to continue after the last item. This has no depth limit. Always pass an explicit `sort`, for example `_created_at:asc`. + +On its own, `search_after` still runs every page as an independent query: entities that change between requests, or that share the same sort value, can be skipped or repeated. Combine it with a stable query to rule that out. + +### Stable queries (`stable_for`) + +For a guaranteed consistent walk through a result set, combine `search_after` with a [stable query](/docs/entities/entity-api#stable-queries): + +```json title="listEntities — POST /v1/entity:list" +{ + "filter": [{ "term": { "_schema": "contact" } }], + "sort": "_created_at:asc", + "size": 1000, + "stable_for": 30 +} +``` + +```json title="Next page" +{ + "filter": [{ "term": { "_schema": "contact" } }], + "sort": "_created_at:asc", + "size": 1000, + "stable_for": 30, + "stable_query_id": "", + "search_after": [1747905443332, "0.000023312468"] +} +``` + +`search_after` takes the `sort_end` array of the previous response as is. Always pass the `stable_query_id` of the **latest** response — it can change between pages. + +A stable query freezes the index at the moment of the first request: + +- Every matching entity is returned exactly once — no gaps, no duplicates. +- Writes made after the first request are **not** visible in later pages. +- The snapshot lives for `stable_for` seconds (maximum 60) and is extended by every request that passes `stable_for` again. +- When it expires, the API responds with `400 Point in time query has expired`. Start over with a new query. + +## Repairing a Stale Entity + +Background indexing is retried automatically. In the rare case an entity stays out of date in search, any of these brings it back in sync: + +- Write the entity again (any update re-indexes it). +- Call `reindexEntity`, which re-indexes the entity and all entities related to it: + +```text title="reindexEntity — POST /v1/entity/{slug}/{id}:reindex" +200 OK +``` + +## Upserts and Unique Keys + +`upsertEntity` finds the existing entity for a `unique_key` through the search index. Uniqueness is therefore **not** enforced transactionally: + +- Two concurrent upserts with the same key can both find nothing and create two entities. +- An upsert that follows an `async=true` write of the same key may not find it yet and create a duplicate. + +To keep unique keys unique, run upserts for the same key one after another and use synchronous writes (the default). Pass `strict=true` to get a `409` instead of a silent update when more than one entity already matches. + +## Recommendations + +| Scenario | Approach | +|----------|----------| +| Create an entity, then show it in a list | Default synchronous write, then search | +| Create an entity, then keep working with it | Use the `_id` from the response and `getEntity` | +| Bulk load or migration | `async=true`, verify by ID, not by search | +| Export or batch-process a full result set | `stable_for` + `search_after` | +| More than 25,000 results | `search_after`, never a large `from` | +| React to entity changes | [Webhooks](/docs/integrations/webhooks) instead of polling search | +| Sync by `unique_key` | Sequential, synchronous `upsertEntity` calls |