diff --git a/agents/agent-context.mdx b/agents/agent-context.mdx index 6416d75e..f90f699c 100644 --- a/agents/agent-context.mdx +++ b/agents/agent-context.mdx @@ -5,13 +5,13 @@ description: "Give every agent in a project the same terminology, table choices, doc-type: reference --- -Agent context is shared background knowledge every AI agent in a project reads before answering. It lives in a single file, `lightdash.project_context.yml`, that sits next to `lightdash.config.yml` inside your dbt project directory. Notes you add here change future agent behavior without changing the semantic layer. +Agent context is shared background knowledge every AI agent in a project reads before answering. It lives in a single file, `lightdash.project_context.yml`, that sits next to `lightdash.config.yml` inside your dbt or [native Lightdash YAML](/semantic-layer/yaml) project directory. Notes you add here change future agent behavior without changing the semantic layer. -The file is committed to your dbt repo and travels with the rest of your project metadata. +The file is committed to your project repository and travels with the rest of your project metadata. ## When to use it -Reach for agent context when the data model itself is fine but the agent is missing background knowledge — terminology, which table to prefer, or business rules that aren't expressible as dbt metadata. If the fact would also help a new analyst joining the team, edit the semantic layer instead. +Reach for agent context when the data model itself is fine but the agent is missing background knowledge — terminology, which table to prefer, or business rules that aren't expressible as semantic definitions. If the fact would also help a new analyst joining the team, edit the semantic layer instead. Agents pick up new context on their next turn, so the loop between "notice a gap" and "agents behave better" is short. @@ -19,7 +19,7 @@ Agents pick up new context on their next turn, so the loop between "notice a gap There are two ways entries land in the file: -- **Through Issues writeback.** When a [project context](/agents/issues) fix runs, Lightdash proposes a single entry and opens a pull request against your dbt repo. The first time this happens the file is created (with a header explaining what it's for). Subsequent fixes add or update entries in the same file. +- **Through Issues writeback.** When a [project context](/agents/issues) fix runs, Lightdash proposes a single entry and opens a pull request against your project repository. The first time this happens the file is created (with a header explaining what it's for). Subsequent fixes add or update entries in the same file. - **By hand.** You can edit `lightdash.project_context.yml` directly. Writeback uses the GitHub API to merge one entry at a time, preserving comments, quoting, and key order in the rest of the file, so manual edits and writeback coexist cleanly. Either way, the same validation rules apply: invalid files surface schema-backed errors at ingest time. diff --git a/agents/ai-writeback.mdx b/agents/ai-writeback.mdx index cd6d55e2..2a79eaee 100644 --- a/agents/ai-writeback.mdx +++ b/agents/ai-writeback.mdx @@ -1,23 +1,23 @@ --- title: AI writeback -description: Ask an AI agent to edit your dbt project in chat and open a pull request with the changes +description: Ask an AI agent to edit dbt or native Lightdash YAML files and open a pull request doc-type: tutorial --- -AI writeback lets you ask an AI agent in chat or Slack to change something in the dbt repository that backs your project — rename a metric, add a dimension, edit a model's SQL, fix a YAML description — and have the agent open a pull request (or GitLab merge request) with the change. +AI writeback lets you ask an AI agent in chat or Slack to edit the repository that backs your project and open a pull request with the change. It can update dbt models or [native Lightdash YAML models](/semantic-layer/yaml#ai-writeback-and-issues), including metrics, dimensions, and descriptions. -This is the same writeback capability you would normally trigger from the Custom Metric or SQL Runner menus, surfaced inside the conversation with your AI agent. +Use it for changes that need editing across project files. To promote an individual custom field, use the write-back action in the Explore view. -For changes that don't belong in the dbt project — for example, editing an app repo, a service, or a docs site — use the [AI coding agent](/agents/ai-coding-agent) instead. It runs on the same E2B + Anthropic pipeline but targets any repository your organization has connected, and delegates verification to that repository's own CI. +For changes that don't belong in the semantic layer project — for example, editing an app repo, a service, or a docs site — use the [AI coding agent](/agents/ai-coding-agent) instead. ## When to use it -Use AI writeback when the change you want needs to land in your dbt project files: +Use AI writeback when the change belongs in your semantic layer project files: - Rename a metric in a YAML file -- Add a new metric or dimension to a dbt model -- Edit a model's SQL -- Update a description, label, or `meta` block +- Add a metric or dimension to a model +- Edit field SQL, or a dbt model's SQL +- Update a description, label, or dbt `meta` block - Fix a typo in a column definition For other kinds of changes, the agent will pick a different tool: @@ -26,23 +26,25 @@ For other kinds of changes, the agent will pick a different tool: | --- | --- | | Ask a question about data | Query and discovery tools | | Edit an existing chart or dashboard in Lightdash | `editContent` | -| Change the underlying dbt repository | `editDbtProject` | +| Change dbt or native Lightdash YAML source files | `editDbtProject` | ## Prerequisites -- Your project's dbt connection must be a **GitHub**, **GitLab**, or **Bitbucket Cloud** repository. Azure DevOps, dbt Cloud, and local dbt connections are not supported by this workflow. -- Configure access for your Git provider: +- Use a supported repository connection: **GitHub**, **GitLab.com**, or **Bitbucket Cloud** for dbt; **GitHub** for native Lightdash YAML. Azure DevOps, dbt Cloud, and CLI-only connections are not supported for AI writeback. +- Configure repository access so the agent can open requests: - **GitHub** — install the Lightdash GitHub App on the repository. - **GitLab** — connect the Lightdash GitLab App for your organization. Writeback works against GitLab.com; self-hosted GitLab domains aren't reachable from the writeback sandbox yet. - **Bitbucket Cloud** — configure the [project API token and writeback permissions](/get-started/quickstart/connect-project#bitbucket-cloud-api-token). - You need at least [project Developer](/workspace-admin/roles) permissions on the project. +For native projects, follow [Connect through GitHub](/semantic-layer/yaml#connect-through-github) and select **Native Lightdash YAML** as the semantic layer format. + ## How it works When you ask the agent for a change that belongs in the repo, it calls a tool called `editDbtProject`. The tool: 1. Generates a self-contained instruction from your request and starts a sandboxed environment with no memory of your chat. A follow-up to the same change resumes that environment; an unrelated request gets a fresh one. -2. Edits the relevant files and runs `lightdash compile` to validate the result. +2. Edits the relevant files and validates the result. dbt projects use `lightdash compile`; native projects use [native YAML validation](/semantic-layer/yaml#ai-writeback-and-issues) without setting up dbt. 3. Pushes a new branch and opens a pull request (GitHub or Bitbucket) or merge request (GitLab) against the branch configured in your project connection. 4. Returns the pull request URL to you in chat. @@ -88,7 +90,7 @@ description: change "lifteime" to "lifetime." ### Mentioning project files with `@` -In the agent chat input, type `@` to open the mention menu. Alongside the existing chart and dashboard suggestions, you'll see a **Files** group listing the source files in your project's dbt repository. Pick a file and its path is inserted as a pill in your message. +In the agent chat input, type `@` to open the mention menu. Alongside chart and dashboard suggestions, the **Files** group lists source files in your project's repository. Pick a file and its path is inserted as a pill in your message. Use file mentions to point the writeback agent at the exact file you want changed: @@ -127,7 +129,7 @@ The writeback PR card has two action groups: - Both actions ask you to confirm before sending the request. - Once the PR reaches a terminal state, the button group collapses to a **Merged** or **Closed** marker and the card stops polling. -When you merge from the card, Lightdash recompiles the project automatically — the same refresh as **Settings → Project → Sync dbt project** — so the merged change goes live without a manual sync. If the change renamed or removed a field, the merge also kicks off the content-migration flow described in [Impact and safety checks](#impact-and-safety-checks). +When you merge from the card, Lightdash recompiles the project automatically using its selected semantic layer format, so the merged change goes live without a manual refresh. If the change renamed or removed a field, the merge also kicks off the content-migration flow described in [Impact and safety checks](#impact-and-safety-checks). Merging or closing a writeback PR from the card uses your project permissions @@ -148,21 +150,21 @@ For changes that could break existing content or change results, the agent does **That the numbers still hold.** When a change relies on results staying the same — consolidating two duplicate metrics, replacing one field with another, splitting a metric into parts, or refactoring a field's SQL — the agent proves it rather than asserting it. It either shows the guarantee from the model SQL, or runs the affected fields at a total and across a time dimension and confirms they match, before calling the change safe. If the numbers diverge, it tells you exactly what differs instead of shipping the change. -## Reading the dbt repository +## Reading the project repository -The agent can also read source files in your dbt repository to ground its answers and plan a writeback before opening a pull request. Use it when the question is about how a model is built (its SQL, refs, or `dbt_project.yml` config), or to confirm what the file currently contains before asking for an edit. +The agent can also read source files in your project repository to ground its answers and plan a writeback before opening a pull request. Use it to inspect a native model's `sql_from`, dimensions, or metrics, a dbt model's SQL and configuration, or the current contents of a file before asking for an edit. -For questions about what data exists, what a metric means, or whether two metrics overlap, the agent stays on the semantic-layer tools — the repo is for dbt implementation details, not the catalogue of what's queryable. +For questions about what data exists, what a metric means, or whether two metrics overlap, the agent uses semantic-layer discovery tools. Repository access supplies the source definitions behind those fields. ### Requirements -- Your project's dbt connection must be a **GitHub** or **GitLab** repository with the matching Lightdash app installed. On GitLab, repository-wide code search isn't available, so the agent falls back to reading and searching within files it lists. +- Repository reading is available for **GitHub** or **GitLab** connections with the matching Lightdash app installed. On GitLab, repository-wide code search isn't available, so the agent falls back to reading and searching within files it lists. - The user asking the question needs the **view source code** permission on the project. - In Slack, read access is only available when [AI Agents OAuth](/integrations/slack#ai-agents-configuration) is required, so the agent can evaluate permissions against the asking user. ### What the agent can see -- The dbt project is mounted at the connection's `project_sub_path`, and that is where the agent is pointed. Other repositories readable by your organization's Lightdash app installation (or your linked GitHub account) can also be browsed read-only on their default branch; secret and credential file paths are blocked. +- The agent starts in the configured project directory. Other repositories readable by your organization's Lightdash app installation (or your linked GitHub account) can also be browsed read-only on their default branch; secret and credential file paths are blocked. - The agent can list directories, read files, and search for strings, but it cannot modify anything. To change a file, it still has to call `editDbtProject`. Each repo command shows up as its own sub-step under a terminal icon, so you can see exactly what the agent looked at before answering or proposing a change. @@ -176,9 +178,9 @@ When the preview is ready: - The agent's reply in chat (or the writeback status on a review item) includes a **View preview** link to the preview project. - A bot comment is posted on the pull request with the same preview URL. -The preview project is built using the same dbt connection as your production project, with the branch overridden to the PR's head branch. It's compiled by Lightdash's own scheduler — the customer repository doesn't need to install or run the Lightdash preview-deploy GitHub Action. +The preview project uses the production project's connection and semantic layer format, with the branch overridden to the PR's head branch. Native projects compile their YAML directly; dbt projects use dbt. Lightdash builds the preview without requiring a preview-deploy GitHub Action in your repository. -If the preview can't be built — for example, the project isn't connected to GitHub, the dbt compile fails on the PR branch, or the GitHub App can't reach the repo — the pull request is still opened and the agent reports the PR link without a preview. The writeback itself never fails because the preview failed. +If the preview can't be built — for example, the project isn't connected to GitHub, compilation fails on the PR branch, or the GitHub App can't reach the repo — the pull request is still opened and the agent reports the PR link without a preview. The writeback itself never fails because the preview failed. ## Iterating on an existing pull request @@ -199,7 +201,7 @@ metric is rounded to 2 decimal places. The pasted pull request must: -- Live in the same GitHub repository as the project's dbt connection. +- Live in the same GitHub repository as the project's connection. - Be **open** — merged or closed PRs are rejected. - Have its branch in the same repository (PRs opened from forks are rejected). @@ -209,8 +211,8 @@ If any of those checks fail, the agent stops and tells you why instead of silent | Situation | Result | | --- | --- | -| Project's dbt connection isn't GitHub, GitLab, or Bitbucket Cloud | The agent tells you the project must be connected to a supported git host for writeback. | -| GitHub App isn't installed on the repo | The agent surfaces a setup error. Install the Lightdash GitHub App on the repository (from your project's dbt connection settings) and try again. | +| Project has no supported repository connection | The agent explains which connection is required. See [Prerequisites](#prerequisites). | +| GitHub App isn't installed on the repo | The agent surfaces a setup error. Install the Lightdash GitHub App on the repository from your project's connection settings and try again. | | Writeback agent makes no file changes | No pull request or merge request is opened. The agent reports back that nothing needed to change. | | Pasted PR link is in a different repo, merged, closed, or from a fork | The agent rejects the link with an explanation and does not open a new pull request. | | The pull request behind a change has been merged or closed | The agent can't add to that pull request, so a follow-up opens a fresh one in the same thread. | @@ -219,5 +221,6 @@ If any of those checks fail, the agent stops and tells you why instead of silent ## Related -- [AI coding agent](/agents/ai-coding-agent) — the general-purpose counterpart to AI writeback for changes to any connected repository (not just the dbt project). Verification is delegated to the target repo's own CI. *(Beta)* +- [AI coding agent](/agents/ai-coding-agent) — make changes in other connected repositories. - [dbt write-back](/integrations/dbt/write-back) — write back individual custom metrics, dimensions, and SQL Runner queries from the Lightdash UI. +- [Lightdash YAML write-back](/semantic-layer/yaml#write-back-from-lightdash) — promote custom fields into native models. diff --git a/agents/issues.mdx b/agents/issues.mdx index fd18133c..1f075abd 100644 --- a/agents/issues.mdx +++ b/agents/issues.mdx @@ -106,7 +106,7 @@ For AI-filed findings, the modal also has an **Evidence** section that shows the For each issue, Lightdash proposes the smallest change that would prevent it: -- **Semantic layer fixes** open a pull request against your dbt project (rename a field, add a description, add a metric). +- **Semantic layer fixes** open a pull request against your dbt or native Lightdash YAML project (rename a field, add a description, add a metric). - **Project context fixes** add a short note to a shared file that your agents read before answering future questions. Writeback runs the same way whether the issue was AI-filed or human-filed. A manual issue is eligible for writeback when it has: @@ -119,10 +119,10 @@ For manual issues, the **Related explores** you picked when filing are passed in | Issue type | What it means | How to fix | | --- | --- | --- | -| **Semantic layer** | A field, metric, or description is missing, ambiguous, or wrong in your dbt project. | Run writeback to open a pull request against your dbt repository. | +| **Semantic layer** | A field, metric, or description is missing, ambiguous, or wrong in your semantic layer. | Run writeback to open a pull request against your project repository. | | **Project context** | Your agents are missing background knowledge to answer reliably (e.g. which table to use for "active customers"). | Run writeback to add a note your agents read before answering. | -Project context fixes write to a shared file — `lightdash.project_context.yml` — that lives next to `lightdash.config.yml` in your dbt repo. See [Agent context](/agents/agent-context) for the file shape, entry schema, and how to edit it by hand. +Project context fixes update [agent context](/agents/agent-context) in your project repository. For native projects, see [AI writeback and Issues](/semantic-layer/yaml#ai-writeback-and-issues) for connection and validation behavior. ## Project context vs. semantic layer fixes @@ -130,11 +130,11 @@ Both kinds of fix open a pull request, but they change different things and have | | Semantic layer fix | Project context fix | | --- | --- | --- | -| **What it changes** | Your dbt model YAML — renames a field, adds a description, adds a metric, fixes a join. | A single entry in `lightdash.project_context.yml`. | +| **What it changes** | Your dbt or native model YAML — renames a field, adds a description, adds a metric, fixes a join. | A single entry in `lightdash.project_context.yml`. | | **Who else sees the change** | Everyone querying the model, in Lightdash and downstream. | AI agents only. | -| **How the PR is built** | A writeback agent runs in a sandbox against your dbt project and proposes the edit. | Deterministic merge into the YAML file via the GitHub API — no sandbox. | -| **When to use it** | The data model itself is wrong, ambiguous, or under-described. The fix would help SQL users too, not just the agent. | The data model is fine, but the agent is missing background knowledge — terminology, which table to prefer, business rules that aren't expressible as dbt metadata. | -| **Reversibility** | Same as any dbt change — revert the PR. | Delete or edit the entry in `lightdash.project_context.yml`. | +| **How the PR is built** | A writeback agent runs in a sandbox against your project and proposes the edit. | Deterministic merge into the YAML file via the GitHub API — no sandbox. | +| **When to use it** | The data model itself is wrong, ambiguous, or under-described. The fix would help SQL users too, not just the agent. | The data model is fine, but the agent is missing background knowledge — terminology, which table to prefer, business rules that aren't expressible as semantic definitions. | +| **Reversibility** | Revert the PR. | Delete or edit the entry in `lightdash.project_context.yml`. | A rule of thumb: if a new analyst joining the team would also benefit from the change, it's probably a semantic layer fix. If only the agent needs to know it, it's project context. diff --git a/agents/lightdash-mcp.mdx b/agents/lightdash-mcp.mdx index f4775385..94f1c28b 100644 --- a/agents/lightdash-mcp.mdx +++ b/agents/lightdash-mcp.mdx @@ -444,11 +444,11 @@ MCP can read, create, and edit Lightdash charts and dashboards using the same [c These tools reuse the same permissions, validation, and project context as the [Lightdash CLI](/workflow/cli/install) `download` and `upload` commands, so the user driving the MCP session needs the same access required to manage that content in Lightdash. -#### Editing the dbt project +#### Editing project models -MCP exposes [AI writeback](/agents/ai-writeback) — editing the dbt project that backs the active Lightdash project and opening a pull request with the change — to any MCP client. Ask the assistant to rename a metric, add a dimension, edit a model's SQL, or fix a YAML description; the assistant calls the tools below and hands back a pull request URL when the run finishes. The [AI writeback prerequisites](/agents/ai-writeback#prerequisites) — supported git host, provider access, and project Developer permission — apply to the MCP surface too. +MCP exposes [AI writeback](/agents/ai-writeback) — editing the dbt or native Lightdash YAML project that backs the active Lightdash project and opening a pull request with the change — to any MCP client. Ask the assistant to rename a metric, add a dimension, edit field SQL, or fix a YAML description; the assistant calls the tools below and hands back a pull request URL when the run finishes. The [AI writeback prerequisites](/agents/ai-writeback#prerequisites) — supported Git host, repository access, and project Developer permission — apply to the MCP surface too. -- **Run AI writeback** (`run_ai_writeback`) - Start a writeback run against the active project's dbt repository from a natural-language prompt. The target GitHub, GitLab, or Bitbucket Cloud repository and dbt sub-folder are resolved server-side from the project's dbt connection — you don't specify them. Returns immediately with an `aiWritebackRunUuid`. +- **Run AI writeback** (`run_ai_writeback`) - Start a writeback run against the active project's repository from a natural-language prompt. The repository, project directory, and semantic layer format are resolved server-side from the saved connection — you don't specify them. Returns immediately with an `aiWritebackRunUuid`. - **Get AI writeback status** (`get_ai_writeback_status`) - Poll a writeback run by id. Returns the current pipeline stage while the run is in flight, and the pull request URL (or an error message) once it reaches a terminal state. ##### How the async flow works diff --git a/agents/set-up-agents.mdx b/agents/set-up-agents.mdx index 36284b0a..82adaaf1 100644 --- a/agents/set-up-agents.mdx +++ b/agents/set-up-agents.mdx @@ -220,7 +220,7 @@ Use tags in the Lightdash metadata to control which metrics and dimensions the a ### Enable AI writeback (optional) -Allow your agent to edit your dbt project from chat — renaming metrics, adding dimensions, or updating descriptions — and open a pull request with the changes for review. Read more on the [AI writeback](/agents/ai-writeback) feature. +Allow your agent to edit dbt or native Lightdash YAML models from chat — renaming metrics, adding dimensions, or updating descriptions — and open a pull request for review. See [AI writeback](/agents/ai-writeback) for supported connections and setup. ## Setting up multiple agents diff --git a/agents/use-ai-agents.mdx b/agents/use-ai-agents.mdx index bbe507c4..d60df7fa 100644 --- a/agents/use-ai-agents.mdx +++ b/agents/use-ai-agents.mdx @@ -15,10 +15,10 @@ AI agents in Lightdash allow you to: - **Guide you to the right data** - Direct you to the most relevant explores or tables for your questions - **Discover existing content** - Find and share relevant charts and dashboards that have already been created in your project - **Generate complete dashboards** - Create multiple related visualizations at once that tell a cohesive story about your data, perfect for executive summaries or thematic analyses -- **Open pull requests against your dbt repository** - Ask the agent to rename a metric, add a dimension, or edit a YAML file and it can open a pull request for review. See [AI writeback](/agents/ai-writeback). +- **Open pull requests against your project repository** - Ask the agent to rename a metric, add a dimension, or edit a dbt or native Lightdash YAML model and it can open a pull request for review. See [AI writeback](/agents/ai-writeback). - **Compare across time periods** - Ask for month-over-month, year-over-year, or any custom period offset, and the agent adds a comparison column next to each metric automatically -As mentioned earlier, Lightdash agents use the **semantic layer** defined in your dbt models to understand your data structure, relationships, and business logic. +Lightdash agents use the **semantic layer** defined in your models to understand your data structure, relationships, and business logic. This ensures that the AI generates accurate queries and visualizations based on your specific data context. So, *when an Agent generates an answer, the output is a semantic query, **not SQL!*** This means that you can easily swap between the conversational AI interface and the standard Lightdash exploration experience. diff --git a/get-started/quickstart/connect-project.mdx b/get-started/quickstart/connect-project.mdx index 741d0034..5d7fc6cc 100644 --- a/get-started/quickstart/connect-project.mdx +++ b/get-started/quickstart/connect-project.mdx @@ -1,14 +1,14 @@ --- -title: "Connect your warehouse and dbt project" +title: "Connect your warehouse and project" sidebarTitle: "Connect your project" -description: "Connection settings for every supported warehouse, plus every way to import a dbt project" +description: "Connect your warehouse and choose a dbt or native Lightdash YAML project" doc-type: reference --- To setup your Lightdash connection you'll need to: 1. [Connect to your data warehouse](/get-started/quickstart/connect-project#1-connect-to-a-warehouse) -2. [Connect to your dbt project](/get-started/quickstart/connect-project#2-import-a-dbt-project) +2. Connect a [dbt project](#2-import-a-dbt-project) or [native Lightdash YAML project](/semantic-layer/yaml#connect-through-github) You can see a list of every time your project is compiled by visiting the [project compilation history](/workspace-admin/project-compilation-history) page. @@ -1335,6 +1335,10 @@ Pick your repository type and follow the guide below: [GitHub](#github), [GitLab ### GitHub +##### Semantic layer format + +Choose **dbt** for a dbt project or **Native Lightdash YAML** for standalone Lightdash models. See [Connect through GitHub](/semantic-layer/yaml#connect-through-github) for native setup and supported features. The dbt-specific settings below apply when **dbt** is selected. + ###### OAuth (recommended authorization method) We recommend you connect to Github using OAuth. This gives Lightdash a direct connection to the repo, so it won't lose access when individuals leave the company, and it can create pull requests, which is required for [dbt write-back features](/integrations/dbt/write-back). This connection lives at the organization level, so you can map different dbt project repos to each Lightdash project without needing a new access token. @@ -1383,7 +1387,7 @@ By default, we've set this to `main` but you can change it to whatever you'd lik ##### Project directory path -This is the folder where your `dbt_project.yml` file is found in the GitHub repository you entered above. +For **dbt**, this is the folder where your `dbt_project.yml` file is found in the GitHub repository you entered above. For **Native Lightdash YAML**, use the [native project directory](/semantic-layer/yaml#project-layout). * Put `/` if your `dbt_project.yml` file is in the main folder of your repo (e.g. lightdash/lightdash-analytics/dbt\_project.yml) @@ -1396,7 +1400,7 @@ If you've [customized the domain for your GitHub pages](https://docs.github.com/ By default, this is `github.com` - After adding your Github information, fill out the [dbt project details](/integrations/dbt/projects#dbt-project-settings) and you're all set! + If you selected **dbt**, fill out the [dbt project details](/integrations/dbt/projects#dbt-project-settings). For **Native Lightdash YAML**, save the connection to compile your models. *** diff --git a/integrations/dbt/write-back.mdx b/integrations/dbt/write-back.mdx index a1dfc9e5..046c13c0 100644 --- a/integrations/dbt/write-back.mdx +++ b/integrations/dbt/write-back.mdx @@ -8,6 +8,8 @@ description: "Develop models and metrics in Lightdash and open a pull request ag - You need to be at least a [`project Developer`](/workspace-admin/roles) to use dbt Write-Back. +For a project using standalone YAML, see [Lightdash YAML write-back](/semantic-layer/yaml#write-back-from-lightdash). + ## Bitbucket Cloud setup Connect your project using the [Bitbucket token setup instructions](/get-started/quickstart/connect-project#bitbucket-cloud-api-token). All three workflows below use that stored project token and the configured repository, branch, and dbt directory. @@ -26,7 +28,7 @@ We recommend using dbt write-back for custom metrics that you're creating freque [Metrics](/semantic-layer/metrics) written to your dbt project (or YAML Metrics) are reusable by everyone in your project and will show up in the list of available metrics for a Table. [Custom Metrics](/explore/create-custom-fields#custom-metrics) are only saved in the chart they're used in and will not be available if you or others open the same Table to build a new chart. -To get started, create a [Custom Metric](/explore/create-custom-fields#custom-metrics) in the Explorer. Hover over the Custom Metric in the sidebar and click on the three-dot menu, then `Write back to dbt`. +To get started, create a [Custom Metric](/explore/create-custom-fields#custom-metrics) in the Explorer. Hover over the Custom Metric in the sidebar and click on the three-dot menu, then **Write back to project**. ![Three stages of metric write-back: the custom metric menu with Write back to dbt, the confirmation modal, and the opened GitHub pull request](/images/integrations/dbt/write-back/write-back-metrics.jpg) @@ -35,7 +37,7 @@ To get started, create a [Custom Metric](/explore/create-custom-fields#custom-me **Write-back requires a base dimension column.** - `Write back to dbt` is only available for custom metrics that were built from a dimension (the YAML write-back attaches the metric to that dimension in the dbt model). Custom metrics that were [cloned from an existing metric](/explore/create-custom-fields#creating-a-custom-metric-from-an-existing-metric) do not have a base dimension, so the option is hidden from both the individual metric menu and the bulk `Write back to dbt` action in the sidebar's Custom metrics section. + **Write back to project** is only available for custom metrics that were built from a dimension (the YAML write-back attaches the metric to that dimension in the dbt model). Custom metrics that were [cloned from an existing metric](/explore/create-custom-fields#creating-a-custom-metric-from-an-existing-metric) do not have a base dimension, so the option is hidden from both the individual metric menu and the bulk write-back action in the sidebar's Custom metrics section. Once the pull request with your new Metric is merged, you can click `Refresh dbt` in Lightdash (or, if you're [using GitHub Actions](/workflow/set-up-ci-cd#deploy-changes-to-lightdash), your project will automatically refresh once your changes are merged) and your Custom Metric will be replaced by your new YAML Metric automatically. @@ -60,9 +62,9 @@ We recommend using dbt write-back for Custom Dimensions that you're creating fre [Dimensions](/semantic-layer/dimensions) written to your dbt project (or YAML Dimensions) are reusable by everyone in your project and will show up in the list of available Dimensions for a Table. [Custom Dimensions](/explore/create-custom-fields#custom-dimensions) are only saved in the chart they're used in and will not be available if you or others open the same Table to build a new chart. -To get started, create a [Custom Dimension](/explore/create-custom-fields#custom-dimensions) in the Explorer. Hover over the Custom Dimension in the sidebar and click on the three-dot menu, then `Write back to dbt`. +To get started, create a [Custom Dimension](/explore/create-custom-fields#custom-dimensions) in the Explorer. Hover over the Custom Dimension in the sidebar and click on the three-dot menu, then **Write back to project**. -1. Click `Write back to dbt` +1. Click **Write back to project** ![The three-dot menu on a custom dimension, with Write back to dbt among the options](/images/integrations/dbt/write-back/write-back-custom-dimension-1.png) @@ -151,4 +153,4 @@ To create a job that does this, you'll want to: ![A dbt Cloud job triggered by pull requests, running dbt run --select tag:created-by-lightdash against the Prod environment](/images/integrations/dbt/write-back/dbt-cloud-run-automatically-561d3a59b2dc47a003adaf926fbe5a3b.jpg) - \ No newline at end of file + diff --git a/semantic-layer/yaml.mdx b/semantic-layer/yaml.mdx index 43b9e7d0..5695f9bb 100644 --- a/semantic-layer/yaml.mdx +++ b/semantic-layer/yaml.mdx @@ -1,7 +1,7 @@ --- title: "Lightdash YAML" sidebarTitle: Lightdash YAML -description: "Use Lightdash without a dbt project by defining your semantic layer directly in YAML files" +description: "Define models in YAML, connect them through GitHub, and update them without dbt" --- @@ -10,33 +10,33 @@ description: "Use Lightdash without a dbt project by defining your semantic laye ## What is Lightdash YAML? -Lightdash YAML allows you to use Lightdash without an existing dbt project. Instead of defining your semantic layer within dbt model YAML files, you define it directly in standalone YAML files that point to tables in your data warehouse. +Lightdash YAML defines your semantic layer in standalone model files that point to existing tables in your warehouse. You can build metrics, dimensions, charts, dashboards, and AI agents without installing dbt or maintaining a dbt project. -This approach lets you leverage Lightdash's powerful features without needing to adopt dbt first. +You can [connect the project through GitHub](#connect-through-github) so Lightdash compiles and refreshes it from your repository, or [deploy local files with the CLI](#step-4-deploy-your-project). A GitHub connection also lets you write custom fields and AI-proposed changes back to the repository as pull requests. -## Why use Lightdash YAML? +This page covers the [project layout](#project-layout), [model setup](#define-your-models), [GitHub connection](#connect-through-github), and [write-back support](#write-back-from-lightdash). -Lightdash has always operated with dbt at its core. Traditionally, customers have active dbt projects, and Lightdash builds its semantic layer within that dbt context. +## Choose a semantic layer format -However, not every team uses dbt. If you're interested in Lightdash features like: +Use the [dbt integration](/integrations/dbt/projects) when your semantic definitions live in a dbt project. Use Lightdash YAML when your warehouse tables already exist and you want to define the semantic layer independently of dbt. Lightdash YAML describes those tables; it does not build or transform them. -- **AI agents** that can answer questions about your data -- **A semantic layer** with consistent metric definitions -- **Self-service analytics** for your business users +The formats share Lightdash's metric and dimension concepts, but they use different file structures. Native model files have a top-level `type`, `name`, and `sql_from`; dbt model files use dbt's model and `meta` structure. Changing the connection's format does not convert your files. -...but you don't have dbt set up, Lightdash YAML provides a path forward. You can define your semantic layer directly and start using Lightdash immediately, without the overhead of adopting dbt. +## Project layout -## dbt vs Lightdash YAML: which should you use? +Keep `lightdash.config.yml` in the project directory, with model files under `models/`: -| Scenario | Recommendation | -|----------|----------------| -| You already have a dbt project | Use the standard [dbt integration](/integrations/dbt/projects) | -| You're planning to adopt dbt soon | Consider setting up dbt first, then connecting to Lightdash | -| You don't use dbt and want to try Lightdash quickly | Use Lightdash YAML | -| You want AI agents or semantic layer features without dbt | Use Lightdash YAML | -| You have tables in your warehouse ready to explore | Use Lightdash YAML | +```text +lightdash.config.yml +models/ + users.yml + sales/ + orders.yaml +``` + +Lightdash reads `.yml` and `.yaml` files recursively. It also supports `lightdash/models/` when there is no `models/` directory; if both exist, only `models/` is used. Each model needs a unique `name` and a `type` of `model`, `model/v1beta`, or `model/v1`. -The good news: if you start with Lightdash YAML and later decide to adopt dbt, the YAML formats are compatible, so migration is straightforward. +The model name does not need to match its filename. Lightdash tracks the source path so write-back updates the original file, including models nested in subdirectories. Keep native model files inside the configured project directory. ## Recommended: build with an AI coding agent @@ -62,7 +62,7 @@ The fastest way to get started with Lightdash YAML is to let an AI coding agent Example prompt: - ``` + ```text /developing-in-lightdash Profile my warehouse, create Lightdash YAML models for the users and orders tables with sensible metrics and dimensions, then deploy the project to Lightdash. @@ -72,7 +72,7 @@ The fastest way to get started with Lightdash YAML is to let an AI coding agent Prefer to set things up manually? Follow the step-by-step guide below. -## Getting started with Lightdash YAML +## Define your models ### Prerequisites @@ -96,15 +96,15 @@ Replace `snowflake` with your warehouse type (e.g., `bigquery`, `databricks`, `r ### Step 2: Create your first model -Lightdash YAML uses the same syntax as the [Lightdash semantic layer in dbt](/integrations/dbt/projects), but instead of nesting everything under `meta` tags, all configuration is at the top level of the file. +Define the table in `sql_from`, then add dimensions and metrics using the native model structure below. Create a directory structure for your Lightdash project: ```bash -mkdir -p lightdash/models +mkdir -p models ``` -Create a YAML file for your first model. For example, `./lightdash/models/users.yml`: +Create a YAML file for your first model. For example, `models/users.yml`: ```yaml # Metadata @@ -152,17 +152,25 @@ Run [`lightdash lint`](/workflow/cli/lint) to check your YAML files for errors b lightdash lint ``` +Then [compile the models](/workflow/cli/compile) to check field and join references: + +```bash +lightdash compile --no-warehouse-credentials --no-partial-compilation +``` + +The CLI uses `warehouse.type` in `lightdash.config.yml` to compile SQL for your warehouse. Compilation does not create the physical tables or confirm that every physical column exists. + ### Step 4: Deploy your project -Create your Lightdash project by deploying with the CLI — see [`lightdash deploy`](/workflow/cli/deploy) for the full command reference: +Choose [GitHub](#connect-through-github) to have Lightdash compile the committed files, or create a project from your local files with the CLI: ```bash lightdash deploy --create --no-warehouse-credentials ``` -The `--no-warehouse-credentials` flag tells Lightdash that you're deploying without warehouse credentials embedded in the CLI. After deploying, you'll need to configure your warehouse connection in the Lightdash UI: +With `--no-warehouse-credentials`, the CLI does not read a dbt profile or send warehouse credentials. Configure the created project's warehouse connection in Lightdash before querying it: -1. Go to **Settings** (gear icon in the top right) +1. Go to **Settings** 2. Under **Current project**, click **Connection settings** 3. Configure your **Warehouse connection** with your database credentials @@ -180,6 +188,64 @@ After your initial deployment, you can edit your `.yml` files and redeploy chang lightdash deploy --no-warehouse-credentials ``` +For a GitHub-connected project, commit and merge the model changes, then [refresh from the repository](#refresh-a-github-connected-project). + +## Connect through GitHub + +You need permission to manage the project's connection and access to the GitHub repository. Use the [GitHub connection settings](/get-started/quickstart/connect-project#github) for authentication options. Install the Lightdash GitHub App on the repository to use write-back. + +1. Commit your configuration and models to GitHub. +2. Open **Settings → All projects → Create new**, or **Settings → Current project → Connection settings** for an existing project. +3. Configure the warehouse connection, and choose **GitHub** as the project connection type. +4. Set **Semantic layer format** to **Native Lightdash YAML**. +5. Select the repository and branch. Set **Project directory path** to `/` for the repository root, or the directory containing `lightdash.config.yml` and the model directory. For example, use `/analytics` for `analytics/lightdash.config.yml` and `analytics/models/`. +6. Save the connection. Lightdash reads the committed files and compiles them using the project's saved warehouse connection. + +You can add this connection to an existing CLI-created project. Updating that project's connection keeps its project ID, warehouse credentials, charts, and dashboards. Keep model and field names stable so saved content still resolves. + +Native connections do not use dbt versions, profiles, targets, selectors, or dbt environment variables. GitHub is the supported Git provider for native YAML; native projects cannot be combined with additional dbt sources. + +### Refresh a GitHub-connected project + +After merging changes to the configured branch, click **Refresh models** in Lightdash or run [`lightdash refresh`](/workflow/cli/reference#lightdash-refresh). The server pulls that branch and compiles the YAML using the saved warehouse connection. You do not need dbt or a local checkout to trigger a refresh. + +Invalid YAML, invalid model definitions, duplicate model names, or semantic compilation errors fail the native refresh instead of publishing an incomplete model set. Open [Compilation history](/workspace-admin/project-compilation-history) to inspect the result. + +To test another branch first, [create a preview project](/workflow/preview-projects#lightdash-app). The preview retains the native semantic layer format. + +## Write back from Lightdash + +In a GitHub-connected native project, users with [project Developer permissions](/workspace-admin/roles) or above can open pull requests from the Explore view: + +1. Create a [custom metric or dimension](/explore/create-custom-fields). +2. Open the field's menu and choose **Write back to project**. You can also use the write-back action on the Custom metrics or Custom dimensions section to select several fields. +3. Review the YAML preview and click **Open Pull Request**. +4. Review and merge the pull request, then refresh the project. + +The edit stays in the original model file and preserves comments and unrelated definitions. If a field name already exists, Lightdash rejects the write instead of replacing its definition. + +### Supported fields and limits + +| Field | Native YAML write-back | +| --- | --- | +| Custom metric based on a dimension | Adds the metric under that native dimension's `metrics` | +| Custom metric cloned from an existing metric | Not supported: it has no base dimension | +| Generated period-comparison metric | Not supported | +| Custom SQL dimension | Adds a dimension with its SQL and type | +| Custom bin dimension | Adds a string dimension with warehouse-specific binning SQL | + +Metrics with filters that cannot be represented without changing their meaning are rejected. Write-back requires block-style YAML dimensions rather than inline, flow-style lists. + +Existing saved charts keep their custom bin dimensions so their ordering is unchanged. Use the written YAML dimension in new charts; add a separate numeric ordering dimension in the model when bin order matters. Custom dimensions are not automatically replaced in saved charts after write-back. + +SQL Runner's **Write back to dbt** action creates dbt SQL models and is not available for native projects. Native `sql_from` definitions point to tables that your warehouse workflow already creates. + +## AI writeback and Issues + +[AI writeback](/agents/ai-writeback) can edit native YAML from a chat request. It validates the edited native models before pushing changes, using the connected warehouse's SQL dialect without passing warehouse credentials to the sandbox. A validation error prevents the pull request from being opened or updated. + +[Issues](/agents/issues#fixing-an-issue-with-writeback) uses the same native path for semantic layer fixes. Project context fixes update [agent context](/agents/agent-context) in `lightdash.project_context.yml` beside the project configuration. GitHub AI writeback also supports [automatic preview projects](/agents/ai-writeback#automatic-preview-project). + ## Developing with AI coding agents If you're developing with Cursor, Claude Code, or another AI coding agent, you can speed up your workflow significantly. diff --git a/workflow/cli/compile.mdx b/workflow/cli/compile.mdx index 4d914891..0d49a33c 100644 --- a/workflow/cli/compile.mdx +++ b/workflow/cli/compile.mdx @@ -1,15 +1,17 @@ --- title: "Compilation" sidebarTitle: "Compilation" -description: How Lightdash compiles your dbt project into explores, and how to control compilation strictness +description: Compile dbt or Lightdash YAML models into explores and control compilation strictness --- import StrictCompilationFlags from '/snippets/strict-compilation-flags.mdx'; -`lightdash compile` builds your dbt project's .yml files into Lightdash explores and reports any errors that would break your Lightdash project. For example, a metric that references a dimension that doesn't exist. +`lightdash compile` builds dbt or native Lightdash YAML models into explores and reports semantic errors, such as a metric referencing a dimension that does not exist. Compilation also runs inside [`lightdash validate`](/workflow/cli/validate), [`lightdash deploy`](/workflow/cli/deploy), and [`lightdash preview`](/workflow/preview-projects), so the behaviour and flags on this page apply to those commands too. +For native projects, the CLI detects the [Lightdash YAML project layout](/semantic-layer/yaml#project-layout) and compiles without running dbt or loading a dbt profile. See [Validate your YAML](/semantic-layer/yaml#step-3-validate-your-yaml) for the configuration and command. dbt selectors and warehouse-column probes do not apply to this path. + ## Choose partial or strict compilation diff --git a/workflow/cli/deploy.mdx b/workflow/cli/deploy.mdx index 02adfd3c..b42f24f9 100644 --- a/workflow/cli/deploy.mdx +++ b/workflow/cli/deploy.mdx @@ -1,23 +1,22 @@ --- title: Deploy changes to production sidebarTitle: Deploy changes -description: Push your local dbt project to production, or refresh metadata from your connected git repository +description: Deploy local models or refresh your project from its connected Git repository doc-type: tutorial --- import StrictCompilationFlags from '/snippets/strict-compilation-flags.mdx'; -`lightdash deploy` pushes your dbt project to Lightdash. +`lightdash deploy` pushes your local dbt or native Lightdash YAML models to Lightdash. -Once you've made updates to your dbt project and you're ready to make those changes available in your Lightdash production project, you have a few options for deploying from the CLI or the Lightdash UI. -This guide will walk you through how it works, when to use each option, and what to keep in mind for a safe and scalable workflow. +Use it to publish local changes, or use `lightdash refresh` to compile the files already committed to your connected repository. For native models, follow the [Lightdash YAML deployment steps](/semantic-layer/yaml#step-4-deploy-your-project); the profiles and targets in the CLI examples below apply to dbt. ## Before you deploy: A quick word of caution **`lightdash deploy` pushes directly to your production project.** -Running this command uses your local dbt profile (from your `profiles.yml`). If your default target points to a dev or staging environment, `lightdash deploy` will overwrite your production semantic layer with that configuration, breaking dashboards for all users. +For dbt projects, this command uses your local dbt profile (from your `profiles.yml`). If your default target points to a dev or staging environment, `lightdash deploy` will overwrite your production semantic layer with that configuration, breaking dashboards for all users. If you're working locally, **use [`lightdash preview`](/workflow/preview-projects) instead**. It creates a temporary, isolated project that won't affect production. @@ -73,7 +72,7 @@ Do not combine `--ignore-errors` with this strict CI pattern — `--ignore-error ## Option 2: Safely refresh metadata using `lightdash refresh` -If you only need to refresh your Lightdash project to reflect upstream dbt changes (like updated models or docs), but don’t want to deploy your local dbt setup, you can use: +To pull committed changes from the configured branch of your project's Git repository, run: ```bash lightdash refresh @@ -81,16 +80,15 @@ lightdash refresh Unlike deploy, this: -* Uses the saved dbt credentials from Lightdash, not your local profile +* Uses the connection saved in Lightdash -* Simply re-runs the metadata sync and model parsing +* Compiles the repository using the project's selected semantic layer format -* Makes it safe to trigger from anywhere without risk of pointing to the wrong target +* Leaves your local files and dbt profile out of the refresh You can also do this directly from the Lightdash UI: -Go to your project → "Manage project" → Refresh dbt. -This gives the same result as running `lightdash refresh`. +Click **Refresh dbt** for a dbt project or **Refresh models** for a native YAML project. This gives the same result as running `lightdash refresh`. The Refresh dbt button highlighted at the top of the explore view, above the filters bar diff --git a/workflow/cli/reference.mdx b/workflow/cli/reference.mdx index d8e72b26..308e5bd9 100644 --- a/workflow/cli/reference.mdx +++ b/workflow/cli/reference.mdx @@ -309,9 +309,9 @@ This command displays all available projects (excluding preview projects) that y ### `lightdash compile` -Compile Lightdash resources using your local project and database credentials. dbt gets compiled first, then your Lightdash explores. If you use [dbt node selection](#dbt-node-selection) to only compile a subset of models, Lightdash will also compile models joined to those models to ensure no field references are broken. +Compile Lightdash resources using your local project. For dbt projects, dbt gets compiled first, then your Lightdash explores. If you use [dbt node selection](#dbt-node-selection) to only compile a subset of models, Lightdash will also compile models joined to those models to ensure no field references are broken. -All standard [dbt options](#dbt-options) work with `lightdash compile`. +For native projects, see [Lightdash YAML validation](/semantic-layer/yaml#step-3-validate-your-yaml). The CLI detects native models and skips dbt compilation and the warehouse catalog. Standard [dbt options](#dbt-options) apply to dbt projects. #### Partial compilation @@ -543,7 +543,7 @@ Compiles and deploys the current project to your [selected Lightdash Cloud proje `lightdash deploy` pushes directly to your production project. Use [`lightdash preview`](#lightdash-preview) for local development and [CI/CD](/workflow/set-up-ci-cd#deploy-changes-to-lightdash) for production deploys. See [Deploy changes to production](/workflow/cli/deploy) for the full safe-deploy guidance. -All standard [dbt options](#dbt-options) work with `lightdash deploy`. +For native projects, see [Deploy Lightdash YAML](/semantic-layer/yaml#step-4-deploy-your-project). Standard [dbt options](#dbt-options) apply to dbt projects. **Options:** @@ -617,7 +617,7 @@ lightdash deploy --select "tag:marketing" --combine-manifest ./prod-manifest.jso ### `lightdash refresh` -Refreshes your hosted Lightdash project using the latest code from your linked Github repository. This is equivalent to pressing **Refresh dbt** in the UI as an admin. +Refreshes your hosted Lightdash project from the configured branch of its connected Git repository, using the saved warehouse connection. This is equivalent to pressing **Refresh dbt** for a dbt project or **Refresh models** for a native Lightdash YAML project. This command does not support using dbt options. @@ -627,14 +627,14 @@ The two CLI commands above (`lightdash deploy` and `lightdash refresh`) are ofte | | `lightdash deploy` | `lightdash refresh` | | --- | --- | --- | -| **What it does** | Pushes your local dbt project to a Lightdash project. Runs `dbt compile` (or `dbt list`) locally to produce a fresh `manifest.json`, then sends the compiled models/explores to the Lightdash server. | Tells the Lightdash server to re-pull from the connected git remote and recompile, server-side. No local dbt run, no local files involved — it's the API-driven equivalent of clicking **Refresh dbt** in the UI. | +| **What it does** | Compiles local dbt or native Lightdash YAML models and uploads them to Lightdash. dbt projects run `dbt compile` or `dbt list`; native projects compile directly from YAML. | Tells the Lightdash server to pull the connected Git branch and compile it using the saved semantic layer format and warehouse connection. | | **Where compilation happens** | Locally (laptop or CI runner) | Lightdash servers | | **Requires a remote git connection?** | No | Yes | | **Flags** | Many (`--project`, `--select`, `--full-refresh`, `--create`, `--use-batched-deploy`, etc.) | Essentially none (just `--verbose`) | -| **Use when** | You've changed dbt models or Lightdash YAML metadata locally and want those changes to land in a Lightdash project. | Lightdash already points at a git repo and you just want it to pull the latest. | +| **Use when** | You've changed models locally and want to publish those files to a Lightdash project. | Lightdash already points at a Git repo and you want it to pull the latest committed changes. | **Rule of thumb:** -- Local dbt code you want to push → `deploy` +- Local model changes you want to push → `deploy` - Lightdash already points at a git repo and you just want it to pull the latest → `refresh` ### `lightdash validate` diff --git a/workflow/preview-projects.mdx b/workflow/preview-projects.mdx index 1c3ceff2..6e2d974e 100644 --- a/workflow/preview-projects.mdx +++ b/workflow/preview-projects.mdx @@ -59,6 +59,8 @@ You can leave out the **branch** and **schema** fields and you'll get a preview By choosing a branch (and optionally setting a schema or environment variables) you can create a preview project that uses code from a different branch than your main project. This is perfect for previewing changes you made on a branch in dbt Cloud, for quick edits where you don't want to fire up the CLI, and for teammates who don't have local development or the CLI set up. +For a [native Lightdash YAML project](/semantic-layer/yaml#connect-through-github), the preview keeps the native format and compiles the selected branch without dbt. Its `sql_from` definitions still determine which physical tables are queried; selecting another branch does not create warehouse tables. + Note: this only works with GitHub. It is currently not possible to create previews from GitLab branches in the Lightdash App UI. @@ -283,4 +285,4 @@ If you are an **organization admin**, you can delete any preview project in the ### Pull Requests (GitHub Actions) -If you've added both the [start-preview and stop-preview workflows](/workflow/set-up-ci-cd#add-previews-to-pull-requests) in GitHub, then the preview environments you've created in your pull requests should automatically be deleted when the pull request gets merged. \ No newline at end of file +If you've added both the [start-preview and stop-preview workflows](/workflow/set-up-ci-cd#add-previews-to-pull-requests) in GitHub, then the preview environments you've created in your pull requests should automatically be deleted when the pull request gets merged.