diff --git a/agents/ai-writeback.mdx b/agents/ai-writeback.mdx index 2a79eaee..9d28858d 100644 --- a/agents/ai-writeback.mdx +++ b/agents/ai-writeback.mdx @@ -30,14 +30,14 @@ For other kinds of changes, the agent will pick a different tool: ## Prerequisites -- Use a supported repository connection: **GitHub**, **GitLab.com**, or **Bitbucket Cloud** for dbt; **GitHub** for native Lightdash YAML. Azure DevOps, dbt Cloud, and CLI-only connections are not supported for AI writeback. +- Use a supported repository connection: **GitHub**, **GitLab.com**, or **Bitbucket Cloud** for dbt; **GitHub** or **Bitbucket Cloud** for native Lightdash YAML. Azure DevOps, dbt Cloud, and CLI-only connections are not supported for AI writeback. - Configure repository access so the agent can open requests: - **GitHub** — install the Lightdash GitHub App on the repository. - **GitLab** — connect the Lightdash GitLab App for your organization. Writeback works against GitLab.com; self-hosted GitLab domains aren't reachable from the writeback sandbox yet. - **Bitbucket Cloud** — configure the [project API token and writeback permissions](/get-started/quickstart/connect-project#bitbucket-cloud-api-token). - You need at least [project Developer](/workspace-admin/roles) permissions on the project. -For native projects, follow [Connect through GitHub](/semantic-layer/yaml#connect-through-github) and select **Native Lightdash YAML** as the semantic layer format. +For native projects, follow [Connect through GitHub or Bitbucket Cloud](/semantic-layer/yaml#connect-through-github) and select **Native Lightdash YAML** as the semantic layer format. ## How it works @@ -63,9 +63,9 @@ If you're driving writeback from an external client via MCP rather than the in-p ### Bitbucket Cloud behavior -Bitbucket AI writeback edits the connected dbt repository. Follow-up requests can add commits to the same open pull request. If that request has been merged or declined, a follow-up starts a fresh pull request. +Bitbucket AI writeback edits dbt or native Lightdash YAML files in the connected repository. Follow-up requests can add commits to the same open pull request. If that request has been merged or declined, a follow-up starts a fresh pull request. -Review and merge Bitbucket pull requests in Bitbucket, then sync the project in Lightdash. The integration does not provide automatic preview projects, Bitbucket Pipelines status, or merging from the chat card. Repository browsing and file mentions have separate provider requirements, described below. Standalone Lightdash YAML projects, Bitbucket Server/Data Center, and general-purpose coding-agent access are outside this Bitbucket integration. +Review and merge Bitbucket pull requests in Bitbucket, then sync the project in Lightdash. The integration does not provide automatic preview projects, Bitbucket Pipelines status, or merging from the chat card. Repository browsing and file mentions have separate provider requirements, described below. Bitbucket Server/Data Center and general-purpose coding-agent access are outside this Bitbucket integration. ## Using it in chat diff --git a/get-started/quickstart/connect-project.mdx b/get-started/quickstart/connect-project.mdx index 5d7fc6cc..7bb385ca 100644 --- a/get-started/quickstart/connect-project.mdx +++ b/get-started/quickstart/connect-project.mdx @@ -1485,7 +1485,7 @@ If your dbt project is in a sub-folder in your repo (e.g. `lightdash/lightdash-a ### Bitbucket -Connect a dbt repository on Bitbucket using the project's **API Token** field. Choose **Bitbucket** in your project's connection settings. +Connect a dbt or native Lightdash YAML repository on Bitbucket Cloud using the project's **API Token** field. Choose **Bitbucket** in your project's connection settings. #### Bitbucket Cloud API token @@ -1509,18 +1509,19 @@ The token's owner must have access to the repository and permission to push bran #### Connection fields +- **Semantic layer format:** choose **dbt** or **Native Lightdash YAML**. For native setup and repository layout, see [Connect through GitHub or Bitbucket Cloud](/semantic-layer/yaml#connect-through-github). - **Username:** your Bitbucket username, with the exact capitalization shown in your Bitbucket personal settings. Use the username, not your email address. - **API Token:** the token created above. - **Repository:** the workspace and repository slug, such as `my-workspace/analytics`. Do not enter the full URL. - **Branch:** the branch Lightdash syncs from, such as `main`. Writeback opens a separate change branch and targets this configured branch. -- **Project directory path:** `/` if `dbt_project.yml` is at the repository root, or a subfolder such as `/dbt` if it lives at `dbt/dbt_project.yml`. -- **Host domain:** leave blank for Bitbucket Cloud, or use `bitbucket.org`. +- **Project directory path:** the directory containing `dbt_project.yml` for dbt, or `lightdash.config.yml` and your model directory for native YAML. Use `/` for the repository root. +- **Host domain:** for dbt, leave blank for Bitbucket Cloud or use `bitbucket.org`. Native YAML uses `bitbucket.org` automatically and hides this field. Bitbucket connection fields showing the username, required API token permissions, and repository slug -Save the connection and complete the [dbt project settings](/integrations/dbt/projects#dbt-project-settings). For the available writeback actions, see [dbt write-back](/integrations/dbt/write-back) and [AI writeback](/agents/ai-writeback). +Save the connection. For dbt, complete the [dbt project settings](/integrations/dbt/projects#dbt-project-settings). For the available writeback actions, see [dbt write-back](/integrations/dbt/write-back) and [AI writeback](/agents/ai-writeback). #### Updating credentials diff --git a/integrations/dbt/write-back.mdx b/integrations/dbt/write-back.mdx index 046c13c0..d2ac0fec 100644 --- a/integrations/dbt/write-back.mdx +++ b/integrations/dbt/write-back.mdx @@ -14,7 +14,7 @@ For a project using standalone YAML, see [Lightdash YAML write-back](/semantic-l Connect your project using the [Bitbucket token setup instructions](/get-started/quickstart/connect-project#bitbucket-cloud-api-token). All three workflows below use that stored project token and the configured repository, branch, and dbt directory. -Bitbucket support covers dbt projects on `bitbucket.org`. Standalone Lightdash YAML projects and Bitbucket Server/Data Center are not supported by this Bitbucket writeback integration. Review and merge the pull request in Bitbucket, then sync the project in Lightdash. Writeback does not run your dbt models, display Bitbucket Pipelines status, or create an automatic preview project. +Bitbucket support covers dbt projects on `bitbucket.org`. These UI workflows do not support standalone Lightdash YAML on Bitbucket or Bitbucket Server/Data Center. For native YAML on Bitbucket Cloud, use [AI writeback](/agents/ai-writeback#bitbucket-cloud-behavior). Review and merge the pull request in Bitbucket, then sync the project in Lightdash. Writeback does not run your dbt models, display Bitbucket Pipelines status, or create an automatic preview project. ## Write back Metrics from Custom Metrics diff --git a/self-host/enterprise-features/ai-writeback.mdx b/self-host/enterprise-features/ai-writeback.mdx index ae71d2dd..77556a9a 100644 --- a/self-host/enterprise-features/ai-writeback.mdx +++ b/self-host/enterprise-features/ai-writeback.mdx @@ -7,18 +7,16 @@ description: "Enable AI writeback on a self-hosted Lightdash instance" Enterprise AI writeback needs a valid `LIGHTDASH_LICENSE_KEY` set on your instance ([enterprise features](/self-host/enterprise-features) covers applying the key) before any of the configuration below takes effect. -AI writeback lets an AI agent edit the dbt repository that backs a project and open a pull request (GitHub) or merge request (GitLab) with the change. The edits run inside an isolated [sandbox](/self-host/customize-deployment/sandboxes) where a Claude-driven agent applies the diff, runs `lightdash compile` to validate, and pushes a new branch. To enable the feature on a self-hosted Lightdash instance you need a configured sandbox provider, an Anthropic API key, and the matching Lightdash git app connected to the repositories you want writeback against. +AI writeback lets an AI agent edit the semantic layer repository that backs a project and open a pull request with the change. The edits run inside an isolated [sandbox](/self-host/customize-deployment/sandboxes). Validation depends on the project format; see [how AI writeback works](/agents/ai-writeback#how-it-works). To enable the feature on a self-hosted Lightdash instance you need a configured sandbox provider, an Anthropic API key, and repository access. For an overview of what the feature does and how to use it in chat, see [AI writeback](/agents/ai-writeback). ## Prerequisites - **Enterprise license** — `LIGHTDASH_LICENSE_KEY` must be set on your instance. -- **A configured sandbox provider** — see [sandboxes](/self-host/customize-deployment/sandboxes). The writeback agent edits files and runs `lightdash compile` there. +- **A configured sandbox provider** — see [sandboxes](/self-host/customize-deployment/sandboxes). The writeback agent edits files there. - **An Anthropic API key** — sign up at [console.anthropic.com](https://console.anthropic.com/) and create an API key. Claude is the model that drives the writeback agent. -- **A supported git host** — each project that should support writeback must be connected to a **GitHub** or **GitLab** repository (Bitbucket, Azure DevOps, dbt Cloud, and local dbt projects are not supported), with the matching Lightdash app installed: - - **GitHub** — install the Lightdash GitHub App on the repository. - - **GitLab** — connect the Lightdash GitLab App for your organization. Writeback works against GitLab.com; self-hosted GitLab domains aren't reachable from the writeback sandbox yet. +- **Repository access** — configure a supported provider and semantic layer format using the [AI writeback prerequisites](/agents/ai-writeback#prerequisites). ## Configuration @@ -29,7 +27,7 @@ Add the following environment variables to your Lightdash deployment: | `E2B_AI_WRITEBACK_TEMPLATE_NAME` | `lightdash/lightdash-ai-writeback` | Lightdash's public E2B template for writeback — pulls our prebuilt sandbox image so you don't have to build one yourself. The `lightdash/` prefix points at our public team template; without it E2B looks in your own team and won't find the image. | | `ANTHROPIC_API_KEY` | `` | Your [Anthropic](https://console.anthropic.com/) API key. Claude is the model that drives the writeback agent. | -Restart the backend. Users with AI Analyst access on a project connected to a supported git host will see the agent offer to open a pull request when they ask for a change that belongs in the dbt repo. +Restart the backend. Users with AI Analyst access on a project connected to a supported git host will see the agent offer to open a pull request when they ask for a change that belongs in the semantic layer repo. ### Optional configuration @@ -39,6 +37,8 @@ Restart the backend. Users with AI Analyst access on a project connected to a su ## Supported dbt versions +This section applies to dbt projects. Native Lightdash YAML uses [native validation](/semantic-layer/yaml#ai-writeback-and-issues) and needs no dbt installation. + The writeback sandbox runs `lightdash compile` using the dbt version configured on the project, so validation behaves the same way it would locally. The sandbox ships with dbt **1.8, 1.9, 1.10, 1.11, and 1.12** preinstalled: - Projects pinned to one of those versions compile against that exact version. @@ -58,15 +58,16 @@ Both providers expose usage dashboards. We recommend setting spend limits on bot ## Permissions -AI writeback is gated by the user's project role. Anyone asking the agent to open a pull request needs at least [project Developer](/workspace-admin/roles) permissions on the project the agent is connected to. The git commits themselves are made by the installed Lightdash app, not by the end user: +AI writeback is gated by the user's project role. Anyone asking the agent to open a pull request needs at least [project Developer](/workspace-admin/roles) permissions on the project the agent is connected to. Git authentication depends on the provider: - **GitHub** commits are signed by the Lightdash GitHub App. - **GitLab** commits are pushed over OAuth and are unsigned. +- **Bitbucket Cloud** uses the [project API token owner's identity](/get-started/quickstart/connect-project#bitbucket-cloud-api-token). ## Troubleshooting **The agent never offers to open a pull request.** -Check that `LIGHTDASH_LICENSE_KEY` is set and the project's dbt connection is GitHub or GitLab with the matching Lightdash app installed. +Check that `LIGHTDASH_LICENSE_KEY` is set and the project meets the [repository and permission prerequisites](/agents/ai-writeback#prerequisites). **Writebacks fail immediately with a sandbox creation error.** Check your [sandbox provider's](/self-host/customize-deployment/sandboxes) credentials, and verify `E2B_AI_WRITEBACK_TEMPLATE_NAME` is set to `lightdash/lightdash-ai-writeback` (the `lightdash/` prefix is required to pull our public template — a bare name resolves to your own team). If you've overridden `E2B_AI_WRITEBACK_TEMPLATE_TAG`, double-check the tag exists. diff --git a/semantic-layer/yaml.mdx b/semantic-layer/yaml.mdx index 5695f9bb..1bf1a0be 100644 --- a/semantic-layer/yaml.mdx +++ b/semantic-layer/yaml.mdx @@ -1,7 +1,7 @@ --- title: "Lightdash YAML" sidebarTitle: Lightdash YAML -description: "Define models in YAML, connect them through GitHub, and update them without dbt" +description: "Define models in YAML, connect them through GitHub or Bitbucket Cloud, and update them without dbt" --- @@ -12,9 +12,9 @@ description: "Define models in YAML, connect them through GitHub, and update the Lightdash YAML defines your semantic layer in standalone model files that point to existing tables in your warehouse. You can build metrics, dimensions, charts, dashboards, and AI agents without installing dbt or maintaining a dbt project. -You can [connect the project through GitHub](#connect-through-github) so Lightdash compiles and refreshes it from your repository, or [deploy local files with the CLI](#step-4-deploy-your-project). A GitHub connection also lets you write custom fields and AI-proposed changes back to the repository as pull requests. +You can [connect the project through GitHub or Bitbucket Cloud](#connect-through-github) so Lightdash compiles and refreshes it from your repository, or [deploy local files with the CLI](#step-4-deploy-your-project). A GitHub connection also lets you write custom fields and AI-proposed changes back to the repository as pull requests. -This page covers the [project layout](#project-layout), [model setup](#define-your-models), [GitHub connection](#connect-through-github), and [write-back support](#write-back-from-lightdash). +This page covers the [project layout](#project-layout), [model setup](#define-your-models), [repository connection](#connect-through-github), and [write-back support](#write-back-from-lightdash). ## Choose a semantic layer format @@ -162,7 +162,7 @@ The CLI uses `warehouse.type` in `lightdash.config.yml` to compile SQL for your ### Step 4: Deploy your project -Choose [GitHub](#connect-through-github) to have Lightdash compile the committed files, or create a project from your local files with the CLI: +Choose [GitHub or Bitbucket Cloud](#connect-through-github) to have Lightdash compile the committed files, or create a project from your local files with the CLI: ```bash lightdash deploy --create --no-warehouse-credentials @@ -188,30 +188,30 @@ After your initial deployment, you can edit your `.yml` files and redeploy chang lightdash deploy --no-warehouse-credentials ``` -For a GitHub-connected project, commit and merge the model changes, then [refresh from the repository](#refresh-a-github-connected-project). +For a GitHub- or Bitbucket Cloud-connected project, commit and merge the model changes, then [refresh from the repository](#refresh-a-github-connected-project). -## Connect through GitHub +## Connect through GitHub or Bitbucket Cloud -You need permission to manage the project's connection and access to the GitHub repository. Use the [GitHub connection settings](/get-started/quickstart/connect-project#github) for authentication options. Install the Lightdash GitHub App on the repository to use write-back. +You need permission to manage the project's connection and access to the repository. Use the [GitHub connection settings](/get-started/quickstart/connect-project#github) or [Bitbucket Cloud connection settings](/get-started/quickstart/connect-project#bitbucket) for authentication. To use AI writeback, install the Lightdash GitHub App or configure the Bitbucket API token with writeback permissions. -1. Commit your configuration and models to GitHub. +1. Commit your configuration and models to GitHub or Bitbucket Cloud. 2. Open **Settings → All projects → Create new**, or **Settings → Current project → Connection settings** for an existing project. -3. Configure the warehouse connection, and choose **GitHub** as the project connection type. +3. Configure the warehouse connection, and choose **GitHub** or **Bitbucket** as the project connection type. 4. Set **Semantic layer format** to **Native Lightdash YAML**. 5. Select the repository and branch. Set **Project directory path** to `/` for the repository root, or the directory containing `lightdash.config.yml` and the model directory. For example, use `/analytics` for `analytics/lightdash.config.yml` and `analytics/models/`. 6. Save the connection. Lightdash reads the committed files and compiles them using the project's saved warehouse connection. You can add this connection to an existing CLI-created project. Updating that project's connection keeps its project ID, warehouse credentials, charts, and dashboards. Keep model and field names stable so saved content still resolves. -Native connections do not use dbt versions, profiles, targets, selectors, or dbt environment variables. GitHub is the supported Git provider for native YAML; native projects cannot be combined with additional dbt sources. +Native connections do not use dbt versions, profiles, targets, selectors, or dbt environment variables. Native YAML supports GitHub and Bitbucket Cloud (`bitbucket.org`); Bitbucket Server/Data Center and GitLab are not supported. Native projects cannot be combined with additional dbt sources. -### Refresh a GitHub-connected project +### Refresh a Git-connected project After merging changes to the configured branch, click **Refresh models** in Lightdash or run [`lightdash refresh`](/workflow/cli/reference#lightdash-refresh). The server pulls that branch and compiles the YAML using the saved warehouse connection. You do not need dbt or a local checkout to trigger a refresh. Invalid YAML, invalid model definitions, duplicate model names, or semantic compilation errors fail the native refresh instead of publishing an incomplete model set. Open [Compilation history](/workspace-admin/project-compilation-history) to inspect the result. -To test another branch first, [create a preview project](/workflow/preview-projects#lightdash-app). The preview retains the native semantic layer format. +To test another branch with a GitHub connection, [create a preview project](/workflow/preview-projects#lightdash-app). The preview retains the native semantic layer format. ## Write back from Lightdash @@ -242,9 +242,9 @@ SQL Runner's **Write back to dbt** action creates dbt SQL models and is not avai ## AI writeback and Issues -[AI writeback](/agents/ai-writeback) can edit native YAML from a chat request. It validates the edited native models before pushing changes, using the connected warehouse's SQL dialect without passing warehouse credentials to the sandbox. A validation error prevents the pull request from being opened or updated. +[AI writeback](/agents/ai-writeback) can edit native YAML from a chat request in GitHub- and Bitbucket Cloud-connected projects. It validates the edited native models before pushing changes, using the connected warehouse's SQL dialect without passing warehouse credentials to the sandbox. A validation error prevents the pull request from being opened or updated. -[Issues](/agents/issues#fixing-an-issue-with-writeback) uses the same native path for semantic layer fixes. Project context fixes update [agent context](/agents/agent-context) in `lightdash.project_context.yml` beside the project configuration. GitHub AI writeback also supports [automatic preview projects](/agents/ai-writeback#automatic-preview-project). +[Issues](/agents/issues#fixing-an-issue-with-writeback) uses the same native path for semantic layer fixes. Project context fixes update [agent context](/agents/agent-context) in `lightdash.project_context.yml` beside the project configuration. GitHub AI writeback also supports [automatic preview projects](/agents/ai-writeback#automatic-preview-project). For Bitbucket, review and merge in Bitbucket, then refresh the project; see [Bitbucket Cloud behavior](/agents/ai-writeback#bitbucket-cloud-behavior). ## Developing with AI coding agents