-
Notifications
You must be signed in to change notification settings - Fork 232
docs: update pipedream docs #2945
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
35fa03e
5ccf318
4f3b402
36940b1
ab939a2
d21fb2c
72b40b2
680cbde
50c0457
946dce4
d60a869
f7073fb
7b5dfb1
56e1a2f
fbf25f5
922d862
035ef64
ee9435e
86be24a
91aeb62
9165657
18f1ed1
5126b09
3db1c7b
8e0497b
f8ffb9c
fadb4ae
f308b07
16a904a
9d0a237
c9f1f36
8f70a06
ff3e960
1516899
8b09fe2
b429053
85e18d3
2ed2af7
2b5419d
028e958
e012ba9
53f8de5
25f2a6a
b1b0c1b
23041ae
b396347
0efaca9
f6efc64
e6ed1e5
4e321d8
fceab0d
4014678
1d56490
9ae76cc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,6 +1,6 @@ | ||||||
| --- | ||||||
| title: Pipedream integration | ||||||
| description: Connect Apify with Pipedream to automate workflows by running Actors, managing storage, and reacting to Actor or task run events. | ||||||
| description: Connect Apify with Pipedream to automate workflows by running Actors, scraping single pages, managing storage, and reacting to Actor or task run events. | ||||||
| sidebar_label: Pipedream | ||||||
| slug: /integrations/pipedream | ||||||
| --- | ||||||
|
|
@@ -23,9 +23,9 @@ Before you begin, make sure you have: | |||||
| 1. Log into your Pipedream account and [create a new workflow](https://pipedream.com/docs/workflows). | ||||||
| 1. [Add an Apify step](https://pipedream.com/docs/workflows/building-workflows/steps) (trigger or action) to your workflow. Select one of two Apify apps: | ||||||
|
|
||||||
|  | ||||||
|  | ||||||
|
|
||||||
| - **Apify** - Authenticate with your Apify API token. Find it in [Apify Console](https://console.apify.com/settings/integrations) under **Settings > Integrations**. | ||||||
| - **Apify** - Authenticate with your Apify API token. Find it in [Apify Console](https://console.apify.com/settings/integrations) under **Settings > API & Integrations**. | ||||||
| - **Apify (OAuth)** - Authorize access to your Apify account via OAuth. | ||||||
| 1. Follow the prompts to authenticate your account. | ||||||
|
|
||||||
|
|
@@ -38,10 +38,10 @@ Before you begin, make sure you have: | |||||
|
|
||||||
| 1. [Create a new workflow](https://pipedream.com/docs/workflows) in Pipedream. | ||||||
| 1. Select **Add Trigger** and search for **Apify**. | ||||||
| 1. Select the trigger you want to use, e.g. **New Finished Actor Run**. | ||||||
| 1. Configure the trigger by selecting the Actor or task to monitor. | ||||||
| 1. Select the trigger you want to use, e.g. **New finished Actor run (instant)**. | ||||||
| 1. Configure the trigger by selecting the Actor or task to monitor. Leave **Trigger on run states** empty to fire on every terminal state, or pick the states you care about. | ||||||
|
|
||||||
|  | ||||||
|  | ||||||
| 1. Add subsequent steps to process the output. | ||||||
|
|
||||||
| ## Use Apify as an action | ||||||
|
|
@@ -52,29 +52,102 @@ Before you begin, make sure you have: | |||||
| 1. Click **+** to add a step and search for **Apify**. | ||||||
| 1. Select the action you want to use, e.g. **Run Actor**. | ||||||
| 1. Configure the action parameters: | ||||||
| - Select the Actor from Apify Store or your recently used Actors | ||||||
| - Provide the Actor input as JSON | ||||||
| - Set optional parameters such as timeout, memory, and build tag | ||||||
|
|
||||||
|  | ||||||
| 1. Add another Apify step with **Get Dataset Items** to retrieve the Actor's output. | ||||||
| - Set **Search Actors from** to **Apify Store Actors** or **Recently used Actors**, then pick the Actor. | ||||||
| - Fill in the Actor's input fields, which are generated from the Actor's input schema. Actors without an input schema get a single **Properties** field that accepts raw JSON. | ||||||
| - Leave **Wait for Finish** set to `true` (the default) to wait for the run, or set it to `false` to return run details immediately. | ||||||
| - With `true`, the step returns the key-value store record named by **Output Record Key** (`OUTPUT` by default). It does not return dataset items. | ||||||
| - Set optional fields as needed: **Build** (a build tag or build number), **Timeout (seconds)**, **Memory (MB)** (powers of two from 128 MB to 32 GB), **Max Items**, **Max Total Charge USD**, and **Webhook URL**. | ||||||
|
|
||||||
|  | ||||||
| 1. Add a **Get dataset items** step to fetch the results. Most Actors write scraped rows to a dataset, not to `OUTPUT`. | ||||||
| 1. Add any subsequent steps to process or store the data. | ||||||
|
|
||||||
| :::caution Building workflows with AI | ||||||
|
|
||||||
| Pipedream's [Apify app page](https://pipedream.com/apps/apify) can generate a workflow for you with Pipedream's AI builder. Because the Apify connector is schema-driven, the AI builder can consume your Pipedream AI tokens quickly and might not configure required inputs reliably. For predictable results, add the Apify triggers and actions to your workflow directly, as described above. | ||||||
|
|
||||||
| ::: | ||||||
|
|
||||||
| ## Handle long-running Actor runs | ||||||
|
|
||||||
| Some Actor runs can outlast a single Pipedream step. | ||||||
|
|
||||||
| - **Run task**: waits asynchronously (webhook + 30-second polling fallback), up to one day. | ||||||
| - **Run Actor**: waits inside the step and can hit step timeout. | ||||||
|
|
||||||
| For longer runs with **Run Actor**, split across two workflows: | ||||||
|
|
||||||
| 1. In the first workflow, add **Run Actor** and set **Wait for Finish** to `false`. The step returns the run details immediately. | ||||||
| 1. In a second workflow, use the **New finished Actor run (instant)** trigger for the same Actor. | ||||||
| 1. Add **Get dataset items** after the trigger. | ||||||
|
|
||||||
| Alternatively, save the Actor configuration as an [Actor task](/actors/running/tasks) and use **Run task** with **Wait for Finish** set to `true`, then add **Get dataset items** to fetch the results. | ||||||
|
|
||||||
| ## Handle large Actor output | ||||||
|
|
||||||
| When **Run Actor** waits for a run, it returns the record named by **Output Record Key** (`OUTPUT` by default). If the record is over 256 KB, the step returns a reference instead of inline data. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where does the 256 KB threshold come from, Pipedream's step output or Apify's key-value store? Readers need to know whose limit it is to know what to change, and naming the reference object's fields here grounds the
Suggested change
|
||||||
|
|
||||||
| To read large output: | ||||||
|
|
||||||
| - Fetch `recordUrl` in a later step. | ||||||
| - Use **Get key-value store record** with `keyValueStoreId` and `recordKey`. | ||||||
|
|
||||||
| If the Actor writes to a dataset, use **Get dataset items**. | ||||||
|
|
||||||
| ## Triggers | ||||||
|
|
||||||
| - **New finished Actor run (instant)** - Triggers when a selected Actor run finishes. | ||||||
| - **New finished task run (instant)** - Triggers when a selected task run finishes. | ||||||
| Both triggers register an Apify webhook when you deploy the workflow and remove it when you disable the workflow. | ||||||
|
|
||||||
| - **New finished Actor run (instant)** - Emits an event when a run of the selected Actor finishes. | ||||||
| - **New finished task run (instant)** - Emits an event when a run of the selected task finishes. | ||||||
|
|
||||||
| Each trigger takes a **Trigger on run states** field listing the terminal run states: **Succeeded**, **Failed**, **Timed out**, and **Aborted**. Leave it empty to fire on all four, which is the default. | ||||||
|
|
||||||
| ## Actions | ||||||
|
|
||||||
| - **Run Actor** - Runs a selected Actor with customizable input and configuration. | ||||||
| - **Run task** - Runs a selected Actor task and optionally waits for it to finish. | ||||||
| - **Run task synchronously** - Runs a selected task and returns its dataset items when it finishes. | ||||||
| - **Scrape single URL** - Runs a scraper on a specified URL and returns its content as HTML. Use this for extracting content from a single page, e.g. in LLM workflows. | ||||||
| - **Get dataset items** - Retrieves items from a [dataset](/storage/dataset). | ||||||
| - **Get key-value store record** - Retrieves a record from a [key-value store](/storage/key-value-store). | ||||||
| - **Run Actor** - Runs a selected Actor and, by default, waits for it to finish and returns the `OUTPUT` record. Input fields are generated from the Actor's input schema. Optional fields: **Build**, **Timeout (seconds)**, **Memory (MB)**, **Max Items**, **Max Total Charge USD**, **Webhook URL**, and **Output Record Key**. | ||||||
| - **Run task** - Runs a selected task and, by default, waits for it to finish and returns the run, without its output. Use **Override Input (JSON)** to replace the task's saved input for a single run, and leave it empty to use the saved input. Optional fields: **Build**, **Timeout (seconds)**, and **Memory (MB)**. | ||||||
| - **Scrape single URL** - Runs [Website Content Crawler](https://apify.com/apify/website-content-crawler) on a specified URL and returns its content as HTML. Use this for extracting content from a single page, e.g. in LLM workflows. **Crawler Type** selects the engine: **Firefox (Headless Browser)** renders JavaScript (the default), **Cheerio (Raw HTTP)** sends raw HTTP requests and renders none, and **Adaptive** switches per page. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you confirm the engine list? On master the
Suggested change
|
||||||
| - **Get dataset items** - Retrieves items from a [dataset](/storage/dataset), specified by ID or name. **Limit** and **Offset** page through the items, and **Fields**, **Omit**, **Flatten**, and **Clean** shape each item. | ||||||
| - **Get key-value store record** - Retrieves a record from a [key-value store](/storage/key-value-store). A JSON record is returned as parsed fields, and any other content type as a file reference. | ||||||
| - **Set key-value store record** - Creates or updates a record in a [key-value store](/storage/key-value-store). | ||||||
|
|
||||||
| ## Use Apify with AI agents (MCP) | ||||||
|
|
||||||
| To use Apify Actors directly with AI agents and MCP-compatible clients, use the [Apify MCP server](/integrations/mcp). | ||||||
|
|
||||||
| Pipedream also hosts an [MCP server for Apify](https://mcp.pipedream.com/app/apify). Because Apify provides its own MCP server, use the Pipedream one only if you want Apify available alongside your other Pipedream apps in a single MCP client. | ||||||
|
|
||||||
| ## Troubleshooting | ||||||
|
|
||||||
| ### The workflow times out before the Actor finishes | ||||||
|
|
||||||
| **Run Actor** waits for the run inside the step, so a long run can exceed Pipedream's step limit. Set **Wait for Finish** to `false` and pick the results up from a second workflow, or use **Run task**, which waits asynchronously. See [Handle long-running Actor runs](#handle-long-running-actor-runs). | ||||||
|
|
||||||
| ### The Actor dropdown is empty | ||||||
|
|
||||||
| **Search Actors from** defaults to **Recently used Actors**, which is empty on a new Apify account. Switch it to **Apify Store Actors** to browse [Apify Store](https://apify.com/store), or enter an Actor ID or an `owner~actor-name` identifier directly. | ||||||
|
|
||||||
| ### The Actor has no builds yet | ||||||
|
|
||||||
| An Actor cannot run until it has been built. Open the Actor in [Apify Console](https://console.apify.com/actors) and build it, or run `apify push` with the [Apify CLI](/cli), then run the step again. | ||||||
|
|
||||||
| ### Apify can't find the build you entered | ||||||
|
|
||||||
| The **Build** dropdown lists tagged builds only, because some Actors have thousands of builds. To use an untagged build, type its build number, such as `0.1.2`, into the field instead of picking from the list. | ||||||
|
|
||||||
| ### Apify rejects the memory value | ||||||
|
|
||||||
| **Memory (MB)** must be a power of two between 128 MB and 32 GB, and **Run Actor** narrows the list further to the range the Actor's build declares. Pick a value from the dropdown rather than typing one. | ||||||
|
|
||||||
| ### The step reports invalid JSON input | ||||||
|
|
||||||
| Actors without an input schema take a single **Properties** field, and **Run task** takes **Override Input (JSON)**. Both must contain a valid JSON object. Use `{}` to run with the saved or default input, and copy the input from the Actor's **Input** tab in Apify Console if you are unsure of the shape. | ||||||
|
|
||||||
| ### The trigger won't deploy | ||||||
|
|
||||||
| Both triggers need a target before they can register their Apify webhook. Select an Actor for **New finished Actor run (instant)**, or a task for **New finished task run (instant)**, then deploy again. | ||||||
|
|
||||||
| ## Resources | ||||||
|
|
||||||
| - [Apify integration page on Pipedream](https://pipedream.com/apps/apify) | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be better to name the terminal states here? They're only listed under Triggers further down, so at this point the reader can't tell what they'd be opting out of.