diff --git a/apify-api/openapi/openapi.yaml b/apify-api/openapi/openapi.yaml index 149e19e4fc..cc92e754e2 100644 --- a/apify-api/openapi/openapi.yaml +++ b/apify-api/openapi/openapi.yaml @@ -433,25 +433,45 @@ info: The default rate limit is applied to every API endpoint except a few select ones, which have higher rate limits. Each API endpoint returns its rate limit in `X-RateLimit-Limit` header. + These endpoints have a rate limit of _35 requests per second per resource_: + + * Batch ([add](#/reference/request-queues/batch-request-operations/add-requests), + [delete](#/reference/request-queues/batch-request-operations/delete-requests)) + operations on requests in request queues + * [List and lock head](#/reference/request-queues/queue-head-with-locks/list-and-lock-head) of a request queue + + These endpoints have a rate limit of _100 requests per second per resource_: + + * Download records from a key-value store as a ZIP archive + These endpoints have a rate limit of _200 requests per second per resource_: * CRUD ([get](#/reference/key-value-stores/record/get-record), [put](#/reference/key-value-stores/record/put-record), [delete](#/reference/key-value-stores/record/delete-record)) operations on key-value store records + * [Get list of keys](#/reference/key-value-stores/key-collection/get-list-of-keys) in a key-value store + + These endpoints have a rate limit of _350 requests per second per resource_: - These endpoints have a rate limit of _400 requests per second per resource_: - * [Run Actor](#/reference/actors/run-collection/run-actor) - * [Run Actor task asynchronously](#/reference/actor-tasks/runs-collection/run-task-asynchronously) - * [Run Actor task synchronously](#/reference/actor-tasks/runs-collection/run-task-synchronously) - * [Metamorph Actor run](#/reference/actors/metamorph-run/metamorph-run) - * [Push items](#/reference/datasets/item-collection/put-items) to dataset * CRUD ([add](#/reference/request-queues/request-collection/add-request), [get](#/reference/request-queues/request-collection/get-request), [update](#/reference/request-queues/request-collection/update-request), [delete](#/reference/request-queues/request-collection/delete-request)) operations on requests in request queues + * Request lock operations + ([prolong](#/reference/request-queues/request-lock/prolong-request-lock), + [delete](#/reference/request-queues/request-lock/delete-request-lock)) + in request queues + + These endpoints have a rate limit of _400 requests per second per resource_: + + * [Run Actor](#/reference/actors/run-collection/run-actor) + * [Run Actor task asynchronously](#/reference/actor-tasks/runs-collection/run-task-asynchronously) + * [Run Actor task synchronously](#/reference/actor-tasks/runs-collection/run-task-synchronously) + * [Metamorph Actor run](#/reference/actors/metamorph-run/metamorph-run) + * [Push items](#/reference/datasets/item-collection/put-items) to dataset ### Rate limit exceeded errors diff --git a/sources/platform/storage/dataset/index.md b/sources/platform/storage/dataset/index.md index 6d21dbecb9..4aa3f5afea 100644 --- a/sources/platform/storage/dataset/index.md +++ b/sources/platform/storage/dataset/index.md @@ -6,16 +6,13 @@ toc_max_heading_level: 4 slug: /storage/dataset --- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - Dataset storage enables you to sequentially save and retrieve data. A unique dataset is automatically created and assigned to each Actor run when the first item is stored. -Typically, datasets comprise results from web scraping, crawling, and data processing jobs. You can visualize this data in a table, where each object is forming a row and its attributes are represented as columns. You have the option to export data in various formats, including JSON, CSV, XML, Excel, HTML Table, RSS or JSONL. +Typically, datasets hold results from web scraping, crawling, and data processing jobs. The data displays as a table, where each object forms a row and its attributes form columns. You can export the data in JSON, CSV, XML, Excel, HTML table, RSS, or JSONL formats. :::info Retention period -Named datasets are retained indefinitely. Unnamed datasets expire after 7 days unless otherwise specified. [Learn more](/storage#named-and-unnamed-storages) +Named datasets are retained indefinitely. Unnamed datasets expire after 7 days unless otherwise specified. [Learn more](/storage#data-retention) ::: @@ -36,7 +33,7 @@ Dataset schemas are defined in the Actor's `.actor` folder and apply to the Acto You can access your datasets in several ways: -- [Apify Console](https://console.apify.com) - provides an easy-to-understand interface. +- [Apify Console](https://console.apify.com) - view and manage your datasets in a visual interface. - [Apify API](/api/v2) - to access your datasets programmatically. - [Apify API client](/api) - to access your datasets from any Node.js/Python application. - [Apify SDKs](/sdk) - when building your own JavaScript/Python Actor. @@ -45,7 +42,7 @@ You can access your datasets in several ways: In [Apify Console](https://console.apify.com), you can view your datasets in the [Storage](https://console.apify.com/storage) section under the [Datasets](https://console.apify.com/storage?tab=datasets) tab. -![Datasets in app](../images/datasets-app.png) +![Datasets in Apify Console](../images/storage-types.svg) To view or download a dataset: @@ -53,13 +50,13 @@ To view or download a dataset: 1. Browse the data in **Table** or **JSON** view. 1. Click **Export** to download the data in your preferred format. -Utilize the **Actions** menu to modify the dataset's name, which also affects its [retention period](/storage#data-retention), and to adjust [access rights](/account/collaboration). The **API** button allows you to explore and test the dataset's [API endpoints](/api/v2/storage-datasets). +Use the **Actions** menu to rename the dataset (which affects its [retention period](/storage#data-retention)) and to adjust [access rights](/account/collaboration). Select **API** to view and test the dataset's [API endpoints](/api/v2/storage-datasets). ![Datasets detail view](../images/datasets-detail.png) ### Apify API -The [Apify API](/api/v2/storage-datasets) enables you programmatic access to your datasets using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). +The [Apify API](/api/v2/storage-datasets) gives you programmatic access to your datasets using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). If you are accessing your datasets using the `username~store-name` [store ID format](../index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) tab of **Settings** page of your Apify account. @@ -115,11 +112,7 @@ To add data to a dataset, issue a POST request to the [Put items](/api/v2/datase https://api.apify.com/v2/datasets/{DATASET_ID}/items ``` -:::caution Rate limit - -API data push to a dataset is capped at _400 requests per second_ to avoid overloading the servers. - -::: +Pushes to a dataset are rate-limited; see [Rate limiting](#rate-limiting). Example payload: @@ -143,9 +136,9 @@ For further details and a breakdown of each storage API endpoint, refer to the [ #### JavaScript API client -The [JavaScript API client](/api/client/js/reference/class/DatasetClient) (`apify-client`) enables you access to your datasets from any Node.js application, whether hosted on the Apify platform or externally. +The [JavaScript API client](/api/client/js/reference/class/DatasetClient) (`apify-client`) enables you to access your datasets from any Node.js application, whether hosted on the Apify platform or externally. -After importing and initiating the client, you can save each dataset to a variable for easier access. +After importing and initializing the client, you can save each dataset to a variable for easier access. ```js const myDatasetClient = apifyClient.dataset('jane-doe/my-dataset'); @@ -163,9 +156,9 @@ Check out the [JavaScript API client documentation](/api/client/js/reference/cla #### Python API client -The [Python API client](/api/client/python/reference/class/DatasetClient) (`apify-client`) enables you access to your datasets from any Python application, whether it is running on the Apify platform or externally. +The [Python API client](/api/client/python/reference/class/DatasetClient) (`apify-client`) allows you to access your datasets from any Python application, whether it is running on the Apify platform or externally. -After importing and initiating the client, you can save each dataset to a variable for easier access. +After importing and initializing the client, you can save each dataset to a variable for easier access. ```python my_dataset_client = apify_client.dataset('jane-doe/my-dataset') @@ -185,7 +178,7 @@ Check out the [Python API client documentation](/api/client/python/reference/cla #### JavaScript SDK -When working with a JavaScript [Actor](../../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/result-storage#dataset) is an essential tool, especially for dataset management. It simplifies the tasks of storing and retrieving data, seamlessly integrating with the Actor's workflow. Key features of the SDK include the ability to append data, retrieve what is stored, and manage dataset properties effectively. Central to this functionality is the [`Dataset`](/sdk/js/reference/class/Dataset) class. This class allows you to determine where your data is stored - locally or in the Apify cloud. To add data to your chosen datasets, use the [`pushData()`](/sdk/js/reference/class/Dataset#pushData) method. +In JavaScript [Actors](../../actors/index.mdx), manage datasets with the JavaScript SDK's [`Dataset`](/sdk/js/reference/class/Dataset) class. It works both locally and on the Apify platform. To add data, use the [`pushData()`](/sdk/js/reference/class/Dataset#pushData) method. Additionally the SDK offers other methods like [`getData()`](/sdk/js/reference/class/Dataset#getData), [`map()`](/sdk/js/reference/class/Dataset#map), and [`reduce()`](/sdk/js/reference/class/Dataset#reduce). For practical applications of these methods, refer to the [example](/sdk/js/docs/examples/map-and-reduce) section. @@ -240,7 +233,7 @@ await dataset.pushData({ foo: 'bar' }); await Actor.exit(); ``` -Utilize the `fields` option in the [`getData()`](/sdk/js/reference/class/Dataset#getData) method to specify which data fields to retrieve. This option accepts an array of fields names (string) to include in your results. +Use the `fields` option in the [`getData()`](/sdk/js/reference/class/Dataset#getData) method to specify which data fields to retrieve. This option accepts an array of field names (string) to include in your results. ```js import { Actor } from 'apify'; @@ -263,7 +256,7 @@ Check out the [JavaScript SDK documentation](/sdk/js/docs/guides/result-storage# #### Python SDK -For Python [Actors](../../actors/index.mdx), the [Python SDK](/sdk/python/docs/concepts/storages#working-with-datasets) is essential. The dataset is represented by a [`Dataset`](/sdk/python/reference/class/Dataset) class. You can use this class to specify whether your data is stored locally or in the Apify cloud and push data to the datasets of your choice using the [`push_data()`](/sdk/python/reference/class/Dataset#push_data) method. For further data manipulation you could also use other methods such as [`get_data()`](/sdk/python/reference/class/Dataset#get_data), [`map()`](/sdk/python/reference/class/Dataset#map) and [`reduce()`](/sdk/python/reference/class/Dataset#reduce). +In Python [Actors](../../actors/index.mdx), manage datasets with the Python SDK's [`Dataset`](/sdk/python/reference/class/Dataset) class. It works both locally and on the Apify platform. To add data, use [`push_data()`](/sdk/python/reference/class/Dataset#push_data); to read and transform it, use [`get_data()`](/sdk/python/reference/class/Dataset#get_data), [`map()`](/sdk/python/reference/class/Dataset#map), and [`reduce()`](/sdk/python/reference/class/Dataset#reduce). For datasets stored locally, the data is located at the following path: @@ -301,7 +294,7 @@ async def main(): await dataset.push_data({'foo': 'bar'}) ``` -Utilize the `fields` option in the [`get_data()`](/sdk/python/reference/class/Dataset#get_data) method to specify which data fields to retrieve. This option accepts an array of fields names (string) to include in your results. +Use the `fields` option in the [`get_data()`](/sdk/python/reference/class/Dataset#get_data) method to specify which data fields to retrieve. This option accepts an array of field names (string) to include in your results. ```python from apify import Actor @@ -416,77 +409,17 @@ This feature is also useful when customizing your RSS feeds generated for variou By default, the whole result is wrapped in an `` element, while each page object is contained in an `` element. You can change this using the `xmlRoot` and `xmlRow` URL parameters when retrieving your data with a GET request. -## Share - -You can grant [access rights](/account/collaboration) to your dataset through the **Share** button under the **Actions** menu. For more details, check the [full list of permissions](/account/collaboration/list-of-permissions). - -You can also share datasets by link using their ID or name, depending on your account or resource-level general access setting. Learn how link-based access works in [General resource access](/account/collaboration/general-resource-access). - -For one-off sharing of specific records when access is restricted, you can generate time-limited pre-signed URLs. See [Sharing restricted resources with pre-signed URLs](/account/collaboration/general-resource-access#pre-signed-urls). - -### Share datasets between runs - -You can access a dataset from any [Actor](../../actors/index.mdx) or [task](../../actors/running/tasks.md) run as long as you know its _name_ or _ID_. - -To access a dataset from another run using the [Apify SDK](/sdk), open it using the same method as you would with any other dataset. - - - - -```js -import { Actor } from 'apify'; - -await Actor.init(); - -const otherDataset = await Actor.openDataset('old-dataset'); -// ... - -await Actor.exit(); -``` - - - - -```python -from apify import Actor - -async def main(): - async with Actor: - other_dataset = await Actor.open_dataset(name='old-dataset') - # ... -``` - - - - -In the [JavaScript API client](/api/client/js/reference/class/DatasetClient) as well as in [Python API client](/api/client/python/reference/class/DatasetClient) , you can access a dataset using its client. Once you've opened the dataset, you can read its contents and add new data in the same manner as you would for a dataset from your current run. - - - - -```js -const otherDatasetClient = apifyClient.dataset('jane-doe/old-dataset'); -``` - - - - -```python -other_dataset_client = apify_client.dataset('jane-doe/old-dataset') -``` - - - +## Share and reuse {#share} -The same applies for the [Apify API](#apify-api) - you can use [the same endpoints](#apify-api) as you would normally do. +You can grant access rights to your dataset, share it by link, or generate a time-limited pre-signed URL for specific items. See [Share storage](../share.md). -See the [Storage overview](/storage#share-storages-between-runs) for details on sharing storages between runs. +To read from or write to a dataset that belongs to a different Actor or task run, see [Use storage from another run](../use-from-another-run.md). ## Limits - Data storage formats that use tabulation (like HTML, CSV, and Excel) are limited to a maximum of _2000_ columns. Data exceeding this limit will not be retrieved. -- The `pushData()`method is constrained by the receiving API's size limit. It accepts objects with JSON size under _9MB_. While individual objects within an array must not exceed _9MB_, the overall size has no restriction. +- The `pushData()` method is constrained by the receiving API's size limit. It accepts objects with JSON size under _9MB_. While individual objects within an array must not exceed _9MB_, the overall size has no restriction. - The maximum length for dataset names is 63 characters. @@ -494,6 +427,6 @@ See the [Storage overview](/storage#share-storages-between-runs) for details on The rate limit for pushing data to a dataset through the [API](/api/v2/dataset-items-post) is capped at _400 requests per second_ for each dataset, a measure to prevent overloading Apify servers. -For all other dataset [API endpoints](/api/v2/storage-datasets) , the rate limit is _60 requests per second_ for each dataset. +For all other dataset [API endpoints](/api/v2/storage-datasets), the rate limit is _60 requests per second_ for each dataset. Check out the [API documentation](/api/v2#rate-limiting) for more information and guidance on actions to take if you exceed these rate limits. diff --git a/sources/platform/storage/dataset/multiple_datasets.mdx b/sources/platform/storage/dataset/multiple_datasets.mdx index 7f4327d910..af00d9614b 100644 --- a/sources/platform/storage/dataset/multiple_datasets.mdx +++ b/sources/platform/storage/dataset/multiple_datasets.mdx @@ -1,6 +1,6 @@ --- -title: Multiple datasets -description: Learn how to use multiple datasets within your Actors to organize and store different types of data separately. +title: Use multiple datasets +description: Define more than one dataset in your Actor, write to each from your code, and surface them on the run page so users can find the right output. sidebar_position: 3 slug: /storage/dataset-schema/multiple-datasets --- diff --git a/sources/platform/storage/dataset/validation.md b/sources/platform/storage/dataset/validation.md index ccd3c8675f..da3e080a69 100644 --- a/sources/platform/storage/dataset/validation.md +++ b/sources/platform/storage/dataset/validation.md @@ -1,5 +1,5 @@ --- -title: Dataset validation +title: Validate dataset output description: Specify the dataset schema within your Actors to add monitoring and validation at the field level, ensuring data quality and consistency across runs. sidebar_position: 2 slug: /storage/dataset-schema/validation diff --git a/sources/platform/storage/images/datasets-app.png b/sources/platform/storage/images/datasets-app.png deleted file mode 100644 index 2fe5d36462..0000000000 Binary files a/sources/platform/storage/images/datasets-app.png and /dev/null differ diff --git a/sources/platform/storage/images/find-store-id.png b/sources/platform/storage/images/find-store-id.png deleted file mode 100644 index e9d7c23807..0000000000 Binary files a/sources/platform/storage/images/find-store-id.png and /dev/null differ diff --git a/sources/platform/storage/images/key-value-stores-app.png b/sources/platform/storage/images/key-value-stores-app.png deleted file mode 100644 index 0aad087c3d..0000000000 Binary files a/sources/platform/storage/images/key-value-stores-app.png and /dev/null differ diff --git a/sources/platform/storage/images/overview-api.png b/sources/platform/storage/images/overview-api.png deleted file mode 100644 index ddb87a72ff..0000000000 Binary files a/sources/platform/storage/images/overview-api.png and /dev/null differ diff --git a/sources/platform/storage/images/request-queue-app.png b/sources/platform/storage/images/request-queue-app.png deleted file mode 100644 index fe3f0b836f..0000000000 Binary files a/sources/platform/storage/images/request-queue-app.png and /dev/null differ diff --git a/sources/platform/storage/images/storage-types-kvs.svg b/sources/platform/storage/images/storage-types-kvs.svg new file mode 100644 index 0000000000..3229eeb784 --- /dev/null +++ b/sources/platform/storage/images/storage-types-kvs.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/sources/platform/storage/images/storage-types-rq.svg b/sources/platform/storage/images/storage-types-rq.svg new file mode 100644 index 0000000000..03fb984169 --- /dev/null +++ b/sources/platform/storage/images/storage-types-rq.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/sources/platform/storage/images/storage-types.svg b/sources/platform/storage/images/storage-types.svg new file mode 100644 index 0000000000..d273390612 --- /dev/null +++ b/sources/platform/storage/images/storage-types.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/sources/platform/storage/index.md b/sources/platform/storage/index.md index 5b465fdff2..a1acca9005 100644 --- a/sources/platform/storage/index.md +++ b/sources/platform/storage/index.md @@ -47,11 +47,11 @@ To view your storages in [Apify Console](https://console.apify.com/storage): To view the related API endpoints, select **API** in the top right corner. -![Storages in app](./images/datasets-app.png) +![Storage types in Apify Console](./images/storage-types.svg) :::note Toggle unnamed storages -Use the **Include unnamed storages** checkbox to either display or hide unnamed storages. By default Apify Console displays them. +Use the **Include unnamed storages** checkbox to show or hide unnamed storages. By default, Apify Console shows them. ::: @@ -59,14 +59,6 @@ To rename a store, open the **Actions** menu and select **Rename**. To share a storage, select **Share** in the **Actions** menu and provide an email, username, or user ID. -These URLs link to API _endpoints_ where your data is stored. _Read_ endpoints don't require an [authentication token](/api/v2#authentication). Calls are authenticated by a hard-to-guess ID, which keeps sharing secure. Operations such as _update_ or _delete_ do require the token. - -:::caution Token security - -Never share a URL containing your authentication token. It can compromise your account's security. If the data you want to share requires a token, download it first and share it as a file. - -::: - ### Apify API The [Apify API](/api/v2/storage-key-value-stores) lets you access your storages programmatically using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and share your crawling results. @@ -81,6 +73,12 @@ For read (GET) requests, the alphanumerical ID alone is enough, since it's hard For other request types, and when using `username~store-name`, provide your secret API token in the request's [`Authorization`](/api/v2#authentication) header or as a query parameter. Find your token on the [API & Integrations](https://console.apify.com/settings/integrations) page of your Apify account. +:::caution Token security + +Never share a URL containing your authentication token. It can compromise your account's security. If the data you want to share requires a token, download it first and share it as a file. + +::: + For a breakdown of each storage endpoint, see the [API documentation](/api/v2/storage-datasets). ### Apify API clients @@ -94,13 +92,13 @@ For more details, see the [API client docs](/api). The Apify SDKs are JavaScript and Python libraries for building your own Actors. * JavaScript SDK requires [Node.js](https://nodejs.org/en/) 16 or later. -* Python SDK requires [Python](https://www.python.org/downloads/release/python-380/) 3.8 or above. +* Python SDK requires [Python](https://www.python.org/downloads/) 3.10 or above. ## Named and unnamed storages The default storages for an Actor run are unnamed, identified only by an _ID_. Naming a storage ensures indefinite retention regardless of plan; unnamed storages follow the [data retention](#data-retention) rules below. -Named and unnamed storages are identical except for their retention period. Named storages are easier to identify and confirm. The names `janedoe~my-storage-1` and `janedoe~web-scrape-results` are easier to tell apart than the IDs `cAbcYOfuXemTPwnIB` and `CAbcsuZbp7JHzkw1B`. Storage names can be up to 63 characters long. +Named and unnamed storages are identical except for their retention period. Named storages are easier to identify. The names `janedoe~my-storage-1` and `janedoe~web-scrape-results` are easier to tell apart than the IDs `cAbcYOfuXemTPwnIB` and `CAbcsuZbp7JHzkw1B`. Storage names can be up to 63 characters long. ### Name a storage @@ -109,10 +107,7 @@ You can name a storage via Apify Console or through the API. In Apify Console: 1. Open your run's details and select the **Dataset**, **Key-value store**, or **Request queue** tab as appropriate. -1. Find the store's ID: - - ![Finding your store's ID](./images/find-store-id.png) - +1. Find the store's ID. 1. Click the ID to open the storage details. 1. Click on the **Actions** menu and choose **Rename**. 1. Enter a new name. Your storage is now preserved indefinitely. @@ -140,8 +135,6 @@ Unnamed storages beyond the 10 most recent runs are deleted when the retention p ## Estimate your costs -Use this tool to estimate storage costs by plan and storage type. -
Estimate your storage costs @@ -155,16 +148,13 @@ Use this tool to estimate storage costs by plan and storage type. ## Rate limiting -All API endpoints limit their request rate to protect Apify servers from overload. The default rate limit for storage objects is _60 requests per second_. However, there are exceptions limited to _400 requests per second_ per storage object, including: +All API endpoints limit their request rate to protect Apify servers from overload. The default rate limit for storage objects is _60 requests per second_ per storage object, and some endpoints have higher or lower limits. The tiers differ per storage type: -* [Push items](/api/v2/dataset-items-post) to dataset. -* CRUD ([add](/api/v2/request-queue-requests-post), -[get](/api/v2/request-queue-request-get), -[update](/api/v2/request-queue-request-put), -[delete](/api/v2/request-queue-request-delete)) -operations of _request queue_ requests. +* [Datasets](/storage/dataset#rate-limiting) +* [Key-value stores](/storage/key-value-store#rate-limiting) +* [Request queues](/storage/request-queue#rate-limiting) -If a client exceeds this limit, the API endpoints respond with the HTTP status code `429 Too Many Requests` and the following body: +If a client exceeds its limit, the API endpoints respond with the HTTP status code `429 Too Many Requests` and the following body: ```json { @@ -179,37 +169,11 @@ Go to the [API documentation](/api/v2#rate-limiting) for details and to learn wh ## Share -You can grant [access rights](/account/collaboration) to other Apify users to view or modify your storages. Check the [full list of permissions](/account/collaboration/list-of-permissions). - -You can also share storages by link using their ID or name, depending on your account or resource-level general access setting. Learn how link-based access works in [General resource access](/account/collaboration/general-resource-access). - -For one-off sharing when access is restricted, generate time-limited pre-signed URLs. See [Sharing restricted resources with pre-signed URLs](/account/collaboration/general-resource-access#pre-signed-urls). - -:::tip Accessing restricted storage resources via API - -If your storage resource is set to _restricted_, all API calls must include a valid authentication token in the `Authorization` header. If you're using **apify-client** the header is passed in automatically. - -::: +You can grant access rights to other Apify users, share a storage by link, or generate a time-limited pre-signed URL for one-off access to a restricted resource. See [Share storage](./share.md). ## Concurrent access {#share-storages-between-runs} -Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run, provided you have its _name_ or _ID_. Use the same methods and endpoints you'd use for the current run's storages. - -[Datasets](/storage/dataset) and [key-value stores](/storage/key-value-store) support concurrent use. Multiple Actors or tasks can write to the same dataset or key-value store, and multiple runs can read from them at the same time. - -[Request queues](./request_queue.md), on the other hand, only allow multiple runs to add new data. A request queue can only be processed by one Actor or task run at any one time. - -:::note Concurrent write order - -When multiple runs write to a storage simultaneously, the order of writes is not guaranteed. Data is written as each request is processed. The same applies in key-value stores and request queues: if a delete request precedes a read request for the same record, the read request fails. - -::: - -:::info Accessing restricted storage resources between runs - -If a storage resource access is set to **Restricted**, the run from which it's accessed must have explicit access to it. Learn how restricted access works in [General resource access](/account/collaboration/general-resource-access). - -::: +If you have the storage's _name_ or _ID_, you can access it from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run. Datasets and key-value stores support concurrent reads and writes, while a request queue accepts new data from multiple runs but can only be processed by one run at a time, unless you use [request locking](./request_queue.md#distributivity). See [Use storage from another run](./use-from-another-run.md). ## Delete storages diff --git a/sources/platform/storage/key_value_store/index.md b/sources/platform/storage/key_value_store/index.md index 783dbe5a43..62431f9932 100644 --- a/sources/platform/storage/key_value_store/index.md +++ b/sources/platform/storage/key_value_store/index.md @@ -6,9 +6,6 @@ sidebar_position: 9.3 slug: /storage/key-value-store --- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - The key-value store is simple storage that can be used for storing any kind of data. It can be JSON or HTML documents, zip files, images, or strings. The data are stored along with their [MIME content type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types). Each Actor run is assigned its own key-value store when it is created. The store contains the Actor's input, and, if necessary, other data such as its output. @@ -17,7 +14,7 @@ Key-value stores are mutable - you can both add entries and delete them. :::info Retention period -Named key-value stores are retained indefinitely. Unnamed key-value stores expire after 7 days unless otherwise specified. [Learn more](/storage#named-and-unnamed-storages) +Named key-value stores are retained indefinitely. Unnamed key-value stores expire after 7 days unless otherwise specified. [Learn more](/storage#data-retention) ::: @@ -25,9 +22,9 @@ Named key-value stores are retained indefinitely. Unnamed key-value stores expir ## Basic usage -You can access key-value stores through several methods +You can access your key-value stores in several ways: -- [Apify Console](https://console.apify.com) - provides an easy-to-understand interface. +- [Apify Console](https://console.apify.com) - view and manage your key-value stores in a visual interface. - [Apify API](/api/v2) - for accessing your key-value stores programmatically. - [Apify API clients](/api) - to access your key-value stores from any Node.js/Python application. - [Apify SDKs](/sdk) - when building your own JavaScript/Python Actor. @@ -36,9 +33,9 @@ You can access key-value stores through several methods In [Apify Console](https://console.apify.com), you can view your key-value stores in the [Storage](https://console.apify.com/storage) section under the [Key-value stores](https://console.apify.com/storage?tab=keyValueStores) tab. -![Key-value stores in app](../images/key-value-stores-app.png) +![Key-value stores in Apify Console](../images/storage-types-kvs.svg) -To view a key-value store's content, click on its **Store ID**. Under the **Actions** menu, you can rename your store (which extends its [retention period](/storage#named-and-unnamed-storages)) and grant [access rights](/account/collaboration) using the **Share** button. +To view a key-value store's content, click on its **Store ID**. Under the **Actions** menu, you can rename your store (which affects its [retention period](/storage#named-and-unnamed-storages)) and grant [access rights](/account/collaboration) using the **Share** button. Click on the **API** button to view and test a store's [API endpoints](/api/v2/storage-key-value-stores). ![Key-value stores detail](../images/key-value-stores-detail-header.png) @@ -55,7 +52,7 @@ At the bottom of the page, you can work with records in your key-value store: ### Apify API -The [Apify API](/api/v2/storage-key-value-stores) enables you programmatic access to your key-value stores using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). +The [Apify API](/api/v2/storage-key-value-stores) gives you programmatic access to your key-value stores using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). If you are accessing your key-value stores using the `username~store-name` [store ID format](../index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) tab of **Settings** page of your Apify account. @@ -112,7 +109,7 @@ For further details and a breakdown of each storage API endpoint, refer to the [ The Apify [JavaScript API client](/api/client/js/reference/class/KeyValueStoreClient) (`apify-client`) enables you to access your key-value stores from any Node.js application, whether hosted on the Apify platform or externally. -After importing and initiating the client, you can save each key-value store to a variable for easier access. +After importing and initializing the client, you can save each key-value store to a variable for easier access. ```js const myKeyValStoreClient = apifyClient.keyValueStore( @@ -128,7 +125,7 @@ Check out the [JavaScript API client documentation](/api/client/js/reference/cla The Apify [Python API client](/api/client/python/reference/class/KeyValueStoreClient) (`apify-client`) allows you to access your key-value stores from any Python application, whether it is running on the Apify platform or externally. -After importing and initiating the client, you can save each key-value store to a variable for easier access. +After importing and initializing the client, you can save each key-value store to a variable for easier access. ```python my_key_val_store_client = apify_client.key_value_store('jane-doe/my-key-val-store') @@ -142,11 +139,9 @@ Check out the [Python API client documentation](/api/client/python/reference/cla #### JavaScript SDK -When working with a JavaScript [Actor](../../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/result-storage#key-value-store) is an essential tool, especially for key-value store management. The primary class for this purpose is the [`KeyValueStore`](/sdk/js/reference/class/KeyValueStore). This class allows you to decide whether your data will be stored locally or in the Apify cloud. For data manipulation, it offers the [`getValue()`](/sdk/js/reference/class/KeyValueStore#getValue) and [`setValue()`](/sdk/js/reference/class/KeyValueStore#setValue) methods to retrieve and assign values, respectively. - -Additionally, you can iterate over the keys in your store using the [`forEachKey()`](/sdk/js/reference/class/KeyValueStore#forEachKey) method. +In JavaScript [Actors](../../actors/index.mdx), manage key-value stores with the JavaScript SDK's [`KeyValueStore`](/sdk/js/reference/class/KeyValueStore) class. It works both locally and on the Apify platform. To read and write records, use the [`getValue()`](/sdk/js/reference/class/KeyValueStore#getValue) and [`setValue()`](/sdk/js/reference/class/KeyValueStore#setValue) methods; to iterate over keys, use [`forEachKey()`](/sdk/js/reference/class/KeyValueStore#forEachKey). -Every Actor run is linked to a default key-value store that is automatically created for that specific run. If you're running your Actors and opt to store data locally, you can easily supply the [input](../../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the corresponding directory of the default key-value store. This method ensures that your Actor has all the necessary data readily available for its execution. +Every Actor run is linked to a default key-value store, created automatically for that run. When you run your Actor locally, you can supply its [input](../../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the default key-value store's directory. You can find _INPUT.json_ and other key-value store files in the location below. @@ -212,9 +207,9 @@ Check out the [JavaScript SDK documentation](/sdk/js/docs/guides/result-storage# #### Python SDK -For Python [Actor](../../actors/index.mdx), the [Python SDK](/sdk/python/docs/concepts/storages#working-with-key-value-stores) is essential. The key-value store is represented by a [`KeyValueStore`](/sdk/python/reference/class/KeyValueStore) class. You can use this class to specify whether your data is stored locally or in the Apify cloud. For further data manipulation it offers [`get_value()`](/sdk/python/reference/class/KeyValueStore#get_value) and [`set_value()`](/sdk/python/reference/class/KeyValueStore#set_value) methods to retrieve and assign values, respectively. +In Python [Actors](../../actors/index.mdx), manage key-value stores with the Python SDK's [`KeyValueStore`](/sdk/python/reference/class/KeyValueStore) class. It works both locally and on the Apify platform. To read and write records, use the [`get_value()`](/sdk/python/reference/class/KeyValueStore#get_value) and [`set_value()`](/sdk/python/reference/class/KeyValueStore#set_value) methods. -Every Actor run is linked to a default key-value store that is automatically created for that specific run. If you're running your Actors and opt to store data locally, you can easily supply the [input](../../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the corresponding directory of the default key-value store. This method ensures that your Actor has all the necessary data readily available for its execution. +Every Actor run is linked to a default key-value store, created automatically for that run. When you run your Actor locally, you can supply its [input](../../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the default key-value store's directory. You can find _INPUT.json_ and other key-value store files in the location below. @@ -268,82 +263,34 @@ Check out the [Python SDK documentation](/sdk/python/docs/concepts/storages#work ## Compression -Previously, when using the [Store record](/api/v2/key-value-store-record-put) endpoint, every record was automatically compressed with Gzip before being uploaded. However, this process has been updated. _Now, records are stored exactly as you upload them._ This change means that it is up to you whether the record is stored compressed or uncompressed. +Records are stored exactly as you upload them, compressed or uncompressed. You can compress a record and use the [Content-Encoding request header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding) to let the platform know which compression it uses. We recommend compressing large key-value records to save storage space and network traffic. -_Using the [JavaScript SDK](/sdk/js/reference/class/KeyValueStore#setValue) or the [JavaScript API client](/api/client/js/reference/class/KeyValueStoreClient#setRecord) automatically compresses your files._ We advise utilizing the JavaScript API client for data compression prior to server upload and decompression upon retrieval, minimizing storage costs. - -## Share - -You can grant [access rights](/account/collaboration) to your key-value store through the **Share** button under the **Actions** menu. For more details check the [full list of permissions](/account/collaboration/list-of-permissions). - -You can also share key-value stores by link using their ID or name, depending on your account or resource-level general access setting. Learn how link-based access works in [General resource access](/account/collaboration/general-resource-access). - -For one-off sharing of specific records when access is restricted, you can generate time-limited pre-signed URLs. See [Sharing restricted resources with pre-signed URLs](/account/collaboration/general-resource-access#pre-signed-urls). +The [JavaScript SDK](/sdk/js/reference/class/KeyValueStore#setValue) and the [JavaScript API client](/api/client/js/reference/class/KeyValueStoreClient#setRecord) compress and decompress records automatically. -### Share key-value stores between runs +## Share and reuse {#share} -You can access a key-value store from any [Actor](../../actors/index.mdx) or [task](../../actors/running/tasks.md) run as long as you know its _name_ or _ID_. - -To access a key-value store from another run using the [Apify SDK](/sdk), open it using the same method as you would do with any other store. - - - - -```js -import { Actor } from 'apify'; +You can grant access rights to your key-value store, share it by link, or generate a time-limited pre-signed URL for specific records. See [Share storage](../share.md). -await Actor.init(); - -const otherStore = await Actor.openKeyValueStore('old-store'); -// ... - -await Actor.exit(); -``` - - - - -```python -from apify import Actor +To read from or write to a key-value store that belongs to a different Actor or task run, see [Use storage from another run](../use-from-another-run.md). -async def main(): - async with Actor: - other_store = await Actor.open_key_value_store(name='old-store') - # ... -``` - - - - -In the [JavaScript API client](/api/client/js/reference/class/KeyValueStoreClient) as well as in [Python API client](/api/client/python/reference/class/KeyValueStoreClient), you can access a store using its client. Once you've opened a store, read and manage its contents like you would do with a key-value store from your current run. - - - - -```js -const otherStoreClient = apifyClient.keyValueStore('jane-doe/old-store'); -``` +## Data consistency - - +Key-value storage uses the [AWS S3](https://aws.amazon.com/s3/) service. According to the [S3 documentation](https://aws.amazon.com/s3/consistency/), it provides _strong read-after-write_ consistency. -```python -other_store_client = apify_client.key_value_store('jane-doe/old-store') -``` +## Limits - - +- The maximum length for a key in a key-value store is 256 characters. Keys may only contain the following characters: `a-zA-Z0-9!-_.'()`. -The same applies for the [Apify API](#apify-api) - you can use [the same endpoints](#apify-api) as you would normally do. +- The maximum length for a key-value store name is 63 characters. -Check out the [Storage overview](/storage#share-storages-between-runs) for details on sharing storages between runs. +### Rate limiting -## Data consistency +Operations on a single record ([get](/api/v2/key-value-store-record-get), [put](/api/v2/key-value-store-record-put), [delete](/api/v2/key-value-store-record-delete)) and [getting the list of keys](/api/v2/key-value-store-keys-get) are limited to _200 requests per second_ per store. -Key-value storage uses the [AWS S3](https://aws.amazon.com/s3/) service. According to the [S3 documentation](https://aws.amazon.com/s3/consistency/), it provides _strong read-after-write_ consistency. +[Downloading all records](/api/v2/key-value-store-records-get) as a ZIP archive is limited to _100 requests per second_ per store. -## Limits +All other key-value store [API endpoints](/api/v2/storage-key-value-stores) use the default limit of _60 requests per second_ per store. -- The maximum length for a key in a key-value store is 256 characters. Keys may only contain the following characters: `a-zA-Z0-9!-_.'()`. +Check out the [API documentation](/api/v2#rate-limiting) for more information and guidance on actions to take if you exceed these rate limits. diff --git a/sources/platform/storage/request_queue.md b/sources/platform/storage/request_queue.md index 08dd000fa8..ea8f32ac37 100644 --- a/sources/platform/storage/request_queue.md +++ b/sources/platform/storage/request_queue.md @@ -1,6 +1,6 @@ --- title: Request queue -description: Queue URLs for an Actor to visit in its run. Learn how to share your queues between Actor runs. Access and manage request queues from Apify Console or via API. +description: Queue URLs for an Actor to visit in its run. Add requests in batches, lock them across runs, and manage request queues from Apify Console or via API. toc_max_heading_level: 4 sidebar_position: 9.4 slug: /storage/request-queue @@ -9,13 +9,13 @@ slug: /storage/request-queue import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Request queues enable you to enqueue and retrieve requests such as URLs with an [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and other parameters. They prove essential not only in web crawling scenarios but also in any situation requiring the management of a large number of URLs and the addition of new links. +Request queues store URLs to process, each with an [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and other parameters. They're built for web crawling and for any workload that manages a large list of URLs that grows as it's processed. -The storage system for request queues accommodates both breadth-first and depth-first crawling strategies, along with the inclusion of custom data attributes. This system enables you to check if certain URLs have already been encountered, add new URLs to the queue, and retrieve the next set of URLs for processing. +A request queue supports breadth-first and depth-first crawling and custom data attributes. It lets you check whether a URL was already encountered, add new URLs, and fetch the next URL to process. :::info Retention period -Named request queues are retained indefinitely. Unnamed request queues expire after 7 days unless otherwise specified. [Learn more](/storage#named-and-unnamed-storages) +Named request queues are retained indefinitely. Unnamed request queues expire after 7 days unless otherwise specified. [Learn more](/storage#data-retention) ::: @@ -25,27 +25,26 @@ Named request queues are retained indefinitely. Unnamed request queues expire af You can access your request queues in several ways: -- [Apify Console](https://console.apify.com) - provides an easy-to-understand interface. +- [Apify Console](https://console.apify.com) - view and manage your request queues in a visual interface. - [Apify API](/api/v2) - for accessing your request queues programmatically. -- [Apify API clients](/api) - to access your request queues from any Node.js application. -- [Apify SDK](/sdk) - when building your own JavaScript Actor. +- [Apify API clients](/api) - to access your request queues from any Node.js/Python application. +- [Apify SDKs](/sdk) - when building your own JavaScript/Python Actor. ### Apify Console In the [Apify Console](https://console.apify.com), you can view your request queues in the [Storage](https://console.apify.com/storage) section under the [Request queues](https://console.apify.com/storage?tab=requestQueues) tab. -![Request queues in app](./images/request-queue-app.png) +![Request queues in Apify Console](./images/storage-types-rq.svg) To view a request queue, click on its **Queue ID**. -Under the **Actions** menu, you can rename your queue's name (and, in turn, its -[retention period](/storage#named-and-unnamed-storages)) and [access rights](/account/collaboration) using the **Share** button. +Under the **Actions** menu, you can rename your queue (which affects its [retention period](/storage#named-and-unnamed-storages)) and grant [access rights](/account/collaboration) using the **Share** button. Click on the **API** button to view and test a queue's [API endpoints](/api/v2/storage-request-queues). ![Request queues detail](./images/request-queue-detail.png) ### Apify API -The [Apify API](/api/v2/storage-request-queues) allows you programmatic access to your request queues using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). +The [Apify API](/api/v2/storage-request-queues) gives you programmatic access to your request queues using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). If you are accessing your request queues using the `username~store-name` [store ID format](./index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) page of your Apify account. @@ -108,13 +107,13 @@ Example payload: :::note `clientKey` parameter -When adding or updating requests, you can optionally provide a `clientKey` parameter to your request. It must be a string between 1 and 32 characters in length. This identifier is used to determine whether the queue was accessed by [multiple clients](#sharing). If `clientKey` is not provided, the system considers this API call to come from a new client. See the `hadMultipleClients` field returned by the [`Get head`](/api/v2/request-queue-head-get) operation for details. +When adding or updating requests, you can optionally provide a `clientKey` parameter to your request. It must be a string between 1 and 32 characters in length. This identifier is used to determine whether the queue was accessed by [multiple clients](./use-from-another-run.md). If `clientKey` is not provided, the system considers this API call to come from a new client. See the `hadMultipleClients` field returned by the [`Get head`](/api/v2/request-queue-head-get) operation for details. Example: `client-abc` ::: -For further details and a breakdown of each storage API endpoint, refer to the [API documentation](/api/v2/storage-key-value-stores). +For further details and a breakdown of each storage API endpoint, refer to the [API documentation](/api/v2/storage-request-queues). ### Apify API Clients @@ -122,7 +121,7 @@ For further details and a breakdown of each storage API endpoint, refer to the [ The Apify [JavaScript API client](/api/client/js/reference/class/RequestQueueClient) (`apify-client`) enables you to access your request queues from any Node.js application, whether it is running on the Apify platform or externally. -After importing and initiating the client, you can save each request queue to a variable for easier access. +After importing and initializing the client, you can save each request queue to a variable for easier access. ```js const myQueueClient = apifyClient.requestQueue('jane-doe/my-request-queue'); @@ -136,7 +135,7 @@ Check out the [JavaScript API client documentation](/api/client/js/reference/cla The Apify [Python API client](/api/client/python) (`apify-client`) allows you to access your request queues from any Python application, whether it's running on the Apify platform or externally. -After importing and initiating the client, you can save each request queue to a variable for easier access. +After importing and initializing the client, you can save each request queue to a variable for easier access. ```python my_queue_client = apify_client.request_queue('jane-doe/my-request-queue') @@ -150,11 +149,9 @@ Check out the [Python API client documentation](/api/client/python/reference/cla #### JavaScript SDK -When working with a JavaScript [Actor](../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/request-storage#request-queue) is an essential tool, especially for request queue management. The primary class for this purpose is the [`RequestQueue`](/sdk/js/reference/class/RequestQueue) class. Use this class to decide whether your data is stored locally or in the Apify cloud. +In JavaScript [Actors](../actors/index.mdx), manage request queues with the JavaScript SDK's [`RequestQueue`](/sdk/js/reference/class/RequestQueue) class. It works both locally and on the Apify platform. To add URLs to the queue, use [`addRequests()`](/sdk/js/reference/class/RequestQueue#addRequests). -If you are building a JavaScript [Actor](../actors/index.mdx), you will be using the [JavaScript SDK](/sdk/js/docs/guides/request-storage#request-queue). The request queue is represented by a [`RequestQueue`](/sdk/js/reference/class/RequestQueue) class. You can use the class to specify whether your data is stored locally or in the Apify cloud and [enqueue new URLs](/sdk/js/reference/class/RequestQueue#addRequests). - -Every Actor run is automatically linked with a default request queue, initiated upon adding the first request. This queue is primarily utilized for storing URLs to be crawled during the particular Actor run, though its use is not mandatory. For enhanced flexibility, you can establish named queues. These named queues offer the advantage of being shareable across different Actors or various Actor runs, facilitating a more interconnected and efficient process. +Every Actor run gets a default request queue, created when the first request is added. The run typically uses it to store the URLs it crawls, but doesn't have to. You can also create named queues and share them between Actors or runs. If you are storing your data locally, you can find your request queue at the following location. @@ -223,9 +220,9 @@ Check out the [JavaScript SDK documentation](/sdk/js/docs/guides/request-storage #### Python SDK -For Python [Actor](../actors/index.mdx) development, the [Python SDK](/sdk/python/docs/concepts/storages#working-with-request-queues) the in essential. The request queue is represented by [`RequestQueue`](/sdk/python/reference/class/RequestQueue) class. Utilize this class to determine whether your data is stored locally or in the Apify cloud. For managing your data, it provides the capability to [enqueue new URLs](/sdk/python/reference/class/RequestQueue#add_requests), facilitating seamless integration and operation within your Actor. +In Python [Actors](../actors/index.mdx), manage request queues with the Python SDK's [`RequestQueue`](/sdk/python/reference/class/RequestQueue) class. It works both locally and on the Apify platform. To add URLs to the queue, use [`add_requests()`](/sdk/python/reference/class/RequestQueue#add_requests). -Every Actor run is automatically connected to a default request queue, established specifically for that run upon the addition of the first request. If you're operating your Actors and choose to utilize this queue, it typically serves to store URLs for crawling in the respective Actor run, though its use is not mandatory. To extend functionality, you have the option to create named queue, which offer the flexibility to be shared among different Actors or across multiple Actor runs. +Every Actor run gets a default request queue, created when the first request is added. The run typically uses it to store the URLs it crawls, but doesn't have to. You can also create named queues and share them between Actors or runs. If you are storing your data locally, you can find your request queue at the following location. @@ -262,9 +259,9 @@ async def main(): queue: RequestQueue = await Actor.open_request_queue() # Enqueue requests - await queue.add_request(request={'url': 'http:#example.com/aaa'}) - await queue.add_request(request={'url': 'http:#example.com/foo'}) - await queue.add_request(request={'url': 'http:#example.com/bar'}, forefront=True) + await queue.add_request(request={'url': 'http://example.com/aaa'}) + await queue.add_request(request={'url': 'http://example.com/foo'}) + await queue.add_request(request={'url': 'http://example.com/bar'}, forefront=True) # Get the next requests from queue request1 = await queue.fetch_next_request() @@ -284,15 +281,12 @@ Check out the [Python SDK documentation](/sdk/python/docs/concepts/storages#work ## Features -Request queue is a storage type built with scraping in mind, enabling developers to write scraping logic efficiently and scalably. -The Apify tooling, including [Crawlee](https://crawlee.dev/), [Apify SDK for JavaScript](https://docs.apify.com/sdk/js/), and [Apify SDK for Python](https://docs.apify.com/sdk/python/), incorporates all these features, enabling users to leverage them effortlessly without extra configuration. - -The following sections cover each of the main features in depth. +Request queues are built for scraping workloads. The following sections cover the main features; [Crawlee](https://crawlee.dev/), the [Apify SDK for JavaScript](https://docs.apify.com/sdk/js/), and the [Apify SDK for Python](https://docs.apify.com/sdk/python/) support all of them without extra configuration. ### Persistence and retention -Request queues prioritize persistence, ensuring indefinite retention of your requests in named request queues, and for the data retention period in your subscription in unnamed request queues. -This capability facilitates incremental crawling, where you can append new URLs to the queue and resume from where you stopped in subsequent Actor runs. +Named request queues retain requests indefinitely; unnamed queues follow your subscription's data retention period. +This enables incremental crawling: append new URLs to the queue and resume from where you stopped in subsequent Actor runs. Consider the scenario of scraping an e-commerce website with thousands of products. Incremental scraping allows you to scrape only the products added since the last product discovery. @@ -413,7 +407,7 @@ request_queue_client.batch_delete_requests([ Request queue includes a locking mechanism to avoid concurrent processing of one request by multiple clients (for example Actor runs). You can lock a request so that no other clients receive it when they fetch the queue head, with an expiration period on the lock so that requests which fail processing are eventually unlocked and retried. -This feature is seamlessly integrated into Crawlee, requiring minimal extra setup. By default, requests are locked for the same duration as the timeout for processing requests in the crawler ([`requestHandlerTimeoutSecs`](https://crawlee.dev/api/next/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs)). +Crawlee supports request locking with minimal setup. By default, requests are locked for the same duration as the timeout for processing requests in the crawler ([`requestHandlerTimeoutSecs`](https://crawlee.dev/api/next/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs)). If the Actor processing the request fails, the lock expires, and the request is processed again eventually. For more details, refer to the [Crawlee documentation](https://crawlee.dev/docs/next/experiments/experiments-request-locking). In the following example, we demonstrate how you can use locking mechanisms to avoid concurrent processing of the same request across multiple Actor runs. @@ -421,7 +415,7 @@ In the following example, we demonstrate how you can use locking mechanisms to a :::info Lock mechanism The lock mechanism works on the client level, as well as the run level, when running the Actor on the Apify platform. -This means you can unlock or prolong the lock the locked request only if: +This means you can unlock or prolong the lock of a locked request only if: - You are using the same client key, or - The operation is being called from the same Actor run. @@ -563,71 +557,11 @@ await Actor.exit(); A detailed tutorial on how to process one request queue with multiple Actor runs can be found in [Academy tutorials](https://docs.apify.com/academy/node-js/multiple-runs-scrape). -## Share - -You can grant [access rights](/account/collaboration) to your request queue through the **Share** button under the **Actions** menu. For more details check the [full list of permissions](/account/collaboration/list-of-permissions). - -You can also share request queues by link using their ID or name, depending on your account or resource-level general access setting. Learn how link-based access works in [General resource access](/account/collaboration/general-resource-access). - -For one-off sharing of specific records when access is restricted, you can generate time-limited pre-signed URLs. See [Sharing restricted resources with pre-signed URLs](/account/collaboration/general-resource-access#pre-signed-urls). - -### Share request queues between runs - -You can access a request queue from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_. - -To access a request queue from another run using the [Apify SDK](/sdk), open it using the same method like you would do with any other request queue. - - - - -```js -import { Actor } from 'apify'; - -await Actor.init(); - -const otherQueue = await Actor.openRequestQueue('old-queue'); -// ... - -await Actor.exit(); -``` - - - - -```python -from apify import Actor - -async def main(): - async with Actor: - other_queue = await Actor.open_request_queue(name='old-queue') - # ... -``` - - - - -In the [JavaScript API client](/api/client/js/reference/class/RequestQueueClient) as well as in [Python API client](/api/client/python/reference/class/RequestQueueClient), you can access a request queue using its respective client. Once you've opened the request queue, you can use it in your crawler or add new requests like you would do with a queue from your current run. - - - - -```js -const otherQueueClient = apifyClient.requestQueue('jane-doe/old-queue'); -``` - - - - -```python -other_queue_client = apify_client.request_queue('jane-doe/old-queue') -``` - - - +## Share and reuse {#share} -The same applies for the [Apify API](#apify-api) - you can use [the same endpoints](#apify-api) as you would normally do. +You can grant access rights to your request queue or share it by link. See [Share storage](./share.md). -Check out the [Storage overview](/storage#share-storages-between-runs) for details on sharing storages between runs. +To add requests to a queue that belongs to a different Actor or task run, see [Use storage from another run](./use-from-another-run.md). ## Limits @@ -640,11 +574,11 @@ CRUD ([add](/api/v2/request-queue-requests-post), [get](/api/v2/request-queue-request-get), [update](/api/v2/request-queue-request-put), [delete](/api/v2/request-queue-request-delete)) -operation requests are limited to _400 requests per second_ per request queue. This helps protect Apify servers from being overloaded. +operation requests are limited to _350 requests per second_ per request queue. This helps protect Apify servers from being overloaded. -Request-lock operations ([prolong a lock](/api/v2/request-queue-request-lock-put) and [delete a lock](/api/v2/request-queue-request-lock-delete)) are also limited to _400 requests per second_ per request queue. +Request-lock operations ([prolong a lock](/api/v2/request-queue-request-lock-put) and [delete a lock](/api/v2/request-queue-request-lock-delete)) are also limited to _350 requests per second_ per request queue. -Batch operations ([add](/api/v2/request-queue-requests-batch-post) and [delete](/api/v2/request-queue-requests-batch-delete)) and the [list and lock head](/api/v2/request-queue-head-lock-post) endpoint are limited to _40 requests per second_ per request queue. +Batch operations ([add](/api/v2/request-queue-requests-batch-post) and [delete](/api/v2/request-queue-requests-batch-delete)) and the [list and lock head](/api/v2/request-queue-head-lock-post) endpoint are limited to _35 requests per second_ per request queue. All other request queue API [endpoints](/api/v2/storage-request-queues) are limited to _60 requests per second_ per request queue. diff --git a/sources/platform/storage/share.md b/sources/platform/storage/share.md new file mode 100644 index 0000000000..bc1312add8 --- /dev/null +++ b/sources/platform/storage/share.md @@ -0,0 +1,27 @@ +--- +title: Share storage +sidebar_label: Share +description: Grant access rights to your storages, share them by link using their ID or name, and generate time-limited pre-signed URLs for restricted resources. +sidebar_position: 9.5 +slug: /storage/share +--- + +Share a storage with other Apify users, by link, or through a time-limited URL for one-off access to a restricted resource. This applies to all three storage types: [datasets](./dataset/index.md), [key-value stores](./key_value_store/index.md), and [request queues](./request_queue.md). To use a storage that belongs to a different run, see [Use storage from another run](./use-from-another-run.md). + +## Grant access rights + +Grant [access rights](/account/collaboration) to let other Apify users view or modify your storages. For what each right allows, see the [list of permissions](/account/collaboration/list-of-permissions). + +In [Apify Console](https://console.apify.com/storage), open the storage's detail page and use the **Share** button under the **Actions** menu. + +## Share by link + +Share storages by link using their ID or name. Whether the link works for others depends on your account or resource-level general access setting; learn how link-based access works in [General resource access](/account/collaboration/general-resource-access). + +For one-off sharing of individual records or items when access is restricted, generate time-limited pre-signed URLs. These are available for selected dataset and key-value store endpoints only. See [Sharing restricted resources with pre-signed URLs](/account/collaboration/general-resource-access#pre-signed-urls). + +:::tip Accessing restricted storage resources via API + +If your storage resource is set to _restricted_, all API calls must include a valid authentication token in the `Authorization` header. If you're using `apify-client` the header is passed in automatically. + +::: diff --git a/sources/platform/storage/use-from-another-run.md b/sources/platform/storage/use-from-another-run.md new file mode 100644 index 0000000000..cc2f6fe035 --- /dev/null +++ b/sources/platform/storage/use-from-another-run.md @@ -0,0 +1,98 @@ +--- +title: Use storage from another run +sidebar_label: Use from another run +description: Open a dataset, key-value store, or request queue that belongs to another Actor or task run by its name or ID, from the SDK, API clients, or the API. +sidebar_position: 9.6 +slug: /storage/use-from-another-run +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +If you have the storage's _name_ or _ID_, you can access it from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run. Use the same methods and endpoints you'd use for the current run's storages. + +[Datasets](./dataset/index.md) and [key-value stores](./key_value_store/index.md) support concurrent use. Multiple Actors or tasks can write to the same dataset or key-value store, and multiple runs can read from them at the same time. + +[Request queues](./request_queue.md) only allow multiple runs to add new data. A request queue can be processed by one Actor or task run at a time, unless you use [request locking](./request_queue.md#distributivity) to coordinate multiple runs. + +When multiple runs use the same storage at the same time, the order in which their operations are processed is not guaranteed. For example, if a delete of a key-value store record is processed before a read of the same record, the read fails. + +:::info Accessing restricted storage resources between runs + +If a storage resource access is set to **Restricted**, the run from which it's accessed must have explicit access to it. Learn how restricted access works in [General resource access](/account/collaboration/general-resource-access), and how to grant it in [Share storage](./share.md). + +::: + +## Open a storage with the SDK + +Open the storage with the same method you would use for the current run's storage, and pass the name or ID of the one you want. + + + + +```js +import { Actor } from 'apify'; + +await Actor.init(); + +const otherDataset = await Actor.openDataset('old-dataset'); +// ... + +await Actor.exit(); +``` + + + + +```python +from apify import Actor + +async def main(): + async with Actor: + other_dataset = await Actor.open_dataset(name='old-dataset') + # ... +``` + + + + +Only the method name changes with the storage type: + +| Storage type | [JavaScript SDK](/sdk/js) | [Python SDK](/sdk/python) | +| --------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | +| Dataset | [`Actor.openDataset()`](/sdk/js/reference/class/Actor#openDataset) | [`Actor.open_dataset()`](/sdk/python/reference/class/Actor#open_dataset) | +| Key-value store | [`Actor.openKeyValueStore()`](/sdk/js/reference/class/Actor#openKeyValueStore) | [`Actor.open_key_value_store()`](/sdk/python/reference/class/Actor#open_key_value_store) | +| Request queue | [`Actor.openRequestQueue()`](/sdk/js/reference/class/Actor#openRequestQueue) | [`Actor.open_request_queue()`](/sdk/python/reference/class/Actor#open_request_queue) | + +## Open a storage with an API client + +Construct the storage's client with the name or ID of the storage you want. To use a storage owned by another user, prefix the name with their username, as in `jane-doe/old-dataset`. Then read and write exactly as you would with the current run's storage. + + + + +```js +const otherDatasetClient = apifyClient.dataset('jane-doe/old-dataset'); +``` + + + + +```python +other_dataset_client = apify_client.dataset('jane-doe/old-dataset') +``` + + + + +Only the accessor changes with the storage type: + +| Storage type | JavaScript client | Python client | +| --------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| Dataset | [`apifyClient.dataset()`](/api/client/js/reference/class/DatasetClient) | [`apify_client.dataset()`](/api/client/python/reference/class/DatasetClient) | +| Key-value store | [`apifyClient.keyValueStore()`](/api/client/js/reference/class/KeyValueStoreClient) | [`apify_client.key_value_store()`](/api/client/python/reference/class/KeyValueStoreClient) | +| Request queue | [`apifyClient.requestQueue()`](/api/client/js/reference/class/RequestQueueClient) | [`apify_client.request_queue()`](/api/client/python/reference/class/RequestQueueClient) | + +## Use the Apify API + +Send requests to the same endpoints you would use for the current run's storages, passing the name or ID of the storage you want. See the endpoint reference for [datasets](./dataset/index.md#apify-api), [key-value stores](./key_value_store/index.md#apify-api), and [request queues](./request_queue.md#apify-api).