Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 39 additions & 16 deletions nginx.conf

Large diffs are not rendered by default.

18 changes: 8 additions & 10 deletions website/route-lockfile.txt
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,16 @@
/en/resources/tokenomics/
/en/subgraphs/best-practices/avoid-eth-calls/
/en/subgraphs/best-practices/derivedfrom/
/en/subgraphs/best-practices/grafting-hotfix/
/en/subgraphs/best-practices/enums/
/en/subgraphs/best-practices/grafting/
/en/subgraphs/best-practices/immutable-entities-bytes-as-ids/
/en/subgraphs/best-practices/overview/
/en/subgraphs/best-practices/pruning/
/en/subgraphs/best-practices/secure-api-keys-nextjs/
/en/subgraphs/best-practices/subgraph-composition/
/en/subgraphs/best-practices/subgraph-debug-forking/
/en/subgraphs/best-practices/timeseries/
/en/subgraphs/best-practices/transfer-to-the-graph/
/en/subgraphs/developing/creating/advanced/
/en/subgraphs/developing/creating/assemblyscript-mappings/
/en/subgraphs/developing/creating/graph-node-dev/
Expand All @@ -49,12 +55,6 @@
/en/subgraphs/existing-subgraphs/explorer/
/en/subgraphs/existing-subgraphs/polymarket/
/en/subgraphs/existing-subgraphs/standard-subgraphs/
/en/subgraphs/guides/enums/
/en/subgraphs/guides/grafting/
/en/subgraphs/guides/secure-api-keys-nextjs/
/en/subgraphs/guides/subgraph-composition/
/en/subgraphs/guides/subgraph-debug-forking/
/en/subgraphs/guides/transfer-to-the-graph/
/en/subgraphs/overview/
/en/subgraphs/providers/subgraph-studio/fair-use-policy/
/en/subgraphs/providers/subgraph-studio/introduction/
Expand Down Expand Up @@ -100,7 +100,6 @@
/en/supported-networks/avalanche/
/en/supported-networks/base-sepolia/
/en/supported-networks/base/
/en/supported-networks/blast-mainnet/
/en/supported-networks/bnb-svm/
/en/supported-networks/boba-testnet/
/en/supported-networks/boba/
Expand All @@ -113,6 +112,7 @@
/en/supported-networks/chapel/
/en/supported-networks/chiliz-testnet/
/en/supported-networks/chiliz/
/en/supported-networks/elysium/
/en/supported-networks/eos/
/en/supported-networks/etherlink-mainnet/
/en/supported-networks/etherlink-shadownet/
Expand Down Expand Up @@ -171,8 +171,6 @@
/en/supported-networks/sonic-testnet/
/en/supported-networks/sonic/
/en/supported-networks/stable/
/en/supported-networks/starknet-mainnet/
/en/supported-networks/starknet-testnet/
/en/supported-networks/stellar-testnet/
/en/supported-networks/stellar/
/en/supported-networks/tempo-moderato/
Expand Down
4 changes: 2 additions & 2 deletions website/src/HomePage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ export default function HomePage({ supportedNetworks }: { supportedNetworks: Sup
slotAboveTitle={<TimeIcon variant="reading" minutes={9} />}
/>
<Card
href="/subgraphs/cookbook/timeseries/"
href="/subgraphs/best-practices/timeseries/"
title={t('index.guides.timeseries.title')}
description={t('index.guides.timeseries.description')}
slotAboveTitle={<TimeIcon variant="reading" minutes={4} />}
Expand All @@ -251,7 +251,7 @@ export default function HomePage({ supportedNetworks }: { supportedNetworks: Sup
slotAboveTitle={<TimeIcon variant="reading" minutes={2} />}
/>
<Card
href="/subgraphs/cookbook/transfer-to-the-graph/"
href="/subgraphs/best-practices/transfer-to-the-graph/"
title={t('index.guides.transferToTheGraph.title')}
description={t('index.guides.transferToTheGraph.description')}
slotAboveTitle={<TimeIcon variant="reading" minutes={3} />}
Expand Down
2 changes: 1 addition & 1 deletion website/src/pages/en/indexing/new-chain-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ For EVM chains, there exists a deeper level of data that can be achieved through

## EVM considerations - Difference between JSON-RPC & Firehose

While the JSON-RPC and Firehose are both suitable for Subgraphs, a Firehose is always required for developers wanting to build with [Substreams](https://substreams.streamingfast.io). Supporting Substreams allows developers to build [Substreams-powered Subgraphs](/subgraphs/cookbook/substreams-powered-subgraphs/) for the new chain, and has the potential to improve the performance of your Subgraphs. Additionally, Firehose — as a drop-in replacement for the JSON-RPC extraction layer of `graph-node` — reduces by 90% the number of RPC calls required for general indexing.
While the JSON-RPC and Firehose are both suitable for Subgraphs, a Firehose is always required for developers wanting to build with [Substreams](https://substreams.streamingfast.io). Supporting Substreams allows developers to build [Substreams-powered Subgraphs](https://thegraph.com/blog/substreams-powered-subgraphs/) for the new chain, and has the potential to improve the performance of your Subgraphs. Additionally, Firehose — as a drop-in replacement for the JSON-RPC extraction layer of `graph-node` — reduces by 90% the number of RPC calls required for general indexing.

- All those `getLogs` calls and roundtrips get replaced by a single stream arriving into the heart of `graph-node`; a single block model for all Subgraphs it processes.

Expand Down
5 changes: 2 additions & 3 deletions website/src/pages/en/subgraphs/_meta-titles.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
"providers": "Providers",
"querying": "Querying",
"developing": "Developing",
"guides": "How-to Guides",
"best-practices": "Best Practices",
"tooling": "Tooling"
"best-practices": "Guides & Best Practices",
"tooling": "Testing & Tooling"
}
1 change: 0 additions & 1 deletion website/src/pages/en/subgraphs/_meta.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,4 @@ export default {
querying: titles.querying ?? '',
tooling: titles.tooling ?? '',
'best-practices': titles['best-practices'] ?? '',
guides: titles.guides ?? '',
}
18 changes: 12 additions & 6 deletions website/src/pages/en/subgraphs/best-practices/_meta.js
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
export default {
'immutable-entities-bytes-as-ids': '',
derivedfrom: '',
pruning: '',
'avoid-eth-calls': '',
timeseries: '',
'grafting-hotfix': '',
overview: 'Overview',
'avoid-eth-calls': 'Avoid eth_calls',
'immutable-entities-bytes-as-ids': 'Immutable Entities and Bytes IDs',
derivedfrom: 'Arrays with @derivedFrom',
timeseries: 'Timeseries and Aggregations',
enums: 'Categorize Data with Enums',
pruning: 'Pruning with indexerHints',
'secure-api-keys-nextjs': 'Secure API Keys with Next.js',
'subgraph-debug-forking': 'Debug with Subgraph Forking',
grafting: 'Grafting',
'subgraph-composition': 'Subgraph Composition',
'transfer-to-the-graph': 'Transfer to The Graph',
}
67 changes: 28 additions & 39 deletions website/src/pages/en/subgraphs/best-practices/avoid-eth-calls.mdx
Original file line number Diff line number Diff line change
@@ -1,25 +1,22 @@
---
title: Subgraph Best Practice 4 - Improve Indexing Speed by Avoiding eth_calls
sidebarTitle: 'Avoiding eth_calls'
title: Improve Indexing Speed by Avoiding eth_calls
---

## TLDR
An `eth_call` is a call from a Subgraph handler to a blockchain node to read contract state. Each one is an RPC round trip that takes a significant amount of time, so `eth_calls` are the most common cause of slow syncs. Where you can, design smart contracts to emit all the data your Subgraph needs, so handlers never need to call the contract.

`eth_calls` are calls that can be made from a Subgraph to an Ethereum node. These calls take a significant amount of time to return data, slowing down indexing. If possible, design smart contracts to emit all the data you need so you don’t need to use `eth_calls`.
## Why `eth_calls` Slow Down Indexing

## Why Avoiding `eth_calls` Is a Best Practice
Subgraphs are optimized to index event data emitted from smart contracts. A Subgraph can also index the data coming from an `eth_call`, however, this can significantly slow down Subgraph indexing as `eth_calls` require making external calls to smart contracts. Their speed depends not on the Subgraph but on the connectivity and responsiveness of the node being queried. Minimizing or eliminating `eth_calls` significantly improves indexing speed.

Subgraphs are optimized to index event data emitted from smart contracts. A Subgraph can also index the data coming from an `eth_call`, however, this can significantly slow down Subgraph indexing as `eth_calls` require making external calls to smart contracts. The responsiveness of these calls relies not on the Subgraph but on the connectivity and responsiveness of the Ethereum node being queried. By minimizing or eliminating eth_calls in our Subgraphs, we can significantly improve our indexing speed.
### What an `eth_call` Looks Like

### What Does an eth_call Look Like?
`eth_calls` are often necessary when the data required for a Subgraph is not available through emitted events. For example, consider a scenario where a Subgraph needs to identify whether ERC20 tokens are part of a specific pool, but the contract only emits a basic `Transfer` event that does not contain that data:

`eth_calls` are often necessary when the data required for a Subgraph is not available through emitted events. For example, consider a scenario where a Subgraph needs to identify whether ERC20 tokens are part of a specific pool, but the contract only emits a basic `Transfer` event and does not emit an event that contains the data that we need:

```yaml
```solidity
event Transfer(address indexed from, address indexed to, uint256 value);
```

Suppose the tokens' pool membership is determined by a state variable named `getPoolInfo`. In this case, we would need to use an `eth_call` to query this data:
Suppose the tokens' pool membership is determined by a state variable named `getPoolInfo`. In this case, the handler needs an `eth_call` to read it:

```typescript
import { Address } from '@graphprotocol/graph-ts'
Expand All @@ -44,13 +41,13 @@ export function handleTransfer(event: Transfer): void {
}
```

This is functional, however is not ideal as it slows down our Subgraph’s indexing.
This works, but every `Transfer` now waits on an RPC call, which slows indexing.

## How to Eliminate `eth_calls`

Ideally, the smart contract should be updated to emit all necessary data within events. For instance, modifying the smart contract to include pool information in the event could eliminate the need for `eth_calls`:

```
```solidity
event TransferWithPool(address indexed from, address indexed to, uint256 value, bytes32 indexed poolInfo);
```

Expand All @@ -73,45 +70,37 @@ export function handleTransferWithPool(event: TransferWithPool): void {
}
```

This is much more performant as it has eliminated the need for `eth_calls`.
This is much faster because the handler no longer makes any `eth_calls`.

## How to Reduce `eth_calls` You Cannot Remove

## How to Optimize `eth_calls`
If you cannot change the contract, cut calls in this order:

If modifying the smart contract is not possible and `eth_calls` are required, read “[Improve Subgraph Indexing Performance Easily: Reduce eth_calls](https://thegraph.com/blog/improve-subgraph-performance-reduce-eth-calls/)” by Simon Emanuel Schmid to learn various strategies on how to optimize `eth_calls`.
1. **Read from the event:** Check `event.params` before you add any call.
2. **Cache values that never change:** Fetch values such as `name`, `symbol`, and `decimals` once, when the entity is created, and reuse the stored value.
3. **Compute in the Subgraph:** Derive values such as `totalSupply` from mint and burn events instead of calling the contract.
4. **Declare the calls you keep** (see the next section).

For more strategies, read [Improve Subgraph Indexing Performance Easily: Reduce eth_calls](https://thegraph.com/blog/improve-subgraph-performance-reduce-eth-calls/).

## Reducing the Runtime Overhead of `eth_calls`

For the `eth_calls` that can not be eliminated, the runtime overhead they introduce can be minimized by declaring them in the manifest. When `graph-node` processes a block it performs all declared `eth_calls` in parallel before handlers are run. Calls that are not declared are executed sequentially when handlers run. The runtime improvement comes from performing calls in parallel rather than sequentially - that helps reduce the total time spent in calls but does not eliminate it completely.
For the `eth_calls` that can not be eliminated, the runtime overhead they introduce can be minimized by declaring them in the manifest. When Graph Node processes a block, it runs all declared `eth_calls` in parallel before the handlers run. Calls that are not declared are executed sequentially when handlers run. The runtime improvement comes from performing calls in parallel rather than sequentially - that helps reduce the total time spent in calls but does not eliminate it completely.

Currently, `eth_calls` can only be declared for event handlers. In the manifest, write
You can declare `eth_calls` only for event handlers, and only with `specVersion` 1.2.0 or later. In the manifest, add a `calls` section to the handler:

```yaml
event: TransferWithPool(address indexed, address indexed, uint256, bytes32 indexed)
event: TransferWithPool(indexed address,indexed address,uint256,indexed bytes32)
handler: handleTransferWithPool
calls:
ERC20.poolInfo: ERC20[event.address].getPoolInfo(event.params.to)
```

The portion highlighted in yellow is the call declaration. The part before the colon is simply a text label that is only used for error messages. The part after the colon has the form `Contract[address].function(params)`. Permissible values for address and params are `event.address` and `event.params.<name>`.

The handler itself accesses the result of this `eth_call` exactly as in the previous section by binding to the contract and making the call. graph-node caches the results of declared `eth_calls` in memory and the call from the handler will retrieve the result from this in memory cache instead of making an actual RPC call.

Note: Declared eth_calls can only be made in Subgraphs with specVersion >= 1.2.0.

## Conclusion

You can significantly improve indexing performance by minimizing or eliminating `eth_calls` in your Subgraphs.

## Subgraph Best Practices 1-6

1. [Improve Query Speed with Subgraph Pruning](/subgraphs/best-practices/pruning/)

2. [Improve Indexing and Query Responsiveness by Using @derivedFrom](/subgraphs/best-practices/derivedfrom/)

3. [Improve Indexing and Query Performance by Using Immutable Entities and Bytes as IDs](/subgraphs/best-practices/immutable-entities-bytes-as-ids/)
The `calls` entry is the call declaration. The part before the colon is a text label used only in error messages. The part after the colon has the form `Contract[address].function(params)`. Permissible values for address and params are `event.address` and `event.params.<name>`.

4. [Improve Indexing Speed by Avoiding `eth_calls`](/subgraphs/best-practices/avoid-eth-calls/)
The handler itself accesses the result of this `eth_call` exactly as in the previous section by binding to the contract and making the call. Graph Node caches the results of declared `eth_calls` in memory, so the call in the handler reads the cached result instead of making an RPC call.

5. [Simplify and Optimize with Timeseries and Aggregations](/subgraphs/best-practices/timeseries/)
## Learn More

6. [Use Grafting for Quick Hotfix Deployment](/subgraphs/best-practices/grafting-hotfix/)
- [Declared eth_calls](/subgraphs/developing/creating/advanced/#declared-eth_call): the manifest reference.
- [Guides & Best Practices overview](/subgraphs/best-practices/overview/): the full checklist and how this page fits with the rest of the section.
49 changes: 14 additions & 35 deletions website/src/pages/en/subgraphs/best-practices/derivedfrom.mdx
Original file line number Diff line number Diff line change
@@ -1,15 +1,12 @@
---
title: Subgraph Best Practice 2 - Improve Indexing and Query Responsiveness By Using @derivedFrom
sidebarTitle: 'Arrays with @derivedFrom'
title: Use @derivedFrom Instead of Growing Arrays
---

## TLDR

Arrays in your schema can really slow down a Subgraph's performance as they grow beyond thousands of entries. If possible, the `@derivedFrom` directive should be used when using arrays as it prevents large arrays from forming, simplifies handlers, and reduces the size of individual entities, improving indexing speed and query performance significantly.
Arrays in your schema slow a Subgraph down as they grow beyond thousands of entries, because Graph Node copies the whole array on every update. Use the `@derivedFrom` directive for one-to-many relationships instead. It prevents large arrays from forming, simplifies handlers, and keeps entities small, which speeds up both indexing and queries.

## How to Use the `@derivedFrom` Directive

You just need to add a `@derivedFrom` directive after your array in your schema. Like this:
Add a `@derivedFrom` directive after the array field in your schema:

```graphql
comments: [Comment!]! @derivedFrom(field: "post")
Expand All @@ -21,9 +18,7 @@ comments: [Comment!]! @derivedFrom(field: "post")

An example of a dynamically growing array is a blogging platform where a “Post” can have many “Comments”.

Let’s start with our two entities, `Post` and `Comment`

Without optimization, you could implement it like this with an array:
Start with two entities, `Post` and `Comment`. Without optimization, you might store the comments as an array on the post:

```graphql
type Post @entity {
Expand All @@ -39,9 +34,9 @@ type Comment @entity {
}
```

Arrays like these will effectively store extra Comments data on the Post side of the relationship.
This array stores the comment references on the `Post` side of the relationship, and the array grows with every new comment.

Here’s what an optimized version looks like using `@derivedFrom`:
Here is the optimized version with `@derivedFrom`:

```graphql
type Post @entity {
Expand All @@ -58,31 +53,15 @@ type Comment @entity {
}
```

Just by adding the `@derivedFrom` directive, this schema will only store the “Comments” on the “Comments” side of the relationship and not on the “Post” side of the relationship. Arrays are stored across individual rows, which allows them to expand significantly. This can lead to particularly large sizes if their growth is unbounded.

This will not only make our Subgraph more efficient, but it will also unlock three features:

1. We can query the `Post` and see all of its comments.
2. We can do a reverse lookup and query any `Comment` and see which post it comes from.

3. We can use [Derived Field Loaders](/subgraphs/developing/creating/graph-ts/api/#looking-up-derived-entities) to unlock the ability to directly access and manipulate data from virtual relationships in our Subgraph mappings.

## Conclusion

Use the `@derivedFrom` directive in Subgraphs to effectively manage dynamically growing arrays, enhancing indexing efficiency and data retrieval.

For a more detailed explanation of strategies to avoid large arrays, check out Kevin Jones' blog: [Best Practices in Subgraph Development: Avoiding Large Arrays](https://thegraph.com/blog/improve-subgraph-performance-avoiding-large-arrays/).

## Subgraph Best Practices 1-6

1. [Improve Query Speed with Subgraph Pruning](/subgraphs/best-practices/pruning/)

2. [Improve Indexing and Query Responsiveness by Using @derivedFrom](/subgraphs/best-practices/derivedfrom/)
With `@derivedFrom`, the schema stores the relationship only on the `Comment` side, as a single `post` field. The `comments` field on `Post` is computed at query time, so it never grows in storage.

3. [Improve Indexing and Query Performance by Using Immutable Entities and Bytes as IDs](/subgraphs/best-practices/immutable-entities-bytes-as-ids/)
This makes your Subgraph more efficient and gives you three capabilities:

4. [Improve Indexing Speed by Avoiding `eth_calls`](/subgraphs/best-practices/avoid-eth-calls/)
1. Query a `Post` and see all of its comments.
2. Do a reverse lookup: query any `Comment` and see which post it belongs to.
3. Use [derived field loaders](/subgraphs/developing/creating/graph-ts/api/#looking-up-derived-entities) (`loadRelated()`) to read virtual relationships in your mappings. Derived fields are read-only in mappings.

5. [Simplify and Optimize with Timeseries and Aggregations](/subgraphs/best-practices/timeseries/)
## Learn More

6. [Use Grafting for Quick Hotfix Deployment](/subgraphs/best-practices/grafting-hotfix/)
- [Best Practices in Subgraph Development: Avoiding Large Arrays](https://thegraph.com/blog/improve-subgraph-performance-avoiding-large-arrays/): more strategies for avoiding large arrays.
- [Guides & Best Practices overview](/subgraphs/best-practices/overview/): the full checklist and how this page fits with the rest of the section.
Loading
Loading