docs: add Experimentation guides#7993
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
📝 WalkthroughWalkthroughAdds an Experimentation documentation section covering warehouse connections, metrics, experiment creation, event instrumentation, PayPal testing, result analysis, and anonymous identity handling. It also creates category navigation, updates the statistics glossary, reformats the existing A/B testing guide, and adds redirects for renamed documentation pages. Estimated code review effort: 3 (Moderate) | ~20 minutes Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 52af566f-521f-4125-bce3-1858e21df39d
⛔ Files ignored due to path filters (5)
docs/static/img/experimentation/create-experiment-rollout.pngis excluded by!**/*.pngdocs/static/img/experimentation/create-experiment-setup.pngis excluded by!**/*.pngdocs/static/img/experimentation/experiment-results-analysis.pngis excluded by!**/*.pngdocs/static/img/experimentation/experiment-results-overview.pngis excluded by!**/*.pngdocs/static/img/experimentation/warehouse-tab.pngis excluded by!**/*.png
📒 Files selected for processing (11)
docs/docs/experimentation-ab-testing.mddocs/docs/experimentation/_category_.jsondocs/docs/experimentation/analyse-an-experiment.mddocs/docs/experimentation/connect-a-warehouse.mddocs/docs/experimentation/create-an-experiment.mddocs/docs/experimentation/create-metrics.mddocs/docs/experimentation/example-paypal-button.mddocs/docs/experimentation/index.mddocs/docs/experimentation/run-an-experiment.mdxdocs/docs/experimentation/statistics.mddocs/vercel.json
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/docs/experimentation/connect-a-warehouse.md (1)
26-30: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winClarify the transition between
CreatedandPending Connection.
Createdis defined as “the connection exists but no events have been received yet”, which overlaps withPending Connectionbeing “waiting for the first event to arrive”. Give each status a distinct trigger so the flow is unambiguous.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 84c1557e-8ebd-4bc8-a343-d0ada7aa2857
📒 Files selected for processing (4)
docs/docs/experimentation/analyse-an-experiment.mddocs/docs/experimentation/connect-a-warehouse.mddocs/docs/experimentation/create-an-experiment.mddocs/docs/experimentation/example-paypal-button.md
💤 Files with no reviewable changes (1)
- docs/docs/experimentation/analyse-an-experiment.md
Holmus
left a comment
Holmus
left a comment
There was a problem hiding this comment.
Mostly small comments, looks great overall. The order of doing things and structure makes a lot of sense
| @@ -4,76 +4,111 @@ sidebar_label: Experimentation (A/B Testing) | |||
| sidebar_position: 4 | |||
There was a problem hiding this comment.
The layout is a bit odd from the top-level.
We have Experimentation (A/B Testing) and Experimentation (Beta).
Not sure what a good solution is but maybe we rename this Experimentation (external) or Experimentation (integration). Alternatively, we can bring it all into a shared category called Experimentation and then put the (A/B testing) one as a page in that category.
There was a problem hiding this comment.
Yeah, it's a bit conflicting and outdated the previous one. I'll rename it "A/B Testing with integrations" to dissociate from experimentation entirely
| Combining these two concepts allows you to deliver seamless A/B test. | ||
|
|
||
| We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't | ||
| integrate with the platform you are using, you can still manually send the test data to the downstream platform |
There was a problem hiding this comment.
| integrate with the platform you are using, you can still manually send the test data to the downstream platform | |
| integrate with the platform you are using, you can still send test data to the downstream platform |
| [multivariate flag](/managing-flags/core-management), collect events from your application into a managed data | ||
| warehouse, and read the results with a built-in Bayesian statistics engine. | ||
|
|
||
| Experiments are scoped to an **environment**: the warehouse connection, metrics and experiments all live at the |
There was a problem hiding this comment.
Non-blocking: Should we link to the data model(?)
| ## Set up and run an experiment | ||
|
|
||
| 1. **[Connect a warehouse](/experimentation/connect-a-warehouse)**: enable the managed Flagsmith warehouse for your | ||
| environment. Done once, in a few clicks. |
There was a problem hiding this comment.
| environment. Done once, in a few clicks. | |
| environment. |
|
|
||
| Prefer to see it end to end? Follow the [PayPal button example](/experimentation/example-paypal-button). | ||
|
|
||
| Unfamiliar with a term? The [Statistics Glossary](/experimentation/statistics) explains every concept in plain language. |
There was a problem hiding this comment.
| Unfamiliar with a term? The [Statistics Glossary](/experimentation/statistics) explains every concept in plain language. | |
| Unfamiliar with a term? See the [Statistics Glossary](/experimentation/statistics). |
|
|
||
| ::: | ||
|
|
||
| A running experiment can be **ended** at any time. Its results are frozen as final, and this cannot be undone. |
There was a problem hiding this comment.
What does "and this cannot be undone" mean?
You can not delete the experiment and the results afterwards?
|
|
||
| ## Instrument your application | ||
|
|
||
| Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Enable it with |
There was a problem hiding this comment.
I think this information should be on the first page as well (Experimentation (Beta)). It would be slightly annoying to go through the steps to find out that your SDK is not supported.
|
|
||
| `trackEvent` / `track_event` sends a named event, optionally with a numeric `value`, traits and metadata. | ||
|
|
||
| Any event whose name matches a metric's event name feeds that metric. Conversions are attributed post-exposure: an |
There was a problem hiding this comment.
Case sensitive?
Error handling / expected response?
| While a variation has fewer than 50 identities (or fewer than 5 conversions, for occurrence metrics), it shows | ||
| **Collecting data**. Keep the experiment running. | ||
|
|
||
| If a **sample ratio mismatch** warning appears, traffic didn't split the way you configured. Don't act on the results. |
| ::: | ||
|
|
||
| Flagsmith's statistics engine answers three questions about an experiment: **Am I winning?** (is the variant better than | ||
| control), **by how much?** (the lift), and **can I trust it?** (is the difference real, and was traffic split fairly). |
There was a problem hiding this comment.
would this read nicer as a bulleted list maybe
- am i winning
- by how much
- can i trust it
4 participants
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Documents the Experimentation feature end to end, as a new top-level sidebar category on docs.flagsmith.com:
Create an Experiment, Run an Experiment (JavaScript and Python instrumentation), and Analyse an Experiment.
A/B testing tutorial (branches on variation names via
useExperiment).How did you test this code?
npm run buildindocs/passes with no broken links in the new pages.make lint).