Features
Experiments
Test a change on part of your traffic and find out whether it worked: lift and confidence for every metric you care about, with one line of code.
How it works
A feature flag decides who sees what. An experiment decides whether it worked. The flag splits visitors between variants at random, and each visitor keeps the same one on every visit, in the browser and on your server. The experiment is the report on top: it compares each variant with the control on the metrics you pick.
So there is nothing to instrument for the experiment itself. Reading the flag counts the visitor, and your metrics come from events Flowgrid already collects.
Get started
You need the Flowgrid SDK installed. If you haven't, start with the SDK Quickstart.
1. Create the experiment
In the dashboard, open Product Analytics → Experiments → New experiment. Pick an A/B test flag, or choose New A/B test flag and name its variants, current experience first, like control, urgent. A new flag is created switched off, with visitors split evenly. Then pick the control, what decides the test, and how long to run it.
2. Read the variant in your code
Read the flag where the change renders. While it's off, missing, or Flowgrid can't be reached, you get null or your fallback, so your current experience keeps running.
1import { useFlagVariant } from 'flowgrid-sdk/react';
2
3function CheckoutButton() {
4 const variant = useFlagVariant('checkout-copy');
5 return <button>{variant === 'urgent' ? 'Only 3 left, buy now' : 'Buy now'}</button>;
6}Who gets counted
A visitor enters the experiment the first time their variant is read. In the browser that's automatic. On your server it needs flowgrid-sdk 2.2 or later and the visitor passed in, as visitorIdFromCookies does above. Without the visitor, a server read picks the same variant but counts nobody.
Try each variant
Force a variant in your own browser to check it renders. Open the page with ?fg_flags=checkout-copy:urgent on localhost, or call flags.override({ 'checkout-copy': 'urgent' }). It lasts for the tab and is never counted, so testing doesn't add you to the results. On your public website the URL form only works if you allow it with flags: { allowUrlOverrides: true }.
3. Start it
Once that code is live, press Start experiment. Starting turns on measurement for the flag, and can switch the flag on for you. From then on the control is fixed: results already measure against it.
Metrics
Every experiment has one primary metric, the result that decides it. Secondary metrics explain the result without deciding it. Guardrails must not get worse; feature errors and page speed are added for you.
| Metric | Measures | What your code adds |
|---|---|---|
| Conversion | Share of visitors who did an event at least once | An event you already send, or capture() |
| Times per visitor | How often visitors did an event | Same |
| Revenue per visitor | Money paid, in one currency | A connected payment provider |
| Sessions per visitor | How often they came back | Nothing |
| 7-day return | Share who returned a week or more after first seeing it | Nothing |
| Feature errors | Share of visitors the flagged code threw for | flags.run() or <Flag>, see Feature Flags |
| Page speed | LCP, INP, CLS or TTFB per visitor | Nothing |
Only what a visitor does after they first see the experiment counts, and every rate is out of the visitors in it. For something Flowgrid doesn't see on its own, send an event and it appears in every metric picker:
1fg.capture('checkout_completed', { total: 49 });Reading the results
Each metric shows the control, each variant, and the lift: how much higher or lower the variant is, relative to the control. The bar beside it is the range the true lift is likely in, at the confidence you chose (95% unless you changed it).
- Better or Worse: the whole range is on one side of zero.
- No clear difference: the range crosses zero. It may still move as more visitors arrive.
- Not enough data: fewer than 30 visitors in an arm, or nothing to compare against.
Safe to check any time
New experiments use sequential statistics. A classic interval is only right if you look once, at the end; checking it every day makes a fluke look like a winner far more often than the confidence promises. A sequential interval stays right however often you look, so a verdict you see on day three can be trusted. The price is a wider range early on, so a real winner can take a little longer to call.
You can still choose fixed-horizonunder Statistics when you create an experiment. Read it once, when it's ready. Experiments created before sequential statistics existed keep fixed-horizon, so their verdicts don't change under you. The setting can't change once an experiment starts.
You'll be told when it's ready
Set a duration (days, a number of visitors, or both). When it's met, the experiment says Ready to reviewand its owner gets a notification and an email with where the primary metric stands. The list shows each experiment's latest lift, dated.
Health
Before trusting a result, check the split. The health card shows who entered each variant, day by day, and checks the split against the one configured. If they don't match, something is assigning or counting visitors unevenly (a cached page serving one variant, a redirect that drops the SDK) and the results aren't reliable until it's fixed.
Only visitors a percentage ruleplaced in a variant are compared. Visitors outside the rollout aren't the control group, so they're left out.
Deciding
Record Ship a variant, Ship none or Defer, with a note. Your name and the date are saved with it, and the history keeps every change. Recording a decision never changes the flag.
After shipping a variant, Roll out to everyone switches the flag on and gives every visitor that variant. Then remove the losing branch from your code, and the flag shows as ready to clean up. Editing the analysis of a decided experiment reopens it.
With the agent
Ask the Flowgrid agent to "test two headlines on pricing". It proposes the experiment, and the flag if one is needed, as a card you approve. It's always created as a draft. The agent also reads results (lift, confidence, health) and can explain them, but deciding is always yours. With a repository connected, Ship it with Coder adds the variant read to your code as a pull request.
Running in another tool
Already testing with Optimizely, VWO, LaunchDarkly or your own code? Keep it. Choose Running in another tool when you create the experiment, name it the way your tool does, and list its variants. Then send one event whenever a visitor sees a variant:
1fg.capture("experiment_started", { experiment: "checkout-test", variant: "b" });Already forwarding Segment's Experiment Viewedevent? That works as it is. Your tool keeps deciding who sees what, so there's no split to check and nothing to roll out here: when you ship a winner, ship it in your tool.
Limits
Experiments are on every plan. The Free plan runs up to 5 at once on a website; every paid plan runs as many as you like. Drafts and finished experiments don't count. Results cover up to 90 days.
Upgrading from fg.experiments
The older experiments API, where experiments were defined in code and conversions sent by hand, was removed in flowgrid-sdk 2.2. Experiments made with it no longer collect data. To keep testing, create an A/B test flag for each one, then swap the calls:
| Before | Now |
|---|---|
fg.experiments.variant('id') | flags.variant('key') |
useExperiment('id') | useFlagVariant('key') |
<Experiment> and <Variant> | useFlagVariant('key') and a branch |
getVariant() from flowgrid-sdk/next | getServerFlags() from flowgrid-sdk/server |
convert() / fg.experiment() | Nothing. Pick the event as a metric |
data-no-experiments | Nothing. Remove the attribute |
Conversions don't need a call any more: any event Flowgrid collects can be an experiment's metric, counted only after each visitor first saw the variant.