Эксперименты

Эксперимент измеряет флаг. Каждый пользователь относится к варианту, который он увидел первым, события метрики учитываются только после этого момента, и каждый вариант сообщает вероятность быть лучшим с доверительным интервалом.

How an experiment measures

An experiment reads one flag and its variations; it owns no second assignment mechanism, so the split an SDK evaluates locally is the split the results are read against. Each time an SDK consults a flag (isFeatureEnabled, getVariant and their local forms) it records one exposure event, once per user, flag and response for the life of the client. getAllFlags consults no particular flag and records nothing.

{
  "event_name": "$feature_flag_called",
  "properties": {
    "$feature_flag": "new-checkout",
    "$feature_flag_response": "treatment"
  }
}

A user is attributed to the variation they were first exposed to, and a metric event counts only after that moment. A user seen in two variations cannot be attributed to either and is excluded; the results report how many were.

Reading a flag with a variant

GET /v1/flags answers with a variants map beside flags: the assigned variation key and value per flag. The SDKs expose it as getVariant, and as getVariantLocal over the cached snapshot.

curl "https://api.prodantix.com/v1/flags?distinct_id=u_8f3a" \
  -H "Authorization: Bearer $PRODANTIX_KEY"
# { "flags": { "new-checkout": true },
#   "variants": { "new-checkout": { "key": "treatment", "value": "b" } } }

const variant = await prodantix.getVariant('new-checkout');
if (variant?.key === 'treatment') {
  // ...
}

final variant = await prodantix.getVariant('new-checkout');
if (variant?['key'] == 'treatment') {
  // ...
}

variant = prodantix.get_variant("u_8f3a", "new-checkout")
if variant and variant["key"] == "treatment":
    ...

Creating and running one

An experiment names the flag, the control variation and a primary metric, with up to five secondary metrics and a minimum number of exposures per variation before a winner may be called (100 unless set). A metric is one of three shapes over one event, with the same filters the analytics DSL takes.

{ "kind": "conversion", "event": "checkout_completed" }
{ "kind": "count", "event": "item_viewed" }
{ "kind": "value", "event": "order_placed", "property": "amount" }

mutation Create($projectId: String!) {
  createExperiment(
    projectId: $projectId
    input: {
      key: "checkout-copy"
      name: "Checkout copy"
      flagKey: "new-checkout"
      controlVariation: "control"
      primaryMetric: "{\"kind\":\"conversion\",\"event\":\"checkout_completed\"}"
    }
  ) {
    key
    status
  }
}

A draft can be edited freely. startExperiment opens the data window and locks the flag: its split, targeting and rollout refuse edits until the experiment stops, while enabled stays free so a kill switch always works. stopExperiment closes the window for good; the restart is a new draft. concludeExperiment records a winner and, with ship: true, rewrites the flag so the winner takes every weight, through the same write the flags editor uses.

Reading the verdict

experimentResults returns, per metric and per variation, the rate (a conversion rate, or a mean for count and value metrics) with a 95% credible interval, the probability of being the best variation, and the lift against control with its own interval. A conversion metric is a Beta-Binomial posterior; a mean is a Normal posterior on the sample. The numbers come from a seeded sampler, so the same data always reads the same.

Each metric carries a guard. It reads collecting until every variation has reached the minimum exposures, and ready after. A winner is named only while the guard is ready and exactly one variation clears a probability of 0.95.