Feature flag rollout, failure and rollback
How three local SDKs behaved on one flag, and what a vendor trial still has to cover.
Local SDK check, no vendor account
On 2026-11-19 a script in research/feature-flag-rollback evaluated one flag, checkout-redesign, for 1,000 customer ids and 5 staff ids. GrowthBook 2.2.0 used an in-memory payload. LaunchDarkly Node server SDK 9.17.0 read flags/rollout.json and flags/killed.json, with events disabled and a dummy SDK key. Unleash client 6.12.2 loaded a bootstrap document and did not start its poller. The script repeats. The second pass turned on the same ids as the first.
| SDK | Customers on at 10% | Staff forced on | On after the kill | Failure behavior in this run |
|---|---|---|---|---|
| GrowthBook 2.2.0 | 106 of 1,000. Same 106 on the second pass. | 5 of 5 | 0 of 1,005 | Empty client value was null, source unknownFeature. getFeatureValue returned the fallback false. init against 127.0.0.1:9 returned success false and fetch failed, then the fallback payload turned on the same 106 ids. |
| LaunchDarkly 9.17.0 | 93 of 1,000. Reason FALLTHROUGH. Same 93 on the second pass. | 5 of 5. Reason RULE_MATCH. | 0 of 1,005. Reason OFF. | Offline mode returned false with reason ERROR and errorKind CLIENT_NOT_READY. A missing flag file failed init with ENOENT and returned the same fallback. |
| Unleash 6.12.2 | 103 of 1,000. Same 103 on the second pass. | 5 of 5 | 0 of 1,005, even when the caller passed fallback true. | Before start, the caller fallback was used. An unknown flag name still used that fallback. A disabled flag did not. |
The three SDKs did not pick the same customers. GrowthBook coverage 0.1, a LaunchDarkly fallthrough weighted 10,000 of 100,000, and an Unleash flexibleRollout of 10 are all about 10 percent, and the counts were 106, 93 and 103. Stickiness held inside each SDK. It did not carry across SDKs.
GrowthBook treated the rollout and the experiment differently. The rollout fired the tracking callback 0 times and marked 106 results as force and 894 as defaultValue. A separate 50/50 rule marked all 1,000 results as experiment, fired the callback 1,000 times, and put 492 customers on the treatment. A flag that only controls access did not produce an exposure record in this SDK.
LaunchDarkly's offline-mode page says a server SDK in offline mode returns the fallback value passed into variation, not the flag's own default rule. The value in this run matches that. The reason code on this SDK version was CLIENT_NOT_READY. Their file-data page says a missing or invalid file means the SDK uses none of the file data. The missing-file client did that.
Unleash documents the need for synchronization or bootstrap data. The cold client used the caller's fallback. In a separate local refresh test, the poller got a connection-refused error. The client still reported synchronized and kept 103 of 1,000 customers on from the bootstrap data.
Rollout and experiment columns from the vendor pages
The catalog pairs LaunchDarkly with Statsig, PostHog with GrowthBook, and Flagsmith with Unleash. This pass used LaunchDarkly and Statsig plus Unleash, GrowthBook and Flagsmith, so the shortlist covers a managed rollout product, an experimentation meter, and self-hosted options. The rows below are from the vendor pages. Only GrowthBook, LaunchDarkly and Unleash were executed locally.
| Product | Rollout | Experiment | Where it runs |
|---|---|---|---|
| LaunchDarkly | Percentage rollouts and segments are on Developer, Foundation and Enterprise. The pricing page says seats are not billed. | A/B/n, bandits, Bayesian and frequentist analysis, and warehouse-native metrics are listed on all three plans. Experiment approvals are Enterprise. | Hosted. File data and offline mode are SDK options for tests and for networks that cannot call LaunchDarkly. They do not sync the file for you. |
| Statsig | Percentage rollouts. The pricing FAQ says a 0% or 100% rollout does not create a metered exposure by default. A partial rollout with metric lifts does. | Bayesian and frequentist methods are on the pricing page. Warehouse-native deployment is listed under Enterprise. | The flag service in the docs is Statsig's. Warehouse native moves the analysis queries to your warehouse. This pass did not run a Statsig SDK. |
| Unleash | Open source includes activation strategies and stickiness. The local run used flexibleRollout. | The Enterprise pricing table lists unlimited experiments and A/B/n variants. The open-source comparison does not include change requests, custom roles or advanced audit. | Open source is free to run and is documented at 1 project and 2 environments. Paid plans are cloud, self-hosted or hybrid. Self-hosted pay-as-you-go has a 5-seat minimum. |
| GrowthBook | Percentage rollout and an instant kill switch are on the cloud compare table, including Starter. Safe rollout with auto-rollback is Pro. Ramp schedules are Enterprise. | Starter includes unlimited experiments. Sequential testing, CUPED and multi-arm bandits are Pro. Interleaving, contextual bandits and holdouts are Enterprise. | Open source is self-hosted. The pricing FAQ says the free self-hosted plan has unlimited users. Cloud Starter is free for up to 3 users. |
| Flagsmith | The free plan includes flags. Start-Up adds scheduled flags. | A/B and multivariate tests start on Start-Up. Scale-Up adds a bring-your-own experimentation warehouse. | Enterprise copy lists cloud, private cloud and air-gapped hosting. The Scale-Up feature list does not include on-prem. |
Statsig's testing guide says local mode makes no network calls and returns defaults, and that overrideGate is for tests. The Go server SDK also documents bootstrap values for a period when the server cannot reach Statsig. Those are documented behaviors. They are not results from this fixture.
GrowthBook's Node page says a failed or timed-out init can use a fallback payload, and that the result still reports success false. The local init did that: success was false, source was error, and the fallback assigned the same 106 customers as the direct payload.
What the published rates charge for this sample
The figures below are arithmetic on the rates published on 2026-11-19. They are not invoices. The sample is 8 service connections, 25,000 client-side monthly active users, 12 people who can log in, 8 million events and 4 million flag API requests. LaunchDarkly's own page uses a smaller example: 10 connections and 2,000 client MAU come to $58.33 a month on yearly billing. That matches 5 connections over the free 5 at $10, plus 1,000 MAU over the free 1,000 at $8.33.
| Product | Published meter | This sample | Seat note |
|---|---|---|---|
| LaunchDarkly Foundation | 5 service connections and 1,000 client MAU free, then $10 per connection and $8.33 per 1,000 MAU on the yearly card. Monthly card rates are $12 and $10. The compare table also prints client MAU as $10 per 1,000 a month. | $229.92 a month on the yearly card: 3 times $10 plus 24 times $8.33. The monthly card is $276: 3 times $12 plus 24 times $10. | The pricing page says there is no per-seat price. Developer does not bill overages. The page says to upgrade when a limit is hit. This sample is past both free limits. |
| Statsig Pro | $150 a month, 5 million metered events included, then $0.05 per 1,000. Developer is free for 2 million metered events. | $300 if all 8 million events are metered: $150 plus $150 of overage. A 0% or 100% flag check is not a metered exposure unless metric lifts are on. | The pricing page shows no per-seat price. SSO, role-based access and teams are listed on Enterprise, which is a quote. Event or experiment contracts are also Enterprise. |
| Unleash pay as you go | $75 per seat a month by credit card. Cloud includes 53 million API requests, then $5 per million. Client MAU and service connections are listed as unlimited. | 12 seats are $900. 4 million API requests stay inside 53 million, so the request overage is $0. | Self-hosted pay-as-you-go has a 5-seat minimum, $375, and 12 seats are still $900. Open source has no seat fee and the docs cap it at 1 project and 2 environments. |
| GrowthBook Pro | $40 per seat a month, up to 30 users and 3 projects. Managed warehouse is 2 million events, then $30 per million. CDN is 2 million requests, then $10 per million. | 12 seats are $480. Sending all 8 million events to the managed warehouse adds $180, for $660. A self-hosted open-source install does not use that warehouse meter. | A seat is one login. Starter is free up to 3 users, so 12 people do not fit. The FAQ says self-hosted open source has unlimited users. |
| Flagsmith Scale-Up | Free is 50,000 requests and 1 member. Start-Up is 1 million requests and 3 members. Scale-Up is 5 million or more requests, 5 members included and up to 20. The page shows extra members at $60 and $50 per member a month beside the monthly and yearly toggle. | 4 million requests fit the Scale-Up request line. 12 members are 7 extra seats. At $50 that is $350 before the plan sticker. At $60 it is $420. | This pass did not get a plan sticker price out of the page text. The project catalog checked the same day records Start-Up at $45 a month or $40 yearly and Scale-Up at $300 or $250. Treat those stickers as a prior read, and re-check the toggle before budgeting. |
Statsig defines a metered event as an exposure on a partial rollout with metric lifts, a logged or imported event, an ingested warehouse metric, or a custom metric. Client SDKs dedupe the same user and feature for 60 minutes. Server SDKs dedupe for 1 minute. An extra event dimension is another logged event. Past the Developer allowance, the FAQ says targeting keeps working and metric-lift math stops until you move to Pro.
Flagsmith counts /flags, /identities, /traits and server-side /environment-document. The FAQ says a first paid overage inside 200 percent of the plan gets 30 days before a bill, and that Start-Up overage after that is $7 per 100,000 requests. The free plan gives 7 days to upgrade. Flags keep being served during that window.
LaunchDarkly counts a service connection as one server-side SDK connected to one environment, averaged over the month. Every replica, worker, and dev or staging environment counts. Client MAU is a unique client-side user or device. Those are separate meters from the 8 million product events in the sample.
Who can kill a flag, and what the audit log keeps
The fixture changed flags by loading a new file or payload. It did not sign in as a reviewer, and it did not download an audit log. The checks below are what the docs say a buyer can ask to see, and what this pass could not operate.
| Product | What a change leaves behind | Who can publish it | Export this pass could confirm from docs |
|---|---|---|---|
| LaunchDarkly | Change history records flag and account changes. GET /api/v2/auditlog filters by time, text, resource, member or token. The detail call adds previousVersion, currentVersion, and a JSON-patch delta when that is how the change was made. Limit per call is 1 to 20. | Foundation includes approval requests with 1 to 5 reviewers. Making approvals mandatory is Enterprise. Custom roles are Enterprise. A role with a wildcard action on flags includes bypassRequiredApproval. | Audit API is documented. Data Export of raw flag requests and events is a separate add-on for Foundation or Enterprise. Developer change history is 30 days. Foundation account changes are 30 days and other change history is otherwise unlimited. |
| Statsig | The pricing FAQ describes which checks become metered events. Its public docs don't describe an audit-log download. | SSO, role-based access and teams are Enterprise features on the pricing page. | Not confirmed. A trial still has to export one flag change and show the actor, the old value and the new value. |
| Unleash | The event log can be filtered and exported as CSV or JSON. Admin access is required to open it. Enterprise audit entries can include the IP address and are kept up to 2 years. Open source is described as a basic audit log. | Change requests and custom project roles are Enterprise. Open source has basic roles. Login history is Enterprise and is kept 336 hours. | Flag export is POST /api/admin/features-batch/export and is listed for both open source and Enterprise. The events API returns the last 100 events when called without a project. |
| GrowthBook | The cloud compare table includes flag change history and an experiment history on Starter, Pro and Enterprise. Exportable audit logs are marked absent until Enterprise. | Starter roles are admin only. Pro has basic roles. Custom roles, approval workflows and SCIM are Enterprise. | The pricing page names exportable audit logs as an Enterprise item. This pass did not download one. |
| Flagsmith | Audit log docs say every admin action is logged, and a webhook can POST the author, the project and the log line. Change-request docs say publishing a request writes audit entries and does not send the publish email. | Change requests, roles and audit logs are on the Scale-Up list, and the docs limit change requests to Scale-Up and Enterprise. Identity overrides apply immediately and skip the change request. | The webhook payload is documented. A trial still has to prove an identity override shows up in the audit log even though it skipped approval. |
LaunchDarkly says comments submitted with an update appear in webhooks and in the audit log. The list endpoint is a summary. The single-entry endpoint is the one that carries the previous and current versions.
Unleash open source can export flag definitions. That export is the flag configuration, not the event log. The event log is the audit trail, and the advanced fields are Enterprise.
Kill-switch delay and the approval rule
The fixture answers how three SDKs behave with a local payload. It does not answer how fast a dashboard kill reaches a running app, or whether an approval rule actually stops the person on call.
Time a kill from the real console to a process that already has flags cached. Record the poll or stream interval, and record what the app serves while the flag service is unreachable. This run only swapped a local document. LaunchDarkly offline mode with no cache returned the code fallback and CLIENT_NOT_READY. A client SDK that had stored a variation earlier was not tested. The docs say client SDKs use the last stored variation, then the code fallback.
Repeat the kill with the role your on-call engineer will actually have. On LaunchDarkly, confirm that a mandatory approval cannot be skipped except by a role that holds bypassRequiredApproval. On Flagsmith, change an identity override and confirm whether it skips the change request, which the docs say it does. On Unleash open source, confirm there is no change request at all.
Download the audit record for that kill. For LaunchDarkly, use the audit entry id and check previousVersion, the member and the comment. For Unleash, export the event log and see whether the IP field is present on the plan you would buy. For GrowthBook, confirm whether Starter or Pro can export the log or only view it. For Statsig, the pages read here did not show an audit export, so the trial has to produce one or the gap stands.
Price the sample in the vendor's unit, not in a single event count. LaunchDarkly needs the replica and environment count plus client MAU. Statsig needs a split between partial-rollout exposures and ordinary analytics events, after the 60-minute client and 1-minute server dedupe. GrowthBook cloud needs a seat count plus a decision about whose warehouse runs the queries. Flagsmith needs flag requests, not product events, and a member count against the 1, 3 or 5 included seats.
Do not copy a 10 percent cohort from one product to another. In this fixture the same 1,000 ids produced three different sets of 106, 93 and 103 customers.
Sources
- LaunchDarkly: reading flags from a file
- LaunchDarkly: offline mode
- Unleash Node SDK
- GrowthBook Node.js SDK
- LaunchDarkly pricing
- Statsig pricing
- Statsig: testing gates and local mode
- Statsig Go server SDK: local mode and bootstrap
- Unleash pricing
- Unleash open source and Enterprise comparison
- GrowthBook pricing
- Flagsmith pricing
- LaunchDarkly change history
- LaunchDarkly list audit log entries
- LaunchDarkly get audit log entry
- LaunchDarkly Data Export destinations
- Unleash events
- Unleash export feature flags
- Flagsmith audit logs
- Flagsmith change requests