How to compare a/b test json data across experiment variants
- Step 1Export variant configurations as JSON — Export the JSON configuration for each experiment variant from your A/B testing platform or feature flag system. This could be the rendered page configuration, the feature flag state object, or the API response each variant produces.
- Step 2Diff control against variant — Paste the control (variant A) configuration in the left panel and the treatment (variant B) in the right panel. The diff shows every difference between the two variants.
- Step 3Verify only intended differences exist — Review the diff. Intended differences should be limited to exactly the fields the experiment is testing — button color, price value, UI copy. Any additional differences are unintended confounds.
- Step 4Document the expected variant delta — Copy the diff output as the documented expected delta for the experiment. Include this in the experiment brief as the 'configuration change' section so reviewers can verify the experiment setup.
Frequently asked questions
What is a confound in an A/B test configuration?+
A confound is an unintended difference between variants that could influence user behavior independently of the treatment being tested. For example, if variant B has both a different button color and a different response time (due to an unintended server-side difference), you cannot attribute performance differences to button color alone. The JSON diff reveals these confounds before the experiment runs.
How do I compare the API responses that each variant produces?+
Make API calls with the experiment flag enabled and disabled (or with variant headers), copy the response bodies, and diff them here. The diff shows which data fields change between variants — confirming the experiment's data-layer changes are limited to the intended modifications.
Is the experiment configuration data transmitted to JAD Apps?+
No. Diffing runs entirely in your browser. A/B test configurations and feature flag states are never transmitted to JAD Apps servers.
Privacy first
Conversion runs locally in your browser. No file is uploaded — only metadata counters are saved for signed-in dashboard stats.