A useful test answers a specific customer question. It needs a change you can compare, enough relevant traffic and a measure that matches the decision. A testing tool cannot supply those ingredients on its own.
Shopify Rollouts can schedule a permanent Launch, a temporary Event or an Experiment that compares a treatment with a control. Rollouts are available on Basic or higher; Experiments require Grow or higher. Check the current requirements, permissions and supported changes in your store before planning work.
Choose the customer problem before the variant
Start with evidence of friction: people cannot identify the right product, delivery information is unclear or a mobile journey is difficult to use. Separate a visible usability problem from a hypothesis about commercial impact.
The Czarina delivery record describes navigation and matching-print discovery changes. It is useful context for a clear customer task. It does not report a controlled experiment or verified conversion uplift.
For your own store, write one sentence: changing this part of the journey may help this group of shoppers complete this task. Describe the original and proposed experience. Avoid combining several unrelated changes if you want to understand which change mattered.
Check what Rollouts can measure
Shopify's analytics documentation distinguishes Launch and Experiment reporting. A report available for a Launch is not necessarily available for an Experiment.
- Theme Experiments: current metrics include conversion rate, bounce rate, reached-checkout rate and add-to-cart rate.
- Checkout and account Experiments: the listed metric is checkout conversion rate.
- Coverage: results count consenting visitors and cannot be filtered by device or country in the Rollouts report. Updates arrive after visits rather than in real time.
- Variant identity: Shopify states that themes and apps cannot read a visitor's experiment group. Do not assume a theme snippet can expose that group to an external analytics tool.
If your decision depends on variant-level margin, returns or revenue, establish a supported measurement method before commissioning the test. A separate store-wide revenue chart does not prove which variant caused a change. If reliable assignment data is unavailable, record the limitation rather than inventing attribution.
Write a test brief your team can actually use
- Decision: what will you keep, stop or investigate when the test ends?
- Audience: which eligible visitors and markets should enter? Check exclusions, consent coverage and competing campaigns.
- Primary measure: define the exact event and denominator. A checkout completion rate and an overall store purchase rate answer different questions.
- Guardrails: identify errors, accessibility problems, incorrect prices and operational issues that require a stop, regardless of the headline metric.
- Duration and sample: use the baseline rate, traffic and smallest worthwhile effect to plan the sample. Include the business cycle and agreed stopping rules.
- Release controls: name the approver, review the implementation and agree how to stop or reverse the change.
Low traffic can make a small effect impractical to measure. A result may be inconclusive. Do not promise two or three statistically useful tests each month without checking the available sample and the work required.
A conversion change is not a revenue forecast
A relative percentage change differs from a percentage-point change. Neither can be translated into incremental revenue by multiplying annual sales alone. You also need eligible traffic, the baseline rate, order value, exposure, uncertainty and the costs associated with the change.
Keep observations, estimates and confirmed outcomes separate. Report a test's date range, sample, measurement limits and uncertainty alongside its result. A modelled opportunity is not money already earned, and a short-lived difference is not automatically a repeatable improvement.
Check the surface and the store first
Rollouts have specific limitations: they do not support vintage themes or Liquid-template changes, and theme/checkout configuration experiments do not apply to headless and custom-storefront checkouts. A proposed test must be supported by both the tool and the store setup.
A Shopify conversion audit can help define the question and measurement needs. Mobile performance work may be a better starting point when a journey fails before a comparison is meaningful.
Any implementation needs agreed scope, access, approvals and release checks. Ongoing Shopify improvement has agreed capacity and timing; it does not imply automatic checkout changes, continuous detection of every issue or guaranteed revenue gains. Checkout and payment work needs separate explicit authority.
Sources and review
Reviewed 2 November 2026. Feature availability and report definitions can change; verify them in your own Shopify admin.