App Store A/B testing has a different name on each store, and different limits behind it.
Apple calls it product page optimization. Google Play calls it store listing experiments. Both show alternate art to a slice of your store traffic and then tell you which version performed better. The limits are where they part company. Apple runs one test at a time, with up to three treatments, for up to 90 days. Google runs one default graphics experiment or up to five localized experiments at once, each with up to two variants, and stops any experiment by itself after six months. Every figure and screen name below comes from Apple's and Google's own documentation, read on 10 October 2026. If the basics are not in place yet, do app store optimisation first, because a test compares two versions of your work rather than improving either one.
What follows is how many versions each store will run, what each one declares a winner on, and what happens to a live test when you ship a build. That last answer is a gap in both sets of documentation, and it is the one people assume they already know.
Start hereThe short version
Two features, two vocabularies, and only one lets you pick the metric.
An App Store test reports conversion rate. You do not choose that, and the place to read a running test is App Analytics in App Store Connect, which is not the screen where you built it. For each treatment you get impressions, conversion rate, percent improvement and a confidence level, measured against a baseline. The baseline starts as your original page and you can change it to compare treatments with each other.
A Google Play experiment makes you commit to a target metric before it starts. You choose unique user install clicks, unique user open clicks or unique user pre-registration clicks, and the result is explained with the confidence interval and the minimum detectable effect you set. Set the minimum detectable effect high and small real differences come back as a draw. That setting is the one people skip and then argue about afterwards.
How many versions can you run at once, and for how long?
Apple allows one test at a time, and a test can include up to three treatments with alternate app icons, screenshots and app previews. You then choose a traffic proportion, which is the percentage of people picked at random to see a treatment instead of your original page. Apple's own example: allocate 40% of your traffic with two treatments and each treatment gets 20%, leaving your original page 60%. A person sees the same treatment every time they visit for the whole run, and treatments are served to people on iOS and iPadOS 15 or later.
A test runs for 90 days, or until you stop it inside that window. Before it starts there is an optional step worth doing: pick the improvement in conversion rate you are hoping for, and Apple uses your existing performance data, such as daily impressions and new downloads, to estimate how long the test needs and how many impressions it would take to reach an outcome with at least 90% confidence. If that estimate comes back over 90 days, the test as specified cannot finish, and Apple suggests fewer treatments or more traffic. That is the honest answer to whether this is worth doing on a small app.
Google splits the same idea in two. A default graphics experiment covers your icon, feature graphic and screenshots in your default store listing language. A localized experiment covers the icon, feature graphic, screenshots and your descriptions in up to five languages, and its variants are only shown to people viewing your listing in those languages. Per app you can run one default graphics experiment, or up to five localized ones, at the same time. Each experiment tests up to two variants against your current listing, and the audience percentage you set is split equally between them.
One exclusion catches people with a part-translated listing. If you have added localized graphics in a language, people viewing your app in that language are left out of your default graphics experiments, even when the asset you are testing is the icon. Google sets up all of this under Grow users, Store presence, Store listings, in the Experiment column. There is no minimum duration. There is an automatic end: experiments stop after six months, no further data is collected, no variant is applied, and traffic goes back to your current listing.
Apple, product page optimization, read 10 October 2026
Check yours
What each store will let you test
Tick what you want to try. The answers use each store's own words for its own feature.
Nothing ticked. Pick one asset, not three. Both stores tell you to change one thing at a time, for the same reason: a treatment with two changes in it cannot tell you which change did the work.
What does each store declare the winner on?
Apple's answer is conversion rate. Open App Analytics in App Store Connect, select the test, and the dashboard tells you one of three things: it needs more data, a treatment is performing better than the baseline, or a treatment is performing worse. Alongside that you get impressions, conversion rate, percent improvement and the confidence level for each treatment. Apple's advice on when to act is specific. Wait until at least one treatment has been declared better or worse than the baseline with at least 90% confidence before you apply anything or stop the test.
Google's answer is whichever metric you committed to at setup. Unique user install clicks counts the users who installed your app and did not have it installed on any other device at the time. Unique user open clicks counts installed users who opened the app. Unique user pre-registration clicks counts users who pre-registered. The experiment creation page estimates how long, and how many of those events, you will need for a statistically significant result.
The Play result is phrased more cautiously than people expect. Outcomes include a recommendation to apply a variant, a draw when the gap is smaller than the minimum detectable effect you set, and More data needed, which simply means come back later. Google says it will tell you when a result is less than a week away. If more than one variant beat your current listing, you decide between them, and if nothing did, you click Keep current listing.
Do not put the two percentages side by side in the same report. One is a conversion rate off impressions and the other counts people who installed, deduplicated across their devices. They answer different questions and they move by different amounts. If the goal is volume rather than a better rate, treat this as one lever among several and read more downloads first. A few points of conversion on a page nobody reaches is still nobody.
Google Play Console Help, run A/B tests on your store listing, read 10 October 2026
Does shipping a new build reset a running test?
Neither store says so, and that is worth saying plainly, because the belief that a release wipes your data is common and neither Apple's page nor Google's documents it. What Apple does document is three ways a test ends. It reaches 90 days. You stop it. Or you apply a treatment while it is still running, which Apple states will cause your test to end. Apple also says you cannot change a test once it has started, so a mistake in a treatment is a new test rather than an edit.
Builds do come into it for exactly one asset. If your treatments include alternate app icons, every variant of the icon has to be in the binary of your published app, and if you apply an icon treatment you have to include that icon when you submit a new app version. Apple is explicit that tests without any alternate app icons can be submitted for review independent of a new app version. So a screenshot test is a metadata exercise and an icon test is a code change plus a release. Plan them differently.
Both stores put a review in front of the test. Apple requires any new metadata you want in your treatments to be approved by App Review before you can publish the test. On Google Play, saving your asset variants leaves you with the option to go to the publishing overview page and submit your changes for review. Google's only documented automatic ending is the six month stop. Nothing on its page ties an experiment to a release.
One more precondition that gets skipped. Google frames experiments as something you do for published apps, testing variants against your current version using install and open data, and Apple's treatments are alternate versions of a live App Store product page. An app sitting on TestFlight or an internal testing track has no traffic to split. Before any of this, the originals have to be worth comparing, so get the app screenshots right before you spend 90 days measuring two versions of a weak set.
Apple, configuring and applying a test, read 10 October 2026
What should you test first, and what is not worth a test?
One asset at a time. Google recommends it in its own setup instructions so that you can be as certain as possible about what caused a change. Apple makes the same point from the other side: think about how many elements you change in a treatment, and remember that the more treatments a test carries, the longer it may take to reach a conclusive result. Three treatments sounds generous until you work out what it does to the traffic each one gets.
Then check whether you have the volume. Both consoles will tell you before you commit. Apple estimates the duration and the impressions needed for your chosen improvement, and Google estimates the time and the number of installs, opens or pre-registrations required for a significant result. If the estimate is longer than the window the store gives you, the answer is not to run it and hope. It is to raise the audience percentage, cut a treatment, or accept that your traffic cannot resolve a difference this small yet.
Localization is the quiet multiplier. Apple warns that a test may take longer to reach meaningful results depending on the localizations you select. On Google, a localized experiment only serves people reading your listing in the languages you chose, so spreading a modest audience across five languages makes every arm slower. Pick your largest market and test there. Order matters too: screenshots and previews first, because they are metadata, and the icon last, because it needs the variants in your binary.
Some things are not tests at all. You cannot test App Store text this way, because product page optimization covers icons, screenshots and app previews. If what you want is a different page for a particular audience, campaign or feature rather than a measured comparison, use custom product pages instead. And remember that alternate assets are not confined to your product page: Apple says treatment screenshots, previews and icons may appear in search results and on the Today, Games and Apps tabs, and an alternate icon shows on the device of anyone who downloads while seeing it.
Google Play Console Help, experiment types, variants and autocompletion, read 10 October 2026
The same idea, with different rules on each store
| The question | App Store, product page optimization | Google Play, store listing experiments | Same on both |
|---|---|---|---|
| What the feature is called | Product page optimization, alternate versions are treatments | Store listing experiments, alternate versions are variants | No |
| Alternate versions in one test | Up to 3 treatments | Up to 2 variants | No |
| Tests running at the same time | One test at a time | One default graphics experiment, or up to five localized ones | No |
| How it ends without you | 90 days | 6 months, then traffic reverts to the current listing | No |
| Needs a review before it runs | Yes | Yes | Yes |
| You pick the metric yourself | No, it reports conversion rate | Yes, one of three click metrics | No |
| Text can be tested | No | Yes, in a localized experiment | No |
| Where you read the result | App Analytics in App Store Connect | The Experiments tab on the Store listings page | No |
Where this lands if a builder generated your app
Nothing in either feature changes because an AI wrote the code. The tests live in App Store Connect and Play Console, under your developer account, and you are the one who sets them up. On iOS you are also the one who submits. App Store Review Guideline 4.2.6 rejects apps created from a commercialized template or app generation service unless they are submitted directly by the provider of the app's content, which is you.
Newly is an AI app builder. You describe an app in plain English and it writes a real React Native and Expo project you own, with two-way GitHub sync. It uploads iOS builds to TestFlight, publishes to Google Play internal testing and builds a standalone Android APK. It is $25 a month and there is no free tier. The part that matters for a test is the icon rule: all variants of your icon have to be in the published binary, so an icon test means asking for the extra icon assets in the project and shipping a new build before the test can run.
The honest limit: generating an app does not run a test for you, and a build sitting on TestFlight or an internal testing track has no store traffic to split. Both features act on a live listing, so this work starts after your first release rather than before it. What a builder does shorten is the loop between a result and a shipped change, because the icon variants, the screenshot source screens and the release all come from one project you control.
Questions people ask about App Store A/B testing
Apple calls it product page optimization, and the alternate versions you build are called treatments. Google Play calls it store listing experiments, and the alternate versions are called variants. Google also notes that what used to be called a global experiment is now a default graphics experiment. Searching for the generic phrase finds articles. Searching for each store's own term finds the screen.
Ship the app first, then test the page
Neither store will test a listing that does not exist yet. Describe the app, get a real Expo project you own, publish it, and come back when there is traffic worth splitting.
Start building