One‑Week Playbook for Kicking Off AI Auto‑Optimization

Conclusion: Treat the first week purely as a test week

This playbook aims to get RecoBoost’s AI auto‑optimization (auto_optimize) running safely in the first week after rollout, so by the end of that week you can clearly decide which logics to keep and which ones to drop.

There are three key points.

- See the first week as a learning period and expect volatile numbers
- Narrow down your candidate logics so MAB (multi‑armed bandit) + Thompson Sampling can do their job
- The fewer the data, the more you must stick to two rules: do not stop, do not spread too thin

Before you start: how auto‑optimization and MAB / Thompson Sampling behave

Ignore the math for now and just grasp how the system behaves.

• Auto‑optimization (auto_optimize): a mechanism that serves several recommendation logics in parallel and automatically shifts more traffic toward the ones delivering better results.
• MAB (multi‑armed bandit): a framework that keeps trying multiple options (recommendation logics) while automatically balancing exploration of new options and exploitation of the ones that look like winners.
• Thompson Sampling: one of the main algorithms used to implement MAB. It probabilistically allocates more traffic to logics that appear promising, while still giving some traffic to logics that are not yet well understood.

As a store operator, you only need to remember these two things:

- Traffic will gradually stop going to clearly weak logics
- But in the first few days, the system does not know which are weak yet, so allocations will fluctuate

With this in mind, you can decide the initial settings and how to run things during the first week.

Step‑by‑step: how to run AI auto‑optimization over one week

• 1. Check the app’s target pages in the Shopify admin: In the Shopify admin, go to Online Store → Themes → Customize and identify the pages where you have placed RecoBoost recommendation blocks (product detail, cart, etc.). Narrow down the positions where you will use auto‑optimization to one or two spots.
• 2. Decide which RecoBoost sections will use auto‑optimization: In the Shopify admin, go to Apps → RecoBoost and choose Recommend Sections (or the equivalent name in your environment) from the left menu. Limit yourself to one or two sections, such as “Recommended products related to this item” on product pages or “Cart recommendations” on the cart page.
• 3. Turn on auto_optimize: In the edit screen for the target section, find the recommendation logic settings and switch on Auto‑optimization (auto_optimize) or the equivalent toggle. Make sure the scope is set to this section only.
• 4. Narrow candidate logics down to around three: In the same screen, select which logics will be candidates for auto‑optimization. For example:
  - AI Similar Products (similarity‑based)
  - Bestsellers (based on views or purchases)
  - Recently Viewed or AI Personalize
  
  Pick 2–4 logics with different characteristics. Avoid going above five.
• 5. Confirm the performance metric (goal): In the same auto‑optimization settings, check which metric is used as the optimization target. Typically this is either CVR (purchase rate) or click‑through rate. For cart / pre‑checkout sections use CVR, and for product detail or home pages use click‑through rate. Choose one metric that matches the intent of the section.
• 6. Start running it from a higher‑traffic time of day: Save the settings and confirm that they are reflected in the live theme. If possible, activate auto‑optimization before a relatively busy time window for your store (for example 18:00–24:00). Traffic from the afternoon and evening will feed the initial learning.
• 7. On days 1–2 only check for extreme imbalance: After running for one to two days, open Reports or the Recommend Section detail screen in RecoBoost and check the balance of impressions, clicks, and purchases for each logic included in auto‑optimization. At this stage, ignore which logic is “winning” and only check that no logic is stuck at zero impressions.
• 8. On days 3–4 remove only clear losers: On days three to four, review the reports again and see if there are any logics that have a reasonable number of impressions but significantly lower click‑through rate or CVR than the others. If so, remove only those clear losers from the candidate pool and keep the remaining logics running.
• 9. On day 7 decide what to keep and what to test next: On day seven, judge based on the full week of data. Keep as your base logics those that have enough impressions and consistently higher click‑through or CVR, and remove weaker logics from the pool. Then, if there is another logic you want to try next week, swap in just one new candidate.

Tips for placement and choosing candidate logics

Where and with which logics you use auto‑optimization will affect how fast it learns and how easily you see results.

• On product detail pages, combine “similar” and “high‑selling” items: For product detail pages, build around AI Similar Products and add candidates like Bestsellers or Frequently Bought Together. This lets you test both items similar to what the user is viewing and overall store bestsellers.
• On cart pages, focus tightly on relevance: On cart pages, prioritize Frequently Bought Together and Related Products, and avoid mixing in too many bestsellers from distant categories. The goal here is raising average order value through cross‑sell.
• On the home and collection pages, use broad‑reach logics: For the home and collection pages, include logics like Bestsellers and AI Personalize that can broadly match many users. Here it is generally better to optimize for click‑through rate.
• Reduce overlap between candidate logics: If you include only similar logics, you lose much of the benefit of MAB. For example, instead of adding two different bestseller logics, combine Bestsellers + AI Similar Products + Recently Viewed so that each has a different behavior profile.
• Concentrate auto‑optimization in one place: In the first week, do not scatter auto‑optimization across many areas. Concentrate it in one or two high‑traffic sections so MAB / Thompson Sampling can learn faster.

Common pitfalls and cautions when traffic is low

If your store does not have much traffic or many purchases, MAB / Thompson Sampling will need more time to learn. Here are patterns you want to avoid and practical ways to handle them.

• Mistake 1: Swapping logics too often based only on one or two days of data: When data is thin, a single purchase or non‑purchase can swing your metrics a lot. If you keep changing candidates during this period, MAB learning will keep getting reset. As a rule of thumb, commit to not making major changes for at least seven days.
• Mistake 2: Having too many candidate logics: When traffic is low but the candidate list is long, each logic gets very few impressions, making it hard for Thompson Sampling to decide which ones are good. While you feel data is scarce, keep it to two or three logics.
• Mistake 3: Jumping to conclusions based only on CVR: In weeks with few purchases, CVR easily becomes extreme, like 0% or 100%. In those cases, check click‑through rate and impressions as well to see whether users are showing interest and whether each logic is getting enough exposure in the first place.
• Mistake 4: Frequently toggling auto‑optimization on and off: If you keep turning auto‑optimization on and off at short intervals, the model will not stabilize and you will not get the real benefit of MAB. Once you decide to test it, commit to running it continuously for at least one week.
• Note: Do not judge based on special event days alone (sale days, right after newsletter sends): On the day a sale starts or right after sending a newsletter, user behavior often differs from usual and both CVR and click‑through can swing wildly. When reviewing a week, either average those special days into the full week or separate them out and interpret them differently.

Next steps with RecoBoost

Here are the concrete next actions when you put this playbook into practice in RecoBoost.

• 1. Narrow down to one or two sections that will use auto‑optimization: In Apps → RecoBoost → Recommend Sections, limit the target to either the product detail page, the cart page, or both.
• 2. Choose only two to four candidate logics: In each section’s edit screen, pick logics with different characteristics as auto‑optimization candidates, such as AI Similar Products, Bestsellers, Recently Viewed, and AI Personalize. Keep the total to four or fewer.
• 3. Choose a metric and track reports for one week: In the section settings, select your optimization metric (CVR or click‑through rate), then use the Reports screen to track daily impressions and performance by logic for a full week. On day seven, decide which logics to keep, which to remove, and which new logic to test next, and complete one full cycle.