A/B testing and growth experiments: how to run tests that give real answers
Most marketing "tests" end too early or test too many things at once, producing false winners. This guide explains hypotheses, sample sizes, statistical significance in plain language, test types for ads, pages and messages, and an experiment process for small teams.
A good growth experiment starts with a specific hypothesis, changes one meaningful thing, runs until it has enough data to separate real effects from chance, and ends with a decision recorded in a log. With limited traffic, test bigger changes and accept fewer, longer tests rather than many tiny inconclusive ones.
Module eight of Performance Marketing & Growth Hacking covers experimentation and the growth process. It is where many self-taught marketers have the biggest gap: they run tests, but cannot tell a real winner from luck.
What makes a good hypothesis?
Use the format "If we change X for audience Y, metric Z will improve because of reason R." For example: "If we show the fee and No-Cost EMI on the landing page for paid traffic, the enquiry-to-counselling rate will improve, because price uncertainty is the most common question counsellors hear." The reason matters: it turns a result into learning you can reuse.
Why do so many tests give false winners?
How much data do you need?
It depends on your baseline conversion rate and the smallest improvement you care about. As a rough guide for a page converting at 3%:
These are approximations for a standard test with 95% confidence and 80% power; use any free sample size calculator for your numbers. The lesson for most Indian SMBs: test big changes, because small ones need traffic you do not have.
Source: Careers Ninza approximation using standard two-proportion sample size formulas; illustrative.
What kinds of experiments can you run?
What does an experiment process look like for a small team?
A short weekly growth meeting reviewing experiment results and choosing the next tests is the habit that separates teams that improve steadily from teams that run random campaigns.
What if traffic is too low for A/B tests?
What are the limits of testing?
Tests answer narrow questions about the past period and audience. Results may not hold next season or for other audiences. Some important decisions, such as brand positioning or entering a new city, cannot be A/B tested cleanly and need judgement and research. Use experiments to reduce uncertainty, not to avoid thinking.
What does an experiment log look like?
A shared sheet with one row per experiment keeps learning from getting lost. The columns we use:
After a year, this log is one of the most valuable documents in the company: it captures what works for your customers, which no competitor can copy.
How do you test in WhatsApp and email flows?
Automation tools make it easy to randomise: send version A to people whose user ID ends in an even digit and version B to odd ones, then compare outcomes such as replies, bookings or purchases. Keep the rest of the flow identical and run until each version has enough outcomes. These tests are cheap because the audience already exists, and small improvements compound across every future customer.
How do you avoid fooling yourself?
Many experienced growth teams find that most of their experiments do not produce a winner. That is normal, and it is why the process, log and prioritisation matter more than any single test.
KEY TAKEAWAYS
What to do this week
Our Performance Marketing & Growth Hacking batches run live online, so the same class reaches learners and business owners in metros such as Chennai, Pune, Ahmedabad, Bengaluru and Kolkata, growing cities such as Ludhiana, Nagpur, Kochi, Bhubaneswar and Vadodara, and smaller industrial towns such as Morbi, Jalandhar, Hosur, Asansol and Kota, at the same fee. Each city page explains how local businesses use growth experimentation, for example in Ludhiana, Kochi, Kota.
Learn this live: Performance Marketing & Growth Hacking
Learn experimentation and the growth process, then apply it in a live-budget capstone with real data. 30 live hours, ₹24,999 paid in full or ₹30,000 as No-Cost EMI of ₹15,000 × 2.
Own the growth number — acquisition, experiments and CAC discipline.
Frequently asked questions
Related reading
We teach this, live
Every article here comes from something we teach. Sit in on a free masterclass and judge the mentors yourself.