Why does nobody talk about how broken A/B testing can be with small lists?

I was running A/B tests on email subject lines for months thinking I was being smart. Then I read a post from a guy at HubSpot who pointed out you need at least 1,000 opens per variation for any real data. My list only has around 400 subscribers, so I was basically just flipping a coin. Has anyone else wasted time on tests that were doomed from the start?

2 comments

2 Comments

tylerpark5d agoMost Upvoted

Wait, you ran tests for MONTHS without realizing your list was too small? That's brutal, man. I did the exact same thing with my 300 person email list for like five months before figuring it out. Kept thinking my subject lines were genius when really it was just random noise. Honestly feels like half the "growth hacks" people push online are secretly built for huge lists but nobody says that part out loud.

amy9745d ago

Honestly the "growth hacks for big lists" thing hits hard, I wasted a year chasing the wrong metrics too.