Search Words, Keywords and Statistics

I'd like to share some thoughts about ad stats on Etsy.

(For those of you who don't know what I am talking about, Etsy sellers can use search ads to buy spots on searches. The searches you get a spot in have to match the keywords you use in your shop descriptions. You can choose not to use some of your keywords if you think they will not be effective. You can also choose to only include sections of your shop or even just individual items. Quick definitions: 

Click through rate (CTR) = views divided  by impressions. 

Impressions are the number of times your ad was displayed. 

Views are the number of times people clicked on your ad.)

Search ads have been available on Etsy since 9/28/11. If you started using them right away you will have 10 days of data at the time I am writing this. I have read several posts about how to calculate click through rates and eliminate the key words that are not performing. I think this is a great idea in principal, as long as you make sure that you are basing your decisions on a large enough sample size. How do you know how much data is enough? I'm glad you asked!

Good data is not affected by things you don't want to measure. For example, when search ads first started, click through rates were pretty low. This may have been because a lot of sellers were doing searches to see if they could find their own ads, so there was not a lot of clicking happening. My click through rates were all over the place, as you can see from the graph below. After about 10/3 my click through rates leveled out at around 0.5 %, despite the fact that I have not made any changes to the way I set up my ads.

If I use the data from 9/28 to 10/2 to make decisions, I am using data that does not represent the long term performance of search ads. It would be smarter to use the data from 10/3 onward. The reason for this becomes even more clear when you look at how my top keywords preformed day by day.

Most of my promising words did not have good click through rates until after 10/3. If I had been eliminating words early on, I might be missing out on some really good keywords now.

Looking more closely at this graph leads into my next subject. Notice that the click through rates for some of these words are really high: 15% or more. This is because the number of daily impressions were so low, it only took one more two clicks to reach a very high click through rate. When you only have a few impressions, it is very difficult to determine whether one word is actually better than another.

The sad fact is that when you are looking at an either/or situation (i.e. click/ no click), it takes a lot of data to be able to tell what is going on. For example, look at the data from 10/3 through 10/7 for these two words:

It looks like “snowflake” is not as good a keyword as “bracelet”. But is this really the case? How do I know that it's not just chance? Statistics has a way to find out called the test of proportions. Using the test of proportions I was able to determine that there is not enough data to tell whether these two keywords are performing differently (in statistics, you don't prove two things are the same so much as you prove that they are not different).

The reason I can't tell the difference between these keywords is that “bracelet” does not have enough impressions. The next question on my mind is how many impressions do I need before I can tell if the keywords are different?

The answer depends on how great a difference you are trying to find. By playing around with the Test of Proportions a little bit I came up with the following table for comparing keywords to a baseline click through rate of 0.5%.  I can use these values as a rule of thumb for how many impressions I should have collected on that particular keyword before I make a decision.

So I need about 530 more impressions for "bracelet" to know whether or not it is different than "snowflake".   It will cost me about 50 cents to get that information; I think it will be worth it.