I'd like to share some thoughts about
ad stats on Etsy.
(For those of you who don't know what I
am talking about, Etsy sellers can use search ads to buy spots on
searches. The searches you get a spot in have to match the keywords
you use in your shop descriptions. You can choose not to use some of
your keywords if you think they will not be effective. You can also
choose to only include sections of your shop or even just individual
items. Quick definitions:
Click through rate (CTR) =
views divided by impressions.
Impressions are the number of times
your ad was displayed.
Views are
the number of times people clicked on your ad.)
Search ads have been available on Etsy
since 9/28/11. If you started using them right away you will have 10
days of data at the time I am writing this. I have read several
posts about how to calculate click through rates and eliminate the
key words that are not performing. I
think this is a great idea in principal, as long as you make sure
that you are basing your decisions on a large enough sample size. How do you know how much data is enough?
I'm glad you asked!
Good
data is not affected by things you don't want to measure. For
example, when search ads first started, click through rates were
pretty low. This may have been because a lot of sellers were doing
searches to see if they could find their own ads, so there was not a
lot of clicking happening. My click through rates were all over the
place, as you can see from the graph below. After about 10/3 my
click through rates leveled out at around 0.5 %, despite the fact
that I have not made any changes to the way I set up my ads.
If I
use the data from 9/28 to 10/2 to make decisions, I am using data
that does not represent the long term performance of search ads. It
would be smarter to use the data from 10/3 onward. The reason for
this becomes even more clear when you look at how my top keywords
preformed day by day.
Most
of my promising words did not have good click through rates until
after 10/3. If I had been eliminating words early on, I might be
missing out on some really good keywords now.
Looking
more closely at this graph leads into my next subject. Notice that
the click through rates for some of these words are really high: 15%
or more. This is because the number of daily impressions were so
low, it only took one more two clicks to reach a very high click
through rate. When you only have a few impressions, it is very
difficult to determine whether one word is actually better than
another.
The
sad fact is that when you are looking at an either/or situation (i.e.
click/ no click), it takes a lot of data to be able to tell what is
going on. For example, look at the data from 10/3 through 10/7 for these two words:
It
looks like “snowflake” is not as good a keyword as “bracelet”.
But is this really the case? How do I know that it's not just
chance? Statistics has a way to find out called the test of proportions. Using the test of proportions I was able to determine
that there is not enough data to tell whether these two keywords are
performing differently (in statistics, you don't prove two things are the same so much as you prove that they are not different).
The
reason I can't tell the difference between these keywords is that
“bracelet” does not have enough impressions. The next question
on my mind is how many impressions do I need before I can tell if the
keywords are different?
The
answer depends on how great a difference you are trying to find. By
playing around with the Test of Proportions a little bit I came up with the following
table for comparing keywords to a baseline click through rate of
0.5%. I can use these values as a rule of thumb for how many impressions I should have collected on that particular keyword before I make a decision.
So I need about 530 more impressions for "bracelet" to know whether or not it is different than "snowflake". It will cost me about 50 cents to get that information; I think it will be worth it.
