Wow…so it’s been a little while since I’ve added a post. What can I say…work has been hectic.

It is known that if you target your email, you are increasing the relevancy of each interaction with your customers. The question becomes “At what cost?”. Is there some benefit to sending someone a dog email who has stated or purchased only cat products? Okay, that one may be obvious, but what do you do in the case of apparel. For example, would you send an email promoting bikinis to someone who has only purchased jeans in hopes of converting them. If you are only creating one email and do not plan on segmenting your list, is it better to not send the bikini email to a customer who purchased only jeans?

If we specifically ask customers to give us their preference, do we only make them mad by sending them something else in hopes of converting them to purchase a product which they initially have no interest in?

Or, do we rely on our analytics tools and purchase history and target our customers based on this data. This allows us to be pro-active in anticipating the behavior of our customers without asking the customer to provide one bit of data. Is this then not an added bonus for a customer when they realize they received an email for a pair of shoes that are on sale which just happen to match the top they purchased last week?

This is a constant struggle we as email marketers are faced with. We see the data…we run the tests. But how do we know for sure. How are you segmenting your data?

So now you are beginning to plan your first A|B test.  Upper management has buy-in and is willing to give you everything you need to pull this off (in a perfect world).  Now you need to figure out what will determine your test to be successful. 

 To start, it depends on what variables you are testing.  Here are some thoughts on various tests to get you thinking.

Emails

• In general, open rates are a bogus metric to use except in the case of a subject line test.  Be careful with this as there are considerations you will need to think about when sending to certain ISP’s.
• Clickthrough rate will be a good metric to determine the effectiveness of the content of the email.
• Conversion can be tough to use as it can be more of a measure of a users experience on the site more so than the content of your test

Landing Pages

• Conversion will be the key indicator in this case.  This generally means that the customer saw the page and clicked through a call to action or headline on this page
• Exit rate will be another good measurement to use.  If a user visits your site from a search engine or email, the user is going to expect the content to be relevant.  Obviously you are testing the relavancy of content in this case.  If the exit rate for one site is higher than the other, it is a good indicator that the content is not relevant

Changes to a process (ie…checkout)

• Before you begin your test, take a look at the steps a user needs to follow in order to complete the process you presenting to them.  Most analytics tools allow you to program a sequence of steps (sometimes knowns as a funnel) either through character strings in a URL or through variables coded on a page.  Use this analysis to determine if you see an increase in abandonment at a certain step of the process

Page Layout

• It is best to segment a few pages that you can separate from the remainder of the site.  This will make conversion easier to calculate
• Just a we did with a process, we want to test the number of customers who start at one point, and follow through to another.
• If a change to your page includes aspects which could effect the value of a transaction, you will want to look at the revenue associated with the test
• If any change you make could effect the way the page is viewed in certain screen/browser combinations or have specific technology requirements, be sure you are looking at the effect your change has on those customers.

Overall, you must realize that it will be difficult to perfectly balance traffic in a 50/50 split.  In some cases, you may not be testing a 50/50 split but rather a 75/25.  In these cases, you will want to use weighted metrics such as Average order value, conversion, Units per transaction, etc…

Starting to get in to A|B testing? Are the base reports available through the web optimizer lacking something…like revenue? Want to determine how you can determine your AOV? This article is a great resource that explains how you can use google analytics to drill down into any A|B test you run. A few quick steps and you are on your way

read more | digg story

Sometimes I think the biggest challenge for an analyst comes with something that would seem so easy to fix. Verifying data. I know, you are reading this and thinking. duh! Imagine you are tasked with a daily report that you must fill out and distribute. How repetitive can it be running the same reports and entering data. I know for most of us, this equates to a small portion of our day but at times, it seems as if this could have more of an effect than anything else, depending on who the data is being distributed to.

I have been a victim of this on more than one occasion. It is one thing I am constantly working to fix. Double checking your numbers is necessary. It could be a case of hitting the wrong key or it could be your choice of metrics. Whatever the case may be…double check and double check. Most importantly, if you have no one else to review numbers before they are distributed to management, have someone else check your work.

As I said, I know this is such common sense. However, I see it happen on a daily basis to myself and others.