Below is live coverage of the Paid Search Testing And Reporting panel from the SMX West 2011 conference. This coverage is provided by Keri Morgret of Strike Models.
Disclaimer: The coverage is brought to you in real time, using a custom live blogging tool. Feel free to ask questions or leave comments for inclusion into the live coverage. During the live event, live notes will auto-scroll with newest entries at top. After the session is complete the archive version will have the oldest entries at the top. We ask you to please excuse any typos, as these are live notes.
Finished
Auto-Refresh: Off
Keri Morgret:
11:15:50 am
We're starting the session. Matt Van Wagner notes that this is a new session this conference. I don't have wifi connectivity, so I'm writing this in notepad and will post this when I am online again. Apologies for the delay.
Craig Danuloff starts us off.
Are we testing, or just "giving it a try"?
Paid search really should be a test-driven process, but it rarely is. Reasons: it's really difficult to do.
How is testing different from trying?
Testing benefits from a strategy and a management framework. Need to be able to repeat the experiment.
Testing assumes a goal and a hypothesis.
Testing requires a controlled experiment, valid statistics, and a consulsion and resulting action.
It's also hard when you've got a lot of people in making changes, not everyone makes the same types of changes, so it's really hard to have a controlled experiment.
Sometimes you have a hard time convincing a boss or the organization in general that you should turn off certain keywords. He jokes about geo-targeting for only boss' house.
Strategic considerations in testing.
Are campaigns split by stage in buying cycle? Need to look beyond just the final conversion?
Are there success metrics beyond standard conversions?
Are there distinc values for distinct types of events?
Is there consideration of lifetime value?
Is there a method of tracking cross-channel and offline conversions?
Management Framework considerations:
Clear 'way you do things'? When do you choose to do something or make a change?
Agreed upon key metrics?
Rules for pausing keywords and ad copy?
eBook is available, and printed at their booth.
Control over variables.
Impact of text ad copy change on keywords?
Impact of bid or position changes on text ad copy?
Impact of impression share on keyword or ad copy?
Impact of quality score changes on bid or ad position?
Impact of AvgCPC against MaxCPC (% of MaxCPC taken)
Don't forget about external variables. If it's the end of the month and a competitor runs out of money and stops advertising, it's nothing you necessarily did that resulted in going up a couple of positions.
Tools to manage test? (things he really wished existed that would help this!!)
Lock options/elements around control?
Report if bid or ad copy has changed during test period?
Report if keywords added to ad group during ad copy test?
Report if position changed (due to competition) during bid test?
Verify significance of any change or comparision?
Goal: Is it attracting the right people?
What's the Goal?
Clicks start coming. What do you look at?
- Quality of queries -- are the right people coming?
- Queries tell what the people are thinking and what Google is thinking.
- Are those the people you wanted?
- Does Google have your meaning right?
- If not: rethink keyword or negatives or match type.
What Does the Quality Score Say?
In brief:
Over 7, great job.
Equal to 7, good job.
Less than 7, this keyword may not be for you.
Does the keyword work for you?
Consider key metrics
Decide if you should be advertising on this keyword
Act: kill it, work harder, or make modifications
Craig Danuloff starts us off.
Are we testing, or just "giving it a try"?
Paid search really should be a test-driven process, but it rarely is. Reasons: it's really difficult to do.
How is testing different from trying?
Testing benefits from a strategy and a management framework. Need to be able to repeat the experiment.
Testing assumes a goal and a hypothesis.
Testing requires a controlled experiment, valid statistics, and a consulsion and resulting action.
It's also hard when you've got a lot of people in making changes, not everyone makes the same types of changes, so it's really hard to have a controlled experiment.
Sometimes you have a hard time convincing a boss or the organization in general that you should turn off certain keywords. He jokes about geo-targeting for only boss' house.
Strategic considerations in testing.
Are campaigns split by stage in buying cycle? Need to look beyond just the final conversion?
Are there success metrics beyond standard conversions?
Are there distinc values for distinct types of events?
Is there consideration of lifetime value?
Is there a method of tracking cross-channel and offline conversions?
Management Framework considerations:
Clear 'way you do things'? When do you choose to do something or make a change?
Agreed upon key metrics?
Rules for pausing keywords and ad copy?
eBook is available, and printed at their booth.
Control over variables.
Impact of text ad copy change on keywords?
Impact of bid or position changes on text ad copy?
Impact of impression share on keyword or ad copy?
Impact of quality score changes on bid or ad position?
Impact of AvgCPC against MaxCPC (% of MaxCPC taken)
Don't forget about external variables. If it's the end of the month and a competitor runs out of money and stops advertising, it's nothing you necessarily did that resulted in going up a couple of positions.
Tools to manage test? (things he really wished existed that would help this!!)
Lock options/elements around control?
Report if bid or ad copy has changed during test period?
Report if keywords added to ad group during ad copy test?
Report if position changed (due to competition) during bid test?
Verify significance of any change or comparision?
Goal: Is it attracting the right people?
What's the Goal?
Clicks start coming. What do you look at?
- Quality of queries -- are the right people coming?
- Queries tell what the people are thinking and what Google is thinking.
- Are those the people you wanted?
- Does Google have your meaning right?
- If not: rethink keyword or negatives or match type.
What Does the Quality Score Say?
In brief:
Over 7, great job.
Equal to 7, good job.
Less than 7, this keyword may not be for you.
Does the keyword work for you?
Consider key metrics
Decide if you should be advertising on this keyword
Act: kill it, work harder, or make modifications
Keri Morgret:
11:16:17 am
Next up is Benjamin Vigneron is now up.
Topic: what is a good control experiment, main requirements, how to make test successful.
1) Implement a valid testing procedure
Identify market and account factors that potentially affect the test.
- There are things you don't have control over, like seasonality, competition, consumer behavior, algo updates, etc.
- Identify account factors (internal factors). Your budgets, editorial changes, bid management, targeting settings, etc.
Anticipate Market Factors
The external factors from above -- try to make predictions about them, look at last year, etc.
Again, what Craig said. Don't make multiple changes at the same time.
One client had only phrase and exact match. Agency wanted to try broad match. To help control, they did a broad match campaign, and did negative match for phrase match keywords.
When evaluating a test, take into account what you expected (anticipating market factors), not what happened last week.
Use statistical laws to determine whether the standard deviation is statistically significant.
He shows how to use the chi square test in excel.
Topic: what is a good control experiment, main requirements, how to make test successful.
1) Implement a valid testing procedure
Identify market and account factors that potentially affect the test.
- There are things you don't have control over, like seasonality, competition, consumer behavior, algo updates, etc.
- Identify account factors (internal factors). Your budgets, editorial changes, bid management, targeting settings, etc.
Anticipate Market Factors
The external factors from above -- try to make predictions about them, look at last year, etc.
Again, what Craig said. Don't make multiple changes at the same time.
One client had only phrase and exact match. Agency wanted to try broad match. To help control, they did a broad match campaign, and did negative match for phrase match keywords.
When evaluating a test, take into account what you expected (anticipating market factors), not what happened last week.
Use statistical laws to determine whether the standard deviation is statistically significant.
He shows how to use the chi square test in excel.
Keri Morgret:
11:17:14 am
It looks like I have some connectivity now, so hopefully we'll have faster updates.
Keri Morgret:
11:17:38 am
Testing tips:
Since you'll need a lot of stats, run the tests on high traffic keywords.
Since you'll need a lot of stats, run the tests on high traffic keywords.
Keri Morgret:
11:17:56 am
Not a question of time for how long to run a test, but a question of how much data you get.
Keri Morgret:
11:18:18 am
Use even ad rotation. It's in advanced settings in AdWords.
Keri Morgret:
11:18:39 am
First test very different ad copy.
Keri Morgret:
11:19:51 am
Again, use a chi square test to determine which one is better performing. Keep best performing ad, then do a new ad to competing against this ad..but this time, use smaller changes, test just one line of ad.
Keri Morgret:
11:20:03 am
This process is never finished. Things do change over time, user behavior, etc.
Keri Morgret:
11:20:50 am
Next, you can test multiple landing pages. Isolate one thing, only the landing page varies. Use same ad test.
Keri Morgret:
11:21:31 am
When you pull a report, pull JUST the metrics you need.
Keri Morgret:
11:22:33 am
Compare the ad performance using the chi square test per ad group.
Focus on the CTR, or the conversion rate, or both: Conversions/Impressions.
Focus on the CTR, or the conversion rate, or both: Conversions/Impressions.
Keri Morgret:
11:23:44 am
Email him [email protected] if you want a copy of this excel sheet he was showing.
Keri Morgret:
11:24:22 am
Next up is Addie Conner. Let's hope I can type fast enough to keep up with her. I've loved her great presentations in the past.
Keri Morgret:
11:25:12 am
She's giving a background of SocialCode, her new company. They're a full-service facebook agency, and a facebook beta partner.
Keri Morgret:
11:25:27 am
They use a data-drive approach to advertising in engagement.
Keri Morgret:
11:26:12 am
"What if we used online just to collect information to inform our other channels?" She's giving a case study of a casino in Vegas.
Keri Morgret:
11:26:46 am
Question: Who is responding to our current performers? Is it young, hip people? Senior citizens?
Keri Morgret:
11:27:13 am
Purpose: understand which entertainment acs were attracting which artists from top three feeder cities.
Keri Morgret:
11:27:52 am
Casino client hypothesis: acts were attracting late 30s, early 40s, rich business travelers.
PR agency hypothese: client is wrong.
Agency hypothesis: we don't know, just want to run the test.
PR agency hypothese: client is wrong.
Agency hypothesis: we don't know, just want to run the test.
Keri Morgret:
11:28:05 am
Five performers, three cities examined.
Keri Morgret:
11:28:49 am
Needed to make assumptions to inform client. Using CTR as response metric, want to detect 10% diff in CTR for ages and genders, 95% confidence, and compare between cells.
Keri Morgret:
11:29:06 am
"Combinatorics are a bitch".
Keri Morgret:
11:29:42 am
1260 combinations of things they wanted signficiance for.
Keri Morgret:
11:30:36 am
Their cost per cell was $560..and there were 1260 cells. Plus adding more money that I missed. Test would have been a couple of million dollars.
Keri Morgret:
11:31:08 am
Compromise to get cost/time down.
Reduced the number of age groups.
Reduced from 42 individual ages to five decade buckets.
Reduced number of cells considerably.
Reduced the number of age groups.
Reduced from 42 individual ages to five decade buckets.
Reduced number of cells considerably.
Keri Morgret:
11:32:12 am
Structure:
Keep creative constant and standardized.
Used CPM bidding to ensure equal distribution.
Artist, age, city, gender targeting on ad level.
Separated individual ages within ad structure but categorically tagged w/in system.
Set significance threshold for impressions.
Keep creative constant and standardized.
Used CPM bidding to ensure equal distribution.
Artist, age, city, gender targeting on ad level.
Separated individual ages within ad structure but categorically tagged w/in system.
Set significance threshold for impressions.
Keri Morgret:
11:32:31 am
She goes into more details about how this was set up.
Keri Morgret:
11:33:19 am
To me, this is really interesting about how to use this as a proxy for offline behavior.
Keri Morgret:
11:34:08 am
The simple report version:
No matter what the city or artist, CTR goes up with age.
Most acts skew male. Donny & Marie skew female, George Wallace closer to gender-neutral.
No matter what the city or artist, CTR goes up with age.
Most acts skew male. Donny & Marie skew female, George Wallace closer to gender-neutral.
Keri Morgret:
11:34:35 am
She's showing graphic results. My camera app isn't working so I can't upload.
Keri Morgret:
11:35:21 am
Recommendations:
Target travel packages to older temo
Take picture of 30 something couple in bed off website.
If looking to get 300something travelers, get lively looking folks in their 40s, maybe there will be perimeter effect on people in 30s.
Target travel packages to older temo
Take picture of 30 something couple in bed off website.
If looking to get 300something travelers, get lively looking folks in their 40s, maybe there will be perimeter effect on people in 30s.
Keri Morgret:
11:36:16 am
Next steps:
Incorporate learning in other media and marketing efforts
Focus acquisition efforts on highest responding groups
Now that you know the relative response rates by demo, start message testing to find optimal language for each bucket -- speak to them in the right way.
Incorporate learning in other media and marketing efforts
Focus acquisition efforts on highest responding groups
Now that you know the relative response rates by demo, start message testing to find optimal language for each bucket -- speak to them in the right way.
Keri Morgret:
11:36:43 am
Her email is [email protected] if you have any more questions.
Keri Morgret:
11:38:18 am
Randy Wong is now up.
Keri Morgret:
11:38:39 am
Challenges of paid search testing, from an in-house perspective.
Keri Morgret:
11:39:08 am
Two main areas of their company: instruments (big, expensive) and consumables (stuff you put into big expensive instruments)
Keri Morgret:
11:39:37 am
They've been doing paid search for about six years, started getting serious about three years ago, looked at optimizing things in past two years.
Keri Morgret:
11:40:04 am
What should be tested?
So many different variables! You want to test them all, but you really need to focus on what you can test, what's feasible, what brings customer value.
So many different variables! You want to test them all, but you really need to focus on what you can test, what's feasible, what brings customer value.
Keri Morgret:
11:41:04 am
What is feasible?
Biggest challenge getting internal buyin
- Who cares about testing?
- Is there internal buy in/ From what level?
- Is testing a priority?
- Are there dedicated resources?
You need funding, you need cooperative IT department, etc.
Biggest challenge getting internal buyin
- Who cares about testing?
- Is there internal buy in/ From what level?
- Is testing a priority?
- Are there dedicated resources?
You need funding, you need cooperative IT department, etc.
Keri Morgret:
11:41:45 am
Getting help from others in org can be tough, because this often isn't written in their job description.
Keri Morgret:
11:42:28 am
Figure out what can be changed. What can you even change? If you're in particular industries, like banking or biosciences, you've got external regulations about the type of ad copy you can use, for example.
Keri Morgret:
11:43:01 am
Are approvals required for budget and keyword changes?
What is the culture of your company? Is it comfortable with wild changes, or very conservative?
What is the culture of your company? Is it comfortable with wild changes, or very conservative?
Keri Morgret:
11:44:00 am
What matters to your customers?
Qualitative data:
- why do customers come to your site?
- What influences customers to make a purchase? What matters to them?
- What type of customers come? What motivates them to buy? what are their goals?
Qualitative data:
- why do customers come to your site?
- What influences customers to make a purchase? What matters to them?
- What type of customers come? What motivates them to buy? what are their goals?
Keri Morgret:
11:45:11 am
End-users (researchers) and purchasers come to the site. End-users want things that help the lab, don't want individual giveaways (and in some cases can't accept them).
Purchasers don't use search, just got straight to the site.
Purchasers don't use search, just got straight to the site.
Keri Morgret:
11:45:29 am
In their case, purchaser isn't the one who consumed your marketing material. Makes attribution tricky.
Keri Morgret:
11:46:03 am
Start with what's working now (in relating to your site) and optimize that, rather than starting from scratch.
Keri Morgret:
11:47:03 am
Internal stakeholders -- what matters to them?
- Make their jobs easier
- Make them more efficient
- Help them reach their goals
- Make testing and optimization part of their job.
- Make their jobs easier
- Make them more efficient
- Help them reach their goals
- Make testing and optimization part of their job.
Keri Morgret:
11:47:50 am
He's giving an example of a recent test that focused on translation efforts (they focused on translation efforts in Europe).
Keri Morgret:
11:48:18 am
Internal benefits: Reduce cost and improve focus on future translation efforts.
Customer/business benefit: reach and engage more customers.
Customer/business benefit: reach and engage more customers.
Keri Morgret:
11:48:42 am
Test: compare English keywords and English ads vs translated keywords and ads in five regions in Europe.
Keri Morgret:
11:49:32 am
Clear difference in diff't regions for both impressions and CTR.
Different results for different regions. Now they're only going to focus on translations for regions that had good results.
Different results for different regions. Now they're only going to focus on translations for regions that had good results.
Keri Morgret:
11:50:16 am
Tips on gaining momentum for a testing program:
Education to gain top down support
Engage resources: communicate and attribute results
Establishing and maintaining a rhythm: get tests out the door, start with tests that matter internally and externally.
Education to gain top down support
Engage resources: communicate and attribute results
Establishing and maintaining a rhythm: get tests out the door, start with tests that matter internally and externally.
Keri Morgret:
11:50:44 am
You really need to make it matter to people to get them to buy in, and you make have to convert people who had had previous bad experiences with PPC.
Keri Morgret:
11:51:26 am
He can be emailed [email protected]
Keri Morgret:
11:51:45 am
And it's Q&A time now.
Keri Morgret:
11:54:31 am
Question about what to test first. Craig: ad copy has big impact.
Keri Morgret:
11:55:44 am
Randy says structuring and organization of ad groups and campaigns impacted his results.
Keri Morgret:
11:56:27 am
I have an appointment at noon, so I'm shutting down a couple of minutes early.
