Below is live coverage of the The Great Correlation vs. Causation Debate panel from the SMX East 2011 conference.
Disclaimer: The coverage is brought to you in real time, using a custom live blogging tool. Feel free to ask questions or leave comments for inclusion into the live coverage. During the live event, live notes will auto-scroll with newest entries at top. After the session is complete the archive version will have the oldest entries at the top. We ask you to please excuse any typos, as these are live notes.
Finished
Auto-Refresh: Off
Barry Schwartz:
9:46:41 am
We should be starting in a minute or so, but the wifi is getting a bit weird.
Barry Schwartz:
9:46:54 am
Just switched to my iPhone's internet, let's see how that goes...
Barry Schwartz:
9:47:39 am
The Great Correlation vs. Causation Debate (#smx #12B)
All SEOs know that making changes to your site can have a significant impact on rankings. The challenge is in understanding what changes actually cause ranking changes, and which changes merely correlate with rankings (e.g. they’re coincidental byproducts of change). In this lively session expert SEOs who’ve dived deeply into understanding search algorithms debate correlation vs. causation and make their cases for the types of ranking factors that do (or don’t) have significant meaning.
All SEOs know that making changes to your site can have a significant impact on rankings. The challenge is in understanding what changes actually cause ranking changes, and which changes merely correlate with rankings (e.g. they’re coincidental byproducts of change). In this lively session expert SEOs who’ve dived deeply into understanding search algorithms debate correlation vs. causation and make their cases for the types of ranking factors that do (or don’t) have significant meaning.
Barry Schwartz:
9:50:01 am
Promo speech now...
Barry Schwartz:
9:51:40 am
Danny gives a summary on why we are having this session, which is described a bit.
Barry Schwartz:
9:53:06 am
Two presentations and then the rest of the panelists will just do small talks.
Barry Schwartz:
9:53:21 am
Kristine Schachinger, Owner, SitesWithoutWalls.com (@schachin) is up first.
Barry Schwartz:
9:55:04 am
Correlation is when you have a mutual relationship between two or more things where one has a measurable effect on the other.
Barry Schwartz:
9:55:29 am
Causation is a relationship in which one action or event is the direct consequence of another.
Barry Schwartz:
9:56:30 am
If I hit a ball with a tennis racket, racket hits ball, ball goes so far, the length of how far the ball goes is a causal effect. But the breakfast you had that morning might be correlated to how far you hit the ball...
Barry Schwartz:
9:56:49 am
Often correlation is mistaken for causation...
Barry Schwartz:
9:58:52 am
We could make assumptions based on old ideas or incorrect perception. For example, site code on rankings - pre-Google's site speed announcement.
Barry Schwartz:
9:59:33 am
You always want to test and test to validate.
Barry Schwartz:
10:00:01 am
So when we have a misperception, then we have no correlation or a spurious correlation
Barry Schwartz:
10:00:57 am
Spurious correlation is for example, the weekend before SMX is in town, there is an increase in beer sales. So they might think it is related to SEOs. But there is also Fashion Week going on. So that variable is there also.
Barry Schwartz:
10:01:23 am
So do we know it is also fashion week? No. So we have to measure and test. We need to measure and test it exists.
Barry Schwartz:
10:01:50 am
In cause relationships you an directly measure A causes B.
Barry Schwartz:
10:02:04 am
Also it will happen over and over again, it is reproducable.
Barry Schwartz:
10:02:24 am
Correlative relationships you can only measure the strength of the effect on A on B.
Barry Schwartz:
10:03:17 am
She gives more examples, sorry, can't do pictures, because my iPhone is now my hotspot and makes it hard to stay connected... long story...
Barry Schwartz:
10:04:12 am
You see this in polls often... political polls. we are 93% sure they will win +/- 3% margin of error...
Barry Schwartz:
10:05:29 am
Whenever you measure something, you have to do testing. There are always so many things to test. You need to learn statistics. You need to know your variable types. You need to have enough things in your sample to normalize the data, i.e. the outliers don't ruin your results.
Barry Schwartz:
10:05:44 am
Side note: She 100% knows what she is talking about, it is refreshing.
Barry Schwartz:
10:06:10 am
Most SEO is correlative, because there are so many factors we cannot validate.
Barry Schwartz:
10:06:42 am
Unless Matt Cutts of Google said something is true or not. Everything else is not 100% true. And then Google changes things every day.
Barry Schwartz:
10:07:08 am
Next up...
Barry Schwartz:
10:07:11 am
Micah Fisher-Kirshner, Senior SEO Manager, Become, Inc
Barry Schwartz:
10:07:33 am
Barry Schwartz:
10:08:34 am
Public Correlations: how do you tell the good vs the bad?
Barry Schwartz:
10:08:57 am
Correlation should be shown with scatter plots, it helps show to visualize what it represents.
Barry Schwartz:
10:09:37 am
He likes to avoid integers because it makes it hard to visualize correlations.
Barry Schwartz:
10:09:58 am
(1) Correlation is a place to start, it is not a place to end.
Barry Schwartz:
10:10:19 am
(2) SCheck the results with an SEO
Barry Schwartz:
10:10:29 am
(3) Does this make sense, what if we factor for X
Barry Schwartz:
10:11:08 am
When you use the linear regression side...
Barry Schwartz:
10:11:31 am
How much time an effort will it take if you go down that route, and is there ROI on those efforts?
Barry Schwartz:
10:11:48 am
Is the data trustworthy? Are you factoring all the relevant data that matters?
Barry Schwartz:
10:13:12 am
They looked at Panda and the average time on site. They saw a 42% correlation, but the scatter plot doesn;t really show a correlation?
Barry Schwartz:
10:13:45 am
So they looked deeper at other sites like theirs, by categories. Maybe there was a group punishment? I.e. How tos, CSEs, marketplaces, News, etc.
Barry Schwartz:
10:13:56 am
So plot them all. and then drill down even deeper.
Barry Schwartz:
10:14:35 am
You need to look at stuff day over day, how about week over week, and then look for seasonality... Account for your data is, in the time of year.
Barry Schwartz:
10:14:50 am
Then look at subdirectories, subdoamins and domains.
Barry Schwartz:
10:15:18 am
With all tests, you always need a control to avid bias. Two controls are always better than one.
Barry Schwartz:
10:15:35 am
The less amount of data you have, the longer you need to run the test.
Barry Schwartz:
10:15:45 am
Common Correlating Pitfalls
Barry Schwartz:
10:15:51 am
- Other marketing channels
Barry Schwartz:
10:16:06 am
- Extraneous online events, like other Google updates.
Barry Schwartz:
10:16:18 am
- Did you break something, did you launch something new, etc.
Barry Schwartz:
10:16:55 am
Offline events, such as holidays, world or national events. Such as the impact of the DC earthquake.
Barry Schwartz:
10:17:17 am
For about two hours, people stopped shopping during the earthquake. It lowered their traffic by 2% that day.
Barry Schwartz:
10:18:23 am
Bottom line is you need to question everything!
Barry Schwartz:
10:18:47 am
Short comments from rest of panel...
Barry Schwartz:
10:19:09 am
Mitul Gandhi from seoClarity shares...
Barry Schwartz:
10:19:25 am
He says, what can you actually do with it?
Barry Schwartz:
10:19:47 am
(1) Measurement : How do you report on Correlation vs. Causation
Barry Schwartz:
10:20:07 am
Avoid the pitfall of looking at all the variables in mind, like said before.
Barry Schwartz:
10:20:42 am
(2) Look at it as not as snap shots in time...
Barry Schwartz:
10:22:29 am
(3) It has to be reproducible.
Barry Schwartz:
10:23:20 am
Tony Wright, CEO, WrightIMC (@tonynwright)
Barry Schwartz:
10:24:35 am
Secret on how to do SEO:
Barry Schwartz:
10:25:42 am
(1) Code - clean
(2) Content - good content
(3) Connections - links from other web sites
(4) Conversations - social media
(2) Content - good content
(3) Connections - links from other web sites
(4) Conversations - social media
Barry Schwartz:
10:26:28 am
Do these things cause you to rank? I cannot statistically tell you if they do. I would be spinning my wheels trying to prove it. For 95% of you, that is where you need to go.
Barry Schwartz:
10:27:34 am
Eric Enge, President, Stone Temple Consulting (@stonetemple)
Barry Schwartz:
10:27:55 am
He said, it just doesn't matter.
Barry Schwartz:
10:28:07 am
The great majority of you dont have the time to do this analysis.
Barry Schwartz:
10:29:17 am
(1) Does Correlation cause Causation? Does a FB share result in links?
Barry Schwartz:
10:29:32 am
(3) Does correlation cause traffic?
Barry Schwartz:
10:29:53 am
(4) Aren't the signals just going to change anyway?
Barry Schwartz:
10:30:54 am
That is all, Q&A time.
Barry Schwartz:
10:31:10 am
30 minutes of Q&A time...
Barry Schwartz:
10:31:23 am
Normally I don't cover it, but maybe I'll cover some.
Barry Schwartz:
10:33:46 am
Danny said back in 1998, there was a ranking tool named web position. It was really cool tool back then. We didn't have analytics back then. But they weren't satisfied with that, and they came out with a page advisor tool later on. You ran a report, and it would tell you how to optimize each page for each search engine. People asked Danny if they should use it and he said, the tips per search engine were not that different.
Barry Schwartz:
10:34:52 am
Even if you followed these tips 100%, it wouldn't always work b/c there would be so many more factors.
Barry Schwartz:
10:36:03 am
Can we know the ranking algorithm? One person said, "we can know enough of it." and most said yea, there are good "major factors" we all know.
Barry Schwartz:
10:38:34 am
There is some debate about to do testing or not.
Barry Schwartz:
10:41:54 am
Is it worth testing with all the variables? Should you test at all? If so, what? Etc.
Barry Schwartz:
10:46:23 am
Alright, next session is Panda-Proofing Your Content at 1:45 EDT.
