I probably covered or sat on the Meet the Crawlers session dozens of times. So not sure how much new stuff you'll get out of me from this panel. This used to be the go-to panel, until people realized the presentations were all pretty much fluff.
Chris Serman announces the session, and says that this time to change it up, it will be Q&A pretty much. Cool. We got Google, Ask, Yahoo and MSN on the panel.
Q: Is there a particular order or pattern a crawler goes on your site? A: Yahoo says anything that anything that works for your users, we have to add to our crawlers. Yahoo doesn't want to recommend things to webmasters. Do what is best for you. Optimize for users and not spiders. MSN said, "no" as the short answer. Most spiders will look for all links, not the links on the top left or right, etc. Do not wrap links in JavaScript, etc. Make your links simple. Page size is also a factor, so keep it small. Chris Sherman said your comment was valid in the early days of engines. So if you read this somewhere, make sure that SEO tip stuff is recent and updated.
Q: Will Ask.com be supporting the standard sitemaps protocol? A: They are using sitemaps for select sites. They may, they are currently looking to see how well this is being adopted and they may in the future.
Q: Duplicate content in terms of appending tracking variables to URLs A: MSN said those pages are redundant, so you dont want them in the index. Try to take them out of the index. Ask.com said the search engine wont get confused by it, but you are wasting your bandwidth. Yahoo said look at Site Explorer, see where the URL was found at and get rid of it. Google said, I agree with MSN, its best to remove those IDs in the URL. You wont get a penalty, but Google will pick a URL for you, if you don't.
Q: Buying text links, does the crawler know this, do you downgrade those links? A: Google said he recommends that editorial links are better, don't really worry so much as purchased links, so get editorial links. Ask.com said you wont get much value in terms of crawling the page.
Q: PHP generated links treated differently? A: MSN said no matter what coding language you use, it generates HTML, so the crawler will see the HTML... Yahoo added the thing about looking at your site through the eyes of a user (kinda getting old Yahoo)
Q: If I have a 95% flash site with content in XML, will the XML file be read? A: Ask.com said XML is not human readable, so not right now. Yahoo said if all the content is also available in text, if the person has flash off, then ok.
Q: What are the most significant steps I can get my new site doing well, as opposed to the old site. Old site to new site? Domain the same, URLs are different I guess. A: Google said, 301 the old URLs to the new URLs. Make a new sitemaps file. Yahoo agreed with Google. Submit an old site map if you have the 301s to the one to one mapping. Ask.com same. MSN said they MSDN network, and they did tons of work to 301 from old URLs to new ones (live.com)
Some more basic Qs that I wont cover right this second. I am sorry, Ill save the battery on this computer for others. I will leave early. Hoping the next session is good.
Q: Are misspellings a black hat thing? A: MSN said it is based on how it was designed and the detail. If it was done for search engines, then don't do it. If it was done for users, then possibly, it is ok. Also search engines invest in spell checkers. Google said, he agrees, if you are adding text to the bottom of the pages just for search engines then avoid that technique. Yahoo and Ask agree. Chris added, if you have misspellings, then it is not professional.
Q: Someone asks Google about how do I get Google to show site links under the main listing? A: Google says it is purely algorithmic and he doesnt know a way to get them to show up.
K I am done with this session...
These posts may have spelling and grammar issues. These are session notes, written quickly and posted immediately after the session has been completed. Please excuse any grammar or spelling issues with session posts.
