Google scraper ban

I posted something similar to this on the WMW forum and then decided it belogs here on my blog. This was a thread about collateral damage from the 7-29 Google scraper ban.

Andilinks has a lot of original content, both on pages full of text written entirely by me and as commentary about various listed links. I was however also using exerpts and meta descriptions from the linked sites to describe them and must have passed some threshold for that. I was seeding with these descriptions and re-editing later. I certainly will admit that the AdSense revenue was very good and that did provide an incentive to produce more pages, especially high-traffic pages.

There are far too many pages on my site like this for me to change them legitimately in a timely fashion and using text-scrambling software would degrade the descriptions and make my site more like the scrapers themselves. But I may write a script to swap out certain synonyms wholesale.

I am disallowing googlebot from those pages for now (they still do well with Y and M but I fear these may have their own scraper crack-down soon as well). I am keeping my fingers crossed.

In this recent recovery process I accidentally used too many 301 redirects for the server load and probably confused slurp and msnbot too...

But I have almost four years work invested in the database and am working day and night to recover this. If you stop hearing from me you'll know I 'died trying.'

andiFASHION

I am just heartbroken that Google dropped Andilinks from its index on July 28. I am finally recovering but I doubt I'll ever get back to where I was with it...

I have started a new site with just the Women's Wear pages. I have gotten two pages into Google but I'm blocking the googlebot from the directory until I figure out why Andilinks was banned.

I was probably swept up in the scraper spam ban, but I don't know how to keet the robot from mistaking me for a scraper. Well, I'll have more to say on this later. I'm finally getting my spirits back with having a new website.

New Items, Thursday

Victor Davis Hanson Interview Chrenkoff
wists yet another social bookmark tagger
Peeking Into Google internetnews.com

New Items, Wednesday

Lawrence Lessig Interview, P2P O'Reilly Network
Long Tail Chris Anderson's blog
Yahoo Netrospective
Kitchen & Housewares new Andilinks page

New Items, Tuesday

Bubble Redux?. CBS via techdirt
Banned Cartoon Russell Beattie
Artists Break With Industry on File Sharing via Techdirt

New Items, Monday

The FUD based Encyclopedia pro Wikipedia article
PVRblog Tivo etc.
hack a day
Free Tech Support.org
ComputerGeek tips & gadgets
JimsTips.com
Gmail Tips

New Items, Sunday

Old Europe Mark Steyn, Sun Times
Lindsay Lohan new Andilinks page
Hilary Duff new Andilinks page

New Items, Saturday

cnet top 100
Odeo Blog
NegativWorldWideWebland
Postmodern War City Journal
Jessica Simpson new Andilinks page
Cold War approach to beating radical Islamists usnews.com

New Items, Friday

Science of Matter, Space and Time Fermilab
Evan Williams' Podcasting Venture NYT reg req
iRiver iPod alternative
MovieLens via RedFerret
Trust Project about web based social networks

New Items, Thursday

Norah Jones new Andilinks page
Sunglasses new Andilinks page
InfoVisual Visual dictionary
VoIP Now weblog/news
AutoLink Disabler Jscript for webmasters
Though if G truly has passed over to the "evil" side they simply could penalize SERPS for running this script. So, "Do no evil" = "Trust me."

SendtoFTP right click FTP utility
blogads about blogs
FilmTrust via RedFerret
Maybe a YASN that doesn't scale well, but worth keeping an eye on.

The Overstretch Myth Foreign Affairs via A&LDaily

An excellent analysis, but a difficult subject for many.

Summary: The United States' current account deficit and foreign debt are not dire threats to its global position, as would-be Cassandras warn. U.S. power is firmly grounded on economic superiority and financial stability that will not end soon.

New Items, Wednesday

Lots of video links today. This routine is going to work well, today I have written a macro to change the HTML from the other format to this one in a single click, so I'll have more time for comments in the future. Today was a very busy day, errands, shopping etc.

TVexe.com broadband TV
Dell's 42" Plasma HDTV
Vimeo like flickr for video
10 Minute RSS Primer threadwatch

New Items, Tuesday

I will be posting "New Items" here. There will be a delay, fresher postings can be found at http://www.andilinks.com/index.htm#n

Info Sector Statistics new Andilinks page
Openwares freeware development
Google Draws Ire with Autolink
Liveplasma music/movie engine, via RedFerret
.htaccess tool
Global Warming der Spiegel via A&LDaily
Competitive Intelligence Res Index ResearchBuzz
TVforus online streams
Web Search Garage Pandia Award

The sound of a pendulum changing directions

There is still some hope that the left and the Democratic party is not actually going to follow Kos and his friends into the "**** you oblivion" that appears to be their goal.
It's possible, just possible that the grand ocean liner that is "the left" may be able to change course before I am forced to jump off. If so this piece may signal that change.
http://www.tnr.com/doc.mhtml?i=20050228&s=peretz022805

I thank The Anchoress for this and the metaphor in my title.

More on "why not a blog?"

In order to be noticed one must be:

1) The first
2) The best
3) Different

1) I can't be the first or even among the first. My first blogger account began in early 2002 and had I pursued it then I perhaps could have been among the first. But I let it lie fallow and I don't think I even have the password to that account anymore, it is defunct. I began Andilinks instead because it seemed like the best vehicle for learning about the internet (and organizing what I learned). It is true that the entire site is the external portion of my study notes. :)

2) As much as I admire my own discipline and abilities I entertain no fantasy that I could be among the best bloggers.

3) Andilinks is different, and I am very good at this different modality. Here I have a shot at being noticed. BTW, Andilinks is doing well with ad revenues, better I suspect than all but the very top ranks of the blogosphere. So compared with blogs, Andilinks is a better business model and much more scalable. And different.

It is also idiosyncratic enough to be very difficult to copy and maintain. Yes, it would be very easy to copy and paste, but to maintain the back end would require a programming effort such that it would be easier to just begin from scratch. Something I may have to do one day myself--with the benefit of having done it once the hard way.

"New Items" working well...

Why not just a blog? Well, blogs are more clumsy than a headline service like "New Items." The home page already gets a lot of built-in traffic which is increasing more rapidly than I had anticipated with the "New Items" feature.

The drawback of less comment space can be overcome simply by linking to this page and commenting here--if need be. But most often I'd prefer not to comment anyway, and that's why a blog seems too clumsy. There is the temptation to do too much with it and "be all things" and I think I'm pushing the edge of that boundary now. I must define the limits of "New Items" more carefully. I cannot add too many news items, just those that I think may be under covered elsewhere. "New Items" should be limited to just useful links and links to useful tools.

RSS syndication for "New Items" would be difficult, perhaps beyond my level of competence. I don't want to mess with coding any more if I don't have to, I want to spend my time surfing, selecting and posting. I'd prefer that people just visit the site and not get it fed to their reader. Maybe that's not realistic for the long-term but it seems to be working now.

Fashion Designers

I just split these pages from two to three and I'm going to be spending a few days this week updating this section, adding links and details. I will probably end up with four or more pages of fashion designers...

I also added an new separate page for Prom Dresses, though really it is just a sub-set of Bridal Wear right now. I will spend some time this week adding more links there too.

Blog-like Feature, New Items

My performance as a blogger has always been spotty and early efforts consisted of too many links and not enough explanation. That is because the "blog" grew out of a page that I used to proof my new selection of links for Andilinks and was never really intended as a blog.

Later efforts added various opinions and the occasional story but I never was comfortable with it. I have a new pen-pal on the other side of the world who may collaborate with me on a blog, but the latest thing I'm doing is a new section at the bottom of my Home Page called New Items.

http://www.andilinks.com/

Alicia Keys

Not as much material on Alicia as for Britney, but enough for a nice selection. I am enjoying this celebrity section, I hope it begins drawing some visitors because I'd like to continue it for a while.

Celebrity Pages

I've completed a Britney Spears Page and while I have been moving in many directions lately this is the first time I've devoted a page to one person. I won't deny that the purpose of this is to build traffic for the site. But building celebrity pages is not boring and in that way it is compatible with the overall spirit of the site even if it is not at all the vision I had at the beginning. I will be adding more celebrities, my choice will be determined by their popularity and the amount of material available on the web.

But this is not a new direction or character for the site, it is simply another section. I will see how well this works, but I intend to get back to e-commerce and women's fashions soon.

Americanism—and Its Enemies

David Gelernter's article in Commentary is one of the most thought provoking analyses of Americanism that I have read. I have always had an interest in American history as an American and I am quite intrigued to read that Americanism is tied to Zionism through the Puritans--with suitable moments of importance given to 1776, Abraham Lincoln, Woodrow Wilson, Harry Truman, and Ronald Reagan.

Viewing current events through this lens will be an interesting exercise during these coming days.

http://www.commentarymagazine.com/article.asp?aid=11901043_1

via Arts & Letters Daily