The Way Back Machine and Robots.txt

On July 8th, a complaint was filed in the United States District Court for the Eastern District of Pennsylvania, Healthcare Advocates, Inc. v. Harding, Early, Follmer & Frailey, et al. This is such an extraordinary document that I will break with my usual practice of not commenting on complaints or motions. Those who decry the DMCA as an (attempted) tool of oppression will find more than ample support in this effort. Other laws are implicated too, including some I venture to guess most IP lawyers have never heard of at least in the IP context, for example, a Greta Garbo like claim for "Intrusion upon Seclusion." Others, such as the Computer Fraud & Abuse Act and trespass to chattels have become better known recently but are invoked here in a novel way, to say the least. In my opinion (and all this is opinion whether denominated as such or not), the Healthcare Advocates complaint represents a misuse of the legal process.

The complaint appears to be the result of an earlier failed suit brought by Kevin Flynn and Healthcare Advocates (Flynn is the President) against Health Advocate, Inc. and others for various trademark and related type claims. Three opinions in that case should be noted, 2004 U.S. Dist. LEXIS 293 (E.D. Pa. January 13, 2004)(dismissing a number of claims), 2004 U.S. Dist. LEXIS 12536 (E.D. Pa. July 8, 2004)(denying plaintiff's motion to amend complaint and denying defendant's motion for in camera review of the documents in question), and 2005 U.S. Dist. LEXIS 1704 (E.D. Pa. Feb. 8, 2005)(dismissing remaining federal claims and declining to exercise pendent jurisdiction over state fraud claim).

During the investigation of plaintiff's claims, a law firm for some of the defendants utilized the not-for-profit Internet Archive Wayback Machine. The Wayback Machine lets one access archived versions of websites. You type in the URL, select a date range, and presto, you can surf an archived version of the web page in question. It is a phenomenally important archive, useful to people throughout the world, including parties in lawsuits who want to find out what their adversary was saying in the past on a website that has been updated or revised potentially hundreds of times since the events in question. The Wayback machine contains about 1 petabyte of data, more than that in the Library of Congress, even though the archiving only began in 1996. The archiving is accomplished by the Alexa webcrawler.

The Wayback machine is not, however, interested in archiving material website administrators don't want archived, so it has developed a number of ways for people to say, "Please don't collect our stuff." You could telephone the Internet Archive and tell them not to. Or, you can utilize the SRE (Standard for Robot Exclusion) to specify files or directories that cannot be crawled. This is accomplished by a file called robots.txt. (Here is a short article on the Wayback machine and robot exclusion from Wikipedia, and here is a more technical explanation, Robots.txt.) Use of robots.txt is entirely voluntary and many webcrawlers do not utilize it, although the Alexa webcrawler is programmed to obey the robots.txt instructions, and in fact is constructed so as to block, retroactively, files in existence before the instructions were inserted.

Back to the Healthcare Advocates case. The complaint in the earlier suit against Health Advocate, Inc. was filed in June 26, 2003. Healthcare Advocates had been operating a website, www.healthcareadvocates.com since 1998. In July 8, 2003, the robots.txt instructions were inserted. The next day, it is alleged, defendant's law firm tried to access archived Healthcare Advocates website material. In the court's July 8, 2004 opinion, an allegation is recited that between July 8, 2003 and July 15, 2003, 849 attempts were made to access the archived information, of which about 112 attempts were successful. Presumably, all of the material was pre-July 8, 2003 information.

Plaintiff sought to amend the complaint to bring claims against the law firm for this activity, but the court denied the motion. After plaintiff's complaint was dismissed as noted above, this new complaint against the law firm, its members and employees, and the Internet Archive was brought last Friday, July 8th.

There are 12 counts, too many to recite on this already too long blog. I will only talk about one, the DMCA claim, an alleged violation of Section 1201(a): "No person shall circumvent a technological measure that effectively controls access to a work protected under this title." It is alleged that the robots.txt denial text string is a technological circumvention measure and that defendant law firm circumvented it. This claim, in my opinion, is factually and legally wrong. Factually, at least from the complaint, it does not appear that the law firm "circumvented" anything, if by circumvent we mean devised a mousetrap to bypass the denial text string. Instead, it seems as if defendant kept banging on the URL until, for whatever reason, the denial failed to be recognized. This is like going down a row of houses and trying doors to see if they are open. If they aren't you move on until you find one that is. If it is open you walk in, but you certainly haven't circumvented an access control mechanism.

But as importantly, I don't see how the robots.txt can meet the 1201(b)(2)(B) definition of a technological measure: it is a voluntary protocol, operated if at all not by the copyright owner but by a third party, and not all third parties have agreed to use it. The definition of a technological measure is one that "effectively protects a right of a copyright owner ... if the measure, in the ordinary course of its operation, prevents, restricts, or otherwise limits the exercise of a right of the copyright owner under this title."

In the ordinary course of the operation of plaintiff's website only those webcrawlers that had voluntarily agreed to do so would restrict access, and many don't. That can hardly meet the effective protection standard contemplated in the definition. And as a policy matter, plaintiff's theory would encourage good government archivists like the Internet Archive not to use voluntary measures on pain of a DMCA violation. Nor can one say that there was any quid pro quo here: the webpages in question were publicly available long before plaintiff decided to restrict access in conjunction with a much later filed lawsuit. And that is the worst policy of all.