Current Crawlers

Well currently Grub is working away on many peoples computers and is filling up the servers quickly with data. According to Jeremie on the mailing list, we will soon get our first download of results becoming available. Not sure how soon though. However apparently Grub isnt the only crawler being used. According to this edit Heritrix is currently being tested. Heritrix is the open source crawler currently being used by for their Wayback machine.

More coming soon.


(ps – want access to post on the blog?? Email me: )


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: