Screen-Scraping vs. API

On occasion we’re asked to acquire data from a site that already offers an API. In almost all cases it’s going to be simpler to acquire data from a site by accessing an API as opposed to crawling. In theory, there should be no need to scrape data from a site if an API to the content is already made available. That said, there are a number of reasons why it still may make sense to scrape a site that also provides an API.

Read moreScreen-Scraping vs. API


I made a boo-boo.  To make it short, if you downloaded an installer from our web site between May 3 and May 5 you may actually be running version 5.0 of screen-scraper even though it says 5.5.  This was a result of an oversight on my part, and my apologies to those affected.  Fortunately for … Read moreBoo-Boo

Screen-Scraping for iPhone, Andriod, Blackberry, and Most Any Other Mobile Device

The Mobile Problem The proliferation of mobile devices has created a problem.  Most web sites these days are designed to be viewed on desktop computers with high-resolution monitors and via web browsers that allow for sophisticated interactivity.  Anyone who’s tried to view such sites on mobile devices with small screens can attest to a cramped … Read moreScreen-Scraping for iPhone, Andriod, Blackberry, and Most Any Other Mobile Device

Using Client Certificates with screen-scraper

We recently added experimental support to screen-scraper for client/PKI certificates.  Some web sites require that you supply a client certificate, that you would have previously been given, in order to access them.  I say this new feature is “experimental” because we’ve only been able to perform limited testing with it.  So far, it does seem … Read moreUsing Client Certificates with screen-scraper in the Wall Street Journal

On September 5th, 2010 we received a call from a reporter from the Wall Street Journal, Steve Stecklow (2007 Pulitzer Prize winner). He was calling to speak with someone at our company for a story he was doing related to our industry. He and I talked for about 40 minutes where he asked a lot … Read in the Wall Street Journal