Uncategorized

We’ve recently included libraries for Apache Commons Lang. There is a large number of useful things in there, but I find most use for stringUtils and wordUtils. For example, some sites one might scrape might have the results in all caps. You could: import org.apache.commons.lang.*; name = “GEORGE WASHINGTON CARVER”; name = StringUtils.lowerCase(name); name = … Read moreApache Commons

We are pleased to announce our new coaching program. To help get started, our new users can receive up to two free hours of one-on-one coaching (click here for details). Existing users, receive help planning out your project, solving that one tough issue, learn new techniques and refine your current scraping projects. Purchase hours of training … Read moreLet Us Help You Learn screen-scraper

As Scott pointed out, we were featured in a Wall Street Journal article yesterday.  I thought it might be worthwhile to share my point of view on what information it presents. On the whole, I think the article largely misrepresents the type of work we do.  The tone of the article seems to be fairly … Read moreMy Take on the Wall Street Journal Article

I get a lot of requests for help to configure and run screen-scraper to scrape at an optimal rate.  As is often the case with optimization, it is often as much art as science since the many variables that can affect the speed of a scrape are impossible to catalog.  While these steps will help … Read moreScaling & Optimizing screen-scraper

One of the primary design goals of screen-scraper from the very beginning has been to emphasize extensibility.  We’ve tried to build in a number of features and tools to make screen-scraping easier, but we also realize that we can’t fit it all in.  Features such as the internal scripting engine and the ability to invoke … Read moreScreen-Scraper Annotations for Java

The internet can be thought of as the world’s largest database. This is so, because it is comprised of inter-connected databases, files, and computer systems. By simply typing in some keywords, one can access hundreds to millions of websites containing treasure troves of facts, statistics, and other formats of information on an endless array of … Read moreScreen-Scraping Ethics

Sometimes we’re asked how one might hinder a person who is trying to scrape data from their site. (The irony, of course, is that it comess from people who contacted me to scrape data for them.) The standard answer is that if you’re publishing data for the world to see, it can be scraped. There’s … Read moreMethods to hinder scraping

Today on our support forum we had someone inquire about calling scripts from other scripts within screen-scraper. This has been requested a number of times in the past, and I’ve kind of hummed and hahed about it, not sure if it would be opening a can of worms. Some of our internal developers have wanted … Read moreVersion 2.7.2.1a of screen-scraper available