Thoughts, tips, and updates on screen-scraping
Sometimes we’re asked how one might hinder a person who is trying to scrape data from their site. (The irony, of course, is that it comess from people who contacted me to scrape data for them.) The standard answer is that if you’re publishing data for the world to see, it can be scraped. There’s … Read moreMethods to hinder scraping
This one’s definitely a recommended upgrade. It contains a few bug fixes that should remedy some obnoxious behavior you might notice in the previous alpha version. Aside from bug fixes, this version now allows for sub-extractor patterns to be applied in sequence. It’s not something that’s often needed, but once in a while it can … Read moreVersion 3.0.31a of screen-scraper available
Aside from a few bug fixes and other niceties since the last announced alpha, this one now handles file uploads in the proxy server. This isn’t a really common need, but it’s been a hole in screen-scraper’s functionality that, happily, is now filled. Feel free to give it a try and let us know of … Read moreVersion 3.0.28a of screen-scraper available
Thank goodness for alpha versions! That last alpha (3.0.21a) contained somewhat of a nasty bug that would wipe out sub-extractor patterns. Apologies to anyone negatively affected by that (but thanks for helping us test the bleeding edge version!). The upside is that we’ve made a fix for that in version 3.0.22a of screen-scraper. Please give … Read moreVersion 3.0.22a of screen-scraper available
Well, we’ve been a bit delinquent in announcing new alphas. This most recent one fixes a long-standing, but rarely seen bug where extractor patterns could magically appear where they shouldn’t. There are also quite a few other little niceties and bug fixes since 3.0.14a, when we last announced an update. For those interested, feel free … Read moreVersion 3.0.21a of screen-scraper available
Yeah, it’s been a while since we announced an alpha version. This most recent one contains some nice fixes and a few new goodies that you should like. We’re using it internally, and it’s shown to be quite stable. The usual caveats apply before upgrading, though (back up your work!). We’ve also been working on … Read moreVersion 3.0.14a of screen-scraper available
We’ve had requests for this in the past, and I’m happy to report that we’re now able to deliver. You can now invoke screen-scraper from a Ruby script, via a driver we’ve just released. This was cooked up by a former employee of ours (thanks Adam!), who needed the functionality, and graciously donated his code. … Read moreRuby Tuesday
A while back I was contacted by Douglas Hubbard regarding a book he was writing entitled How to Measure Anything. He was interested in finding out more about tools that could automate online data collection, and screen-scraping popped up on his list as one method to go about this. Last week Douglas contacted me indicating … Read moreHow to Measure Anything
Well, this is a topic I’ve been meaning to address for quite a while, and a recent support request on the topic pushed me to finally get it done. What I’ll describe in this article applies to our own screen-scraper software, but would also apply to most any other screen or web scraping software you … Read moreHow to surf and screen-scrape anonymously
Just an eensy-weensy bug fix in this one. Don’t bother to upgrade unless you know you need it. If your screen-scraper was updated from a 2.7.x release, you can go ahead and just update in the usual manner (i.e., select “Check for updates…” from the “Options” menu). If you installed a fresh copy of version … Read moreVersion 3.0.3a of screen-scraper available