Solutions: Data Mining and Extraction

Sometimes referred to as text data mining or web site data extraction, extracting data from websites might entail clicking through links, often logging into first, and extracting only the data that is required. After extraction, the data may then need to be checked for accuracy, converted to a different format and then either downloaded to a spreadsheet or passed to other applications.

Using screen-scraper, navigation of complex sites and automated data collection is under your complete control. Once screen-scraper has navigated to the correct pages, web data extraction occurs easily through patterns that can be tailored to target anything on the page, even non-visual elements such as hidden fields, headers, and cookies. Scraped data can be stored in spreadsheets, a database, or even passed to other applications.

Examples of using screen-scraper for data mining and extraction include:

• Extracting mailing addresses from a web site and downloading them to a spreadsheet
• Mining product data from multiple sites to create meta-search sites
• Extracting news headlines to use in RSS feeds