• Website and Flash Development
  • Content Management Systems
  • .NET, PHP And More!

Screen Scraping Made Pretty Easy

Many public libraries offer a valuable but often overlooked resource – free or discounted museum passes.  Here in Western Massachusetts residents have access to the combined materials of the entire CW/MARS regional system of more than 140 libraries.  Courtesy of our library cards, my family and I have made free trips to the Holyoke Children’s Art on exhibit at the Springfield QuadrangleMuseum, Amherst History Museum, Historic Deerfield, Magic Wings, Brattleboro Art Museum, Smith College Art Museum, Springfield Quadrangle and Boston Science Museum.

I became curious to know the full range of museum passes our library system had to offer.  This information is available via the C/W MARS search engine, but the more than 50 results are spread out over 5 rather verbose pages.  What I was really looking for was a concise, browsable list of available museum passes with links to detailed information. 

A powerful, elegant PHP library called Simple HTML DOM Parser makes constructing such a page relatively easy.  Contributed as open source by S. C. Chen, it features a “Find” function that takes arguments modeled after JQuery’s easy-to-use  syntax.  For instance, this small PHP snippet is virtually all that is needed to dump all the link destinations in a web page:

include_once('/simplehtmldom/simple_html_dom.php');
$html = file_get_html('http://some-site.com/some-page.html');
foreach($html->find('a') as $element)
       echo $element->href . '<br>';

I wrote a program that used Simple HTML DOM Parser to loop through each page of search results from a CW/MARS Library Catalog search query on the term "museum pass." The program parsed each page and extracted all HTML nodes with a class of "briefcitTitle" (find('.briefcitTitle')).  Each node contained a link to an individual museum pass page which was extracted and written back out as a table row. With a couple dozen lines of code I was able to put together the list of museum passes I was after:

Museum Passes Available for Loan at Western Mass. Libraries