Changes between Version 1 and Version 2 of Projects/Scraper

Timestamp:

Dec 28, 2007, 3:45:29 PM (17 years ago)

Author:

Cory McWilliams

Comment:

--

Projects/Scraper

@@ -28 +28 @@
 {{{
 #!python
-#!/usr/bin/env python
-
 import urllib2
 import MySQLdb
@@ -54 +52 @@
     db.commit()
 }}}
+
+== Scraping ==
+Scraping is reasonably simple as well.  All of the real work is done in the loop at the bottom.  It iterates over files and templates that match that file, finds any matches for the template, stores them in the ''data'' table, and then removes any entries from previous runs.
+
+There are currently exactly three types of templates: ''regex'', ''xpath'', and ''urlregex''.  ''regex'' is obvious.  You supply a regular expression, and any captures are stored as values for that match.  ''xpath'' converts the file to an XML tree using lxml.etree.HTMLParser and then executes the xpath query on it.  ''urlregex'' is just like ''regex'' except that it operates on the URL instead of the file contents.  This was just added as an afterthought to be able to extract data from the URL.
+
+Another afterthought was manipulating the extracted data.  This is done through a python string comprehension and the ''!FancyFormatter'' class, which provides access to multiple named values or does a url join on the base URL and a relative path.
+
+{{{
+#!python
+import MySQLdb
+import re
+from StringIO import StringIO
+from lxml import etree
+import datetime
+from urllib import basejoin
+
+db = MySQLdb.connect(user='user', passwd='passwd', host='host', db='db')
+cursor = db.cursor()
+
+def xpath_search(content, query):
+    tree = etree.parse(StringIO(content), etree.HTMLParser())
+    find = etree.XPath(query)
+    return find(tree)
+
+class FancyFormatter(object):
+    def __init__(self, dictionary):
+        self._dict = dictionary
+
+    def __getitem__(self, item):
+        if item == 'makeurl':
+            return basejoin(self._dict['url'], self._dict['value'])
+        else:
+            return self._dict[item]
+
+    def __str__(self):
+        return self._dict['value']
+
+def add_tuple(meaning, url, value, format):
+    if format:
+        value = format % FancyFormatter({'url': url, 'value': value})
+    cursor.execute('INSERT INTO data (created, meaning, url, value) VALUES (NOW(), %s, %s, %s)',
+        (meaning, url, value))
+
+print 'Scraping...'
+cursor.execute('SELECT url, content FROM files WHERE content IS NOT NULL')
+for (url, content) in cursor:
+    start = datetime.datetime.now()
+
+    templates = db.cursor()
+    templates.execute('SELECT type, pattern, meaning, format FROM templates WHERE %s LIKE url', (url,))
+    for (type, pattern, meaning, format) in templates:
+        if type == 'xpath':
+            for value in xpath_search(content, pattern):
+                add_tuple(meaning, url, value, format)
+        elif type == 'regex':
+            for value in re.search(pattern, content, re.S|re.M):
+                add_tuple(meaning, url, value, format)
+        elif type == 'urlregex':
+            match = re.search(pattern, url, re.S|re.M)
+            if match:
+                for value in match.groups():
+                    add_tuple(meaning, url, value, format)
+        else:
+            raise RuntimeError('Unknown template type: "%s".' % (type,))
+    cursor.execute('DELETE FROM data WHERE url=%s AND created<%s', (url, start))
+    db.commit()
+print 'Done.'
+}}}