• View
• Edit
• History
• Print

HTML processing

#############################

sgmllib

Mark Pilgrim has a chapter about HTML processing with sgmllib in his book Dive into Python. Processing HTML pages this way is quite complicated so I'd concentrate on a different solution (see below).

Beautiful Soup

Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping.

Example 1

Extract all the links from a page:

from urllib import urlopen
from BeautifulSoup import BeautifulSoup

text = urlopen('http://python.org').read()
soup = BeautifulSoup(text)

for a in soup.findAll('a'):
    if a.has_key('href'):
        print a

Output:

...
<a href="http://www.zope.org/">Zope</a>
<a href="http://www.djangoproject.com/">Django</a>
...