Charlie Harvey

Visualising RSS feeds with Perl and GraphViz::Data::Grapher

The slider requires javascript. Trying to load it now...

A few weeks back I was browsing round Perlmonks as is my wont, when I came across planetscape's post How can I visualize my complex data structure?. In that post, planetscape mentions the GraphViz::Data::Grapher module for doing a sort of visual equivalent of Data::Dumper.

I thought that module extremely cool and (of course) had to think up an excuse to try it out. I have a friend who is working on a network mapping project, so I took a bit of inspiration from what she's doing, and from my recent renewed obsession with RSS to make a toy RSS feed visualizer. Check out the pretty pictures in my RSS visualizations set on flickr!

The script is pretty simple thanks to the magic of CPAN, it first automagically finds all the feeds from a page with XML::Feed's find_all_feeds method. Then it loops through the feeds, fetching them with LWP::Simple, turning them into XML::Simple data structures and finally chucking that structure through GraphViz::Data::Grapher. Now, I could have used GraphViz::XML, but I think GraphViz::Data::Grapher is a bit prettier.

The Code

#!/usr/bin/perl use strict; use warnings; use 5.10.0; use LWP::Simple; use XML::Feed; use XML::Simple; use GraphViz::Data::Grapher; use Readonly; Readonly my $usage => "$0 <URI of page with feeds>"; die $usage unless($#ARGV==0); my $page_url = $ARGV[0]; $page_url = "http://$page_url" unless ($page_url =~ /^https?:\/\//); my @feeds = XML::Feed->find_feeds($page_url); my $i = 0; for(@feeds) { my $xml = get($_); die "Couldn't fetch feed" unless defined $xml; my $feed = XMLin($xml); my $graph = GraphViz::Data::Grapher->new($feed); $graph->{LAYOUT}='fdp'; $graph->{RATIO}=1.618; # naturally print $graph->as_png("feed_structure-$i.png"); $i++; }

What Did I Learn?

  • My pages RSS feed is probably a little bit too big, judging by the visualisation. Heh.
  • Sometimes you have to read the source to get what you want. In my case I had to override the {RATIO} and {LAYOUT} properties of GraphViz::Data::Grapher's parent class to make the rendering a bit nicer.
  • Even something simple like an RSS feed can make a pretty complex looking map. Data are cool!


Comments

  • Be respectful. You may want to read the comment guidelines before posting.
  • You can use Markdown syntax to format your comments. You can only use level 5 and 6 headings.
  • You can add class="your language" to code blocks to help highlight.js highlight them correctly.

Privacy note: This form will forward your IP address, user agent and referrer to the Akismet, StopForumSpam and Botscout spam filtering services. I don’t log these details. Those services will. I do log everything you type into the form. Full privacy statement.