January 24, 2011

Bash shell script: Scraping and downloading image files from a ffffound RSS feed

I wrote this little script for a friend, as an exercise in bash shell scripting. This script is for OS X.

Step 1:
Install Homebrew - this is a great tool for installing common Unix tools. You should only have to open Terminal, and paste the 1-line installation script found in the link above. Something like this:
ruby -e "$(curl -fsSLk https://gist.github.com/raw/323731/install_homebrew.rb)"

Step 2:
Install wget with Homebrew: type this into Terminal and press Enter:
brew install wget

Step 3:
Save the following code into a text file called "ffffound_sssswiped.sh" and save it into your User/Pictures/ directory:
curl http://feeds.feedburner.com/ffffound/everyone | egrep -o source\ url=\"http://[^[:space:]]*.\(jpg\|png\|gif\) | egrep -o http://[^[:space:]]*.\(jpg\|png\|gif\) | xargs wget -nc -P ~/Pictures/ffffound

Step 4:
Customize! You can replace http://feeds.feedburner.com/ffffound/everyone with your own ffffound RSS feed, or anyone else's.

Step 5:
Run the script: type the following into Terminal, and hit Enter:
bash ~/Pictures/ffffound_sssswiped.sh

You should see the download progress as it scrapes the RSS feed for just the large-format image files. You can run this as often as you want, and it will skip any files you've already downloaded.


1 comment:

  1. This is great, I wrote another version using PHP for a photo frame: