published: July 6, 2016, 11:23 a.m.
Weather Underground Webcam with RaspberryPi
In which I explain how to use a RaspberryPi and a $10 USB camera to power a Weather Underground webcam service.
RaspberryPi (RPi) is an ongoing software and hardware concept project by the RaspberryPi Foundation which seeks to create user-friendly customizable development devices. (Basically really tiny computers designed to connect to all sorts of sensors and be powerful enough to run code to process data from those sensors.) Their business model is essentially to sell patented, low-cost, streamlined blueprints to hardware manufacturers, who in turn sell the hardware to consumers. What's special about the hardware is that it is compact, easy to set up, and because it has a small Linux kernel installed, is extremely versatile. This makes it an incredibly powerful development platform for scientists, electrical engineers, internet of things (IOT) developers, and hobbyists. Recent improvements to the design of RPi's chips and boards have allowed for major advances in computing power and efficiency. (For example, the RPi Foundation's website reports that the latest device's computing and graphics processing power is roughly equivalent to that of the original XBOX device.)
I'll admit that given the industry I work in and all the potential uses of the Pi and Arduino in science and engineering, I was extremely late to this party. It wasn't until a coworker mentioned in passing that the Pi ran Linux that I even realized the potential. My first thought was, "I can set up a small, relatively powerful, extremely efficient home server for how much?!?" Thusfar my only regret in purchasing the Pi is that I did not do it sooner, and I often find myself thinking of excuses to buy more of them. Their uses are really only limited by one's imagination.
I bought the CanaKit Raspberry Pi 3 Ultimate Starter Kit 32 GB, a more expensive kit which arguably I didn't need all the frills of, but I went with it because I was pretty excited to try basically everything, and since this is almost exactly the kind of device I envision setting up en-masse in grad school to monitor water bodies for things like temperature, height, turbidity, salinity, etc. and I wanted some practice. This how-to only covers a very small, simple example of what's possible, and I will be discussing other projects involving the Pi in subsequent posts.
I also had a Havit HV-N5086 USB webcam kicking around, although I think I bought it for even less than what it's listed as on Amazon, because I remember paying around $10. It's not a terribly high-quality camera, but for my purposes it's perfect, since I won't really care if weather damages it, as it almost inevitably will.
I already had a fully functional weatherstation connected to Weather Underground, so really all I felt that I was missing was a webcam image from near the site. The goal of the project was to create an automated script to trigger the webcam to take the image, resample it to be a size that Weather Underground would accept, then upload it to their server using FTP.
That meant three things were required for this be a success.
- A command-line tool that would connect to the camera and write the image to disk.
- A python script, which I call
pywxto handle the command call, the resizing, and the uploading.
- And a
cronjob to manage the repetition of the script.
A bit of side story before I start: originally, I wrote this application for Unix/Mac. I had an old MacMini with a 1.42GHz PowerPC processor (yeah, really old) that was just sitting on a shelf doing nothing. The tough part about that (besides the fact that almost no one writes software for PowerPC architecture anymore) was that the computer itself was just so unreliable. It would go hours sometimes without recording or sending any images, for (as far as I could tell) absolutely no reason whatsoever. It may have to do with the fact that it was not running the original operating system. I believe it had Panther, OS 10.3, to begin, way back in the mid 2000s when it was purchased. 20-year-old me, in my infinite wisdom, decided that it would be great fun to install Snow Leopard Server Edition one summer in college, to host and monitor another (non-web-enabled) weather station in my room in my parents' house. Well, it was indeed great fun. But the computer paid the price. The Mini was a bit underpowered and the OS would simply lock up sometimes, presumably due to insufficient resources devoted to a bunch of Mac OS Server tools that I never used. My theory is that these lockups were related to the unreliability problems, although I never proved that. But I could SSH to it and sometimes could get VNC to work, so for the likes of this project it was an OK intermediate solution prior to buying the RaspberryPi. So that's how this project came to be cross-platform compatible. If I'd had a low-power Linux device when I wrote it, it may not have ended up this way, but I'm pretty happy it did as I still have a soft spot in my heart for the Mac Mini.
For Mac, I went with ImageSnap as my command line image tool. It's a decent enough tool and has some good options that can be set in flags. Plus, it's available on Homebrew. But it's not without its issues. The first day I ran this, I happened to check
top -u to see how much of the machine's resources ImageSnap used when running. What I found were like fifteen different
imagesnap instances all with different pids in my
top window. It seemed to me as if there were times when ImageSnap did not kill its process after being called. In response to this I ended up suggesting in the README file that the user add
&& killall -15 imagesnap to their
cron job, which seemed to work for me. It's a little hacky, but it's a fairly safe variation of the
killall call, and as ImageSnap itself seemed to be the issue, I couldn't do much else.
For Linux I chose the fswebcam tool, which is conveniently available with
apt-get from the command line. It's similar enough to ImageSnap that it seems like ImageSnap may have actually taken ideas for some of its flags from fswebcam. That made it easy for me, as all I had to do to create a cross-platform application was test which operating system Python was running on, and modify the command line call accordingly. What's nice about fswebcam is that it creates its own banners. So there was no need for me to add text to the image using the Python Image Library (PIL) like there was on Mac. Instead, I just added the user-specified banner text after the
--top-banner --title flags and fswebcam took care of the rest. One could conceivably add these to fswebcam's config file also. Don't get me wrong, I do love a good one-liner, but the reason I chose to write these into the command call is because I tend to prefer more explicit commands, so after I've forgotten about fswebcam fifteen months from now, I'm not left wondering why something was done or undone with an oversimplified command call.
One thing about my particular webcam is that it needs a bit of warmup time to produce a properly aperture-adjusted image. Warmup time is handled slightly differently in ImageSnap and fswebcam. ImageSnap will instantiate the camera and begin accepting frames all in one fell swoop. That means all you need to do to wait for the camera to warm up is add the wait flag like this:
-w (seconds) where instead of (seconds) you insert a number of seconds to wait before saving a frame. fswebcam can do the same, but instead of working with seconds, it works with frames (i.e. instead of saying "wait 4 seconds before taking a photo" as with ImageSnap, you tell it "skip 40 frames before taking a photo" and you will achieve a similar result). The flag to do that is
-S 40. It took me the longest time to figure that out...I didn't realize why my fswebcam images were so dark until until I read about others having the same issues.
All of the code for
pywx is on GitHub. I did a little bit of an experiment with this script inspired by this StackOverflow answer. Instead of hosting the functions as global in one python file, I opted to have a single runfile with which I can call individual classes in another python file simply by using the command line. So instead of calling
python foo.py, you'd call
~/dir/run_file foo.Bar.baz to run the
baz method in the
Bar class of the
foo.py file, where the
run_file file is an executable python script instantiating whatever method the user chose to run. I'm not sure how I feel about the results. Testing each method was a lot easier this way, but for this size of an application it may just be a lot of extra work and complication. It was a good experiment though, and it did work nicely for me in terms of speedy and targeted development/testing.
The main code of the
pywx project is housed in
pywx/cap.py. It is organized into five main methods under the Actions class:
all tries to run all other methods, and should fail gracefully if something goes wrong. During development, each method could be run individually of all other parts, which as I mentioned made testing easier. Because I was trying to get this up and running in a few hours without posting any malformed images to Wunderground, having the ability to test each individual part was a plus.
cron job was pretty straightforward. Wunderground accepts FTP-uploaded image files at most once per minute. I opted for once every two minutes to reduce processor load (since the RaspberryPi I'm using is also the one you are currently viewing this site from), and still end up with a nice Wunderground time lapse with a lot of data points. With that in mind, the
cron line on Mac or Linux is as follows:
*/2 * * * * ~/pywx/pywx_run cap.Actions.all 2>&1Pretty simple, right? The
*/2means the command will run every other minute. The second, third, fourth, and fifth wildcards all indicate that the script should run every hour, every day, and every month of every year respectively. Then we point to the runfile using
~/pywx/pywx_runand tell it which methods we want to run using
cap.Actions.all. All STDOUT output is piped to null using
2>&1so that the user does not receive notifications every two minutes. I also copy the image to my web server's static folder using a
&& cp /in/image.jpg /out/image.jpg-style addition to the command.
settings.json file was my way of having easily user-definable settings in one place. Unfortunately, this is a little bit of a security risk. It's incumbent upon the user to be able to
chmod 600 settings.json this file to make it only user read/writable, but even then it's a bit of a risk. In theory, as long as their Wunderground Webcams password is unique, it won't matter that much, right? Right??
The reason I like
pywx is that it's simple enough to essentially be a one-file piece of software, but it really fills a niche too. Most webcam uploaders out there are likely a little heavier on resources and have some GUI setup. Because it uses
cron, once it's set up,
pywx will run like clockwork (as long as you're not on my Mac Mini) until the host device is turned off.