tail -f carlo.log

Site(s) Redesign Complete!

I’m done with the redesign. Both my tumblelog and this blog now share the same layout and (the most important parts of the) sidebar, adding some much needed consistency.

Also I’ve got rid of the Google Coop search and put the Technorati search to use instead, it’s good enough in my eyes.

The biggest change in functionality is the switch from the built-in Wordpress comments system to Disqus, on both my blog and tumblelog. Naturally, I’ve added a tiny bit of logic to the Wordpress templates so the existing comments are preserved, displayed, and those entries are still accepting new comments (assuming the Comments Timeout plugin allows it). But for new entries, Disqus is used. I figured if I don’t like it, I can always switch back and cram the exported Disqus comments back into WP.

Yeah.

Now if I could use Markdown for new posts instead of Textile, I’d be a happy camper. The WP Text Control plugin is no help, unless you consider making the server drop the connection every time helping. I don’t.

Repeated tips of the hat to Bill Israel for sharing his wonderful Letterhead Tumblr theme with the crowds (CC licensing FTW!). Due to its beautifully simple semantic structure I was able to transform it into a Wordpress theme in about two hours. Thanks, Bill!

In somewhat related news: the Hendromat has launched a new tumbelog last night — Make Games, Not Love, MGNL for short. Go check it out.

RescueTime Hack: Log your meetings and phone calls (OSX only)

Over the last week I’ve been using RescueTime. It’s a neat little service that’s keeping track of what you are doing all day long on your machine (Mac or PC) by logging which applications are having the focus. On the site you can then tag the different applications with different keywords (for example, my Mail.app times are tagged with “work”, “communications” and “email”).

And, to make it actually useful, you can analyse your time spent, graphs and all! Check the product tour on RescueTime.com for more infos and some screenshots. It’s pretty neat. (Yes, privacy concerns, blah blah blah. It’s not the topic of this article, so please keep it to yourself for the moment. Thanks. :])

So far so good.

My problem, tho, is that I spend quite a bit of time both on the phone and in meetings. Naturally, RT won’t track these times, since these are basically off-screen activities. Uncool.

So after digging around in the RT log files I was amazed to learn they’re just YAML files. Eeeeexcellent. I can work with that!

But I wanted something with a GUI, something a bit “cooler” than just a Ruby script. So I took this as opportunity to get a bit into Applescript. Unfortunately, RescueTime’s Mac doodad doesn’t offer an Applescript API, and I had to improvise.

After two hours of playing around and cursing at Applescript’s syntax (they’re all on crack…), I had something sufficiently cool to show for.

So, what is it? It’s a little application that you run everytime you want to log some off-screen time. Basically, it’s just a dialog that allows you to enter a number (i.e. minutes spent), and has some buttons to either log these minutes as “was on the phone” or “was in a meeting”. If you hit “Cancel”, it won’t do anything. (Surprise.)

Let’s say I’ve just spent 15 minutes on the phone, I’ll run the application, enter “15” and hit the Return key — and the last 15 minutes will be logged as phone call in RT. The app will quit once that’s done. Another meeting, another phone call? Rinse, repeat.

Data-wise, it’ll then create a new file in the right format in RescueTime’s log directory (~/Library/RescueTime/Logs/Pending/). On the next data upload RT will pick up the new file, and it’ll appear in your stats as “manual_phonecall”. Your meetings will appear as “manual_meeting”.

Please note: Neither existing RescueTime files nor the RescueTime application will be touched or tinkered with.

Below you’ll find two downloads. The first one is the actual script in plain text. The second is the script compiled as application bundle. If you’re unsure which to pick, you’ll most likely want the latter. ;)

• RescueTime Log Time.applescript

• RescueTime Log Time application (ZIP file)

Have fun.

DISCLAIMER: I’ll take no responsibility for loss of data, hair or life. I am very, very sure there won’t be any losses, but still. It works for me, and I’m sharing. Use at your own risk.

Lifestream Tinker Toy, Powered By Yahoo! Pipes

So, (almost) everyone is totally crazy for lifestreams these days. In case you managed to get around the whole issue so far: a lifestream is basically a big bucket (i.e. web page) where all the updates and update notifications from your blog, your ADD-induced Twitter posts, your Flickr uploads etc come together in one concise way so it’s easier for others to ignore them. Also, you only have one URL to hand out to hot women in pubs because the stream inadvertedly works as a hub page, too!

Joy.

Anyways, I was talking with Hendrik about the various stream services out there, and I figured it’d be fairly easy to build something server-less using a wee bit of Javascript and Yahoo! Pipes. So I’ve spent around an hour to do just that.

The basic idea is that you make a pipe containing all your feeds, and then access the output using Javascript from your page. Add some styling, and voilá.

It’s a bit rough around the edges, but hey, it’s just a prototype. It’s not life- or game-changing in any way, but I’d like to share anyways.

Update 2007-11-21: It’s mostly broken right now since I am playing around with something. Not to worry.

Yahoo! Pipes Tutorial: How To Process HTML Pages

Over the weekend I finally had some time to continue playing around with Yahoo! Pipes. (Turns out that quitting World of Warcraft makes your days longer. Huh. Who’d have thought…) It really is a neat toy/tool.

Alas, it is not without hitches. If you’ve worked with Pipes before, you know that it has a module named ‘Fetch Data’, but unfortunately this module is only able to deal with XML and JSON data. It’s just that it will outright reject anything else. And most HTML pages fall in this category.

Here’s my story on how I managed to make Pipes process run-of-the-mill HTML pages. I couldn’t find anything on this topic on the net, so even though I don’t think I am the first guy to tackle this problem, I want to share what I found out.

A little background. I wanted to build a comic feed for Explosm.net’s Cyanide & Happiness, a comic strip for the …wicked. C&H has an official RSS feed, but it only carries links to the new comic pages. Which is fine and dandy for most people, but I am a busy guy, you know, what with my hectic web-3.1 lifestyle and all, and I really can’t afford to even lose 20 seconds by clicking through to anywhere. (Read: I’m really lazy.)

So I tried to make Pipes to parse the Explosm.net comic pages and produce a Pipes-powered feed that not only had the link, but also the particular comic image for that day.

I started out with the Fetch Feed module reading the Explosm.net RSS feed. I am usually not interested in anything but the comics, so in a second step, I made use of the Filter module to ditch feed items that doesn’t contain the word “comic”.

So that left me with a list of feed items, each with a link, a description, a title and some other metadata.

That’s where the trouble started. Boy, what was I thinking?! The HTML pages are far from perfect, they sure as hell doesn’t validate as XML. Then again — that’s alright! The site wasn’t made for scraping or parsing, it was made to display tasteful weird comic strips, and that’s enough. I like it for that. It’s just that there is no Pipes module to grab a glob of textual data and work with it. (Yet? Please, someone say “Yet”, employing a knowing undertone.)

After a while of trying to beat any of the available ‘Fetch *’ modules into submission, I was close to giving up. But then I remember I had come across a HTML Tidy service a few months or years ago. After a few minutes, I had rediscovered it: W3C’s own Tidy service. It’ll digest any HTML page (you just pass the address), and it will clean it up and return valid a XHTML document, which also constitutes as XML. Huzzah!

Well then, commence plumbing! The third module I used was the wonderful Loop module. It’s special because it will process every item in a given dataset, in this case it’d iterate over all remaining feed items. I’ve told the Loop module to use the String Builder module, there I concatenated the Tidy URL, http://cgi.w3.org/cgi-bin/tidy?forceXML=on&docAddr=, with the item.link attribute, and assigned the resulting string to a new a attribute, item.link_tidy.

Step four consisted of another Loop, in which I would use a Fetch Data module to grab the data from the Tidy’d Explosm.net pages, which had now been transformed into neat and valid XML structures and therefore could be digested by the module. It took me a few minutes to find the exact location of the image element (body.div.table.tr.td.div.0.div.1.table.tr.td.1.div.font.0.content), which I then stored in a new attribute named item.content.

(While we’re at it, in the future I would very much like to have the option to use XPath addressing in ‘Fetch Data’… or any related module, for that matter. Thanks in advance. :) )

Anyways, the next steps are using yet another Loop to concatenate the image HTML with the original item.description, and then another one to rewrite the original image URLs a bit so the images are served through the free and excellent Coral Distribution Network.

(Technically, I might not be stealing from Explosm.net, especially since they allow hotlinking of their comic strips from message boards and pretty much anywhere, but for the sake of this example and story here, and the assumption that a few people might end up cloning the Pipe — yes, I am a dreamer —, I feel a bit better knowing I leave less of a footprint on their image servers.)

Then it was applying a Reverse module to flip the order of the items (a cosmetical issue), and that was it.

So, to sum up this long and rather tedious story: you can process HTML pages with Yahoo! Pipes. It’s just not as straightforward as you would expect.

Well, and if you want to take a look at it: here’s the finished Pipe — it might not look like much since the HTML preview doesn’t show the images, but the RSS output is complete.

Tip o’ the hat to Cyanide & Happiness for mindblowing hijinx, the W3C for the free & open Tidy service, and the CDN for being good people.

Now go and use Yahoo! Pipes. It’s one of the cooler products we (as a company) have released in the last few years, and quite frankly, it deserves praise.

So there.

Quick links: Finished Pipe, RSS Output

Update, 2007-09-19: I’ve slightly adjusted the element addressing in the Pipe to waterproof it a bit. The general idea and order of the modules didn’t change, tho.

Update, 2007-12-07: My technique has been superceded by the offical new Fetch Page module — enjoy! :)

New Flickr API Output Format: LOL

Looks like there is a new Flickr API output format: LOL. Not my work, no idea who built this, but I really like it.

Here’s an example of returned data (?tags=cats&format=lol):

HAI
IM IN UR BUCKETS MAKING UP FORMATS
GIMME PHOTOS FROM EVERYONE TAGGED CATS WITH GEODATA
I CAN HAS PHOTO IMG_2326
ITZ AT http://www.flickr.com/photos/blush_response/1377293729/
INVISIBLE METADATA
LOL
KTHX.
I CAN HAS PHOTO IMG_2329
ITZ AT http://www.flickr.com/photos/blush_response/1378198718/
INVISIBLE METADATA
LOL
KTHX.
I IS BORED
KTHXBYE.

Grab the data as usual, the format attribute is lol, apparently.

So… If you always thought about teaching your cat how to parse Flickr data, now might be the right time to jump in. Progress!