Copia

Despite the kind help of the Rojo people I still can't get the service to import my updated feed lists ('An error has occurred...Failed to import: null...We apologize for the inconvenience.'), so I'm still reading my Web feeds on Liferea for now. One nice bonus with Liferea is the ability to add feeds from the command line (or really, any program) courtesy GNOME's DBUS. Thanks to Aristotle for the tip, pointing me to 'a key message on liferea-help'. I've never used DBUS before, so I may be sketchy on some details, but I got it to work for me pretty easily.

I start with a simple script to report on added feed entries. It automatically handles feed lists in OPML or XBEL (I use the latter for managing my feed lists, and Liferea uses the former to manage its feed list).

import amara
import sets

old_feedlist = '/home/uogbuji/.liferea/feedlist.opml'
new_feedlist = '/home/uogbuji/devel/uogbuji/webfeeds.xbel'

def get_feeds(feedlist):
    doc = amara.parse(feedlist)
    #try OPML first
    feeds = [ unicode(f) for f in doc.xml_xpath(u'//outline/@xmlUrl') ]
    if not feeds:
        #then try XBEL
        feeds = [ unicode(f) for f in doc.xml_xpath(u'//bookmark/@href') ]
    return feeds

old_feeds = sets.Set(get_feeds(old_feedlist))
new_feeds = sets.Set(get_feeds(new_feedlist))

added = new_feeds.difference(old_feeds)
for a in added: print a

I then send a subscription request for each new item as follows:

$ dbus-send   --dest=org.gnome.feed.Reader /org/gnome/feed/Reader \
  org.gnome.feed.Reader.Subscribe \
  "string:http://feeds.feedburner.com/DrMacrosXmlRants"

The first time I got an error "Failed to open connection to session message bus: Unable to determine the address of the message bus". I did an apropos dbus and found dbus-launch. I added the suggested stanza to my .bash_profile:

if test -z "$DBUS_SESSION_BUS_ADDRESS" ; then
    ## if not found, launch a new one
    eval ‘dbus-launch --sh-syntax --exit-with-session‘
    echo "D-BUS per-session daemon address is: $DBUS_SESSION_BUS_ADDRESS"
fi

After running dbus-launch the dbus-send worked and Liferea immediately popped up a properties dialog box for the added feed, and stuck it into the feeds tree at the point I happened to last be browsing in Liferea (not sure I like that choice of location). Simple drag&drop to put it where I want. Neat.

[Uche Ogbuji]

So Brian Williams and Bob Costas are sitting there commenting on the parade of athletes in the Winter Olympics opening ceremony. Just to be sure. It's the parade of athletes. Yes athletes. Hello? Could someone phone Brian Williams and Bob Costas and remind them that it's the parade of friggin' athletes?

The Iranian delegation goes by. The discussion is something like "Iran, a country under such pressure right now. The new president is very controversial. He has called for the annihilation of Israel. He has run a competition sponsoring cartoons about the Holocaust. A lot of anti-Americanism from that country". Yes. Not a single word about the athletes. Helllloooo! Last I checked Ahmadinejad is not in the fucking athlete's parade. Oh wait, is that him over there in the burqa? Hell no! Jeez.

Denmark goes by. It's approximately: "you might have heard of all the anti-Islamic cartoons that started in that country. The Danish delegation has been the subject of much attention. As if security wasn't a big enough issue for the hosts." Again not a word about the athletes. What they're competing in. Where they have the best hopes for success.

The spiel on the delegation from Lebanon was, and I got this almost verbatim: "Lebanon. Used to be called the Paris of the Middle East. But it's been wracked by Civil War for so long. Let's hope they can keep the peace there." WTF? I mean what the fuck? Lebanon is not exactly the ultimate crucible of snow and ice sport athletes. I'd like to know who from Lebanon had the guts to give it a shot. In which sports?

When Canada, France, Netherlands, Russia, Sweden and the like go by, OK. They actually say something about sport. To be sure, even this generosity is framed entirely in the parochial context of the U.S., as in: "Netherlands used to be dominant in short track until recently when they've been upstaged by the United States" or "the lone Ethiopian athlete learned to ski in New York and earned a skiing scholarship to [some US college]."

Disgusting. Just disgusting. Not only is this farce an insult to the athletes, who are not just cardboard proxies for whatever tiny window into geopolitics NBC news opens up for its news audience. It's also an insult to the intelligence of the audience. As if we are not capable of appreciating the glorious diversity of competitors that is, after all, the ostensible point of the Olympics.

[Uche Ogbuji]

Updated. New details added.

I got fed up with Evolution recently. My main beef with it is that it's all about magic on the back end. All sorts of details, such as what you've deleted or what you've marked as spam are flagged using proprietary metadatabases. The persistence of this information also seems to be dodgy, which means that often, when Evolution crashes it forgets such details, and makes you re-delete, re-file and re-flag things. To make this worse, Evolution crashes a lot, and it seems to especially crash after you've done a lot of message deleting and moving. Over time, Evolution has cost me a lot of tedious work (I get a lot of spam and throwaway mail) and I got to the point where I could no longer stand a tool that increased my workload for little gain.

The main reason I stuck with Evolution for so long is that I have used the MH system to manage my mail for a long time. I like the idea of one message per file. Evolution was the only very modern MUA I could find that supported MH folders. I'd used EXMH for a long time, but it's showing its age, and Sylpheed-claws has way too many rough edges.

So it was time for a change on my Ubuntu workstation, and this is just my notes about what it took to complete the switch of mail management systems.

MH to Maildir

In the end I decided to switch to Maildir, which is like MH in its one-file-per-message philosophy, but seems to be supported by more modern MUAs. I also decided to hedge my bets by moving to a local IMAP server to expose the Maildir folders to just about any MUA.

For the main conversion I used Dr. Jürgen Vollmer's mh2maildir. It did the trick very well. All my MH folders were stored in a directory Mail, and I wanted to have the resulting Maildir in .maildir. I invoked the script as follows:

./bin/mh2maildir.sh -r -R ~/Mail/ ~/.maildir

I knew I had no unread e-mail, thus the -r option. Most users will need the -R option. The original MH diectory is left alone, as far as I can tell, but of course I always recommend backing up your mail folders before any bulk action.

Update. You may need to make some structural adjustments in the resulting .maildir folder. Firstly, many MH tools use an explicit folder named "inbox", which is thus converted into a Maildir folder named "inbox". Most Maildir tools, however, treat the top folder as the implicit inbox. I addressed this as follows:

cd ~/.maildir
mv tmp tmp.save
mv inbox/* .
rm -rf inbox

Also, some Maildir tools such as Mutt and Dovecot expect to see folders as hidden filees (with a leading dot). I handled this by renaming folders, for example:

mv Writing/ .Writing/

If you have nested folders, you have to use a dot to separate levels. A folder "eggs" within "spam" is represented by a top-level directory called ".spam.eggs". I had to do the likes of:

mv XML/XML-DEV .XML.XML-DEV
mv XML .XML

Of course leave alone the "cur", "new" and "tmp" directories which make up the internal structure of each Maildir folder.

Dovecot for IMAP

I went with the Dovecot server for IMAP. I just used Synaptic to install dovecot-common and dovecot-imapd, and followed the article "Setting up an IMAP server with dovecot". The trickiest line in setting up /etc/dovecot/dovecot.conf seems to be the default_mail_env.

default_mail_env = maildir:/home/%u/.maildir

Update. I also had to enable imap with the lines:

protocols = imap
imap_listen = *

And for some reason I had to manually start dovecot the first time:

sudo dovecot

After this, the usual sudo /etc/init.d/dovecot restart seems to work.

Thunderbird

The thunderbird set-up was very straightforward. In fact, I can't really think of anything to say about it. Just set it up with localhost as an IMAP server, and you're set.

Finally, updating .fetchmail

I use fetchmail to download my mail from my external IMAP server, and procmail to put it into my local folders. I had to change procmail to work with Maildir rather than MH folders. The changes turned out to be very simple. In my .procmailrc having defined MAILDIR=$HOME/.maildir I changed lines such as

|/usr/lib/mh/rcvstore +Writing

to

$MAILDIR/.Writing/

Nested folders are similar. You use periods rather than slashes for level separators, not forgetting the leading period. Thus

|/usr/lib/mh/rcvstore +Writing/Tech

becomes

$MAILDIR/.Writing.Tech/

Since Mutt/Dovecot don't use explicit "inbox" folders,

|/usr/lib/mh/rcvstore +inbox

becomes just

$MAILDIR/

The trailing slashes are essential. They indicate that the destination is a Maildir folder and so procmail does the right thing with it.

Mutt is useful for testing that all is OK with your IMAP set-up. Use regular slashes rather than periods for folder separators in IMAP urls, for example:

mutt -f imap://uogbuji@localhost/Writing/Tech

[Uche Ogbuji]

So a few minutes ago I got a call. Caller ID says 1-999-999-1234. Already suspicious. I worked IT at a call center and know how easy CID is to manipulate. They could have chosen something less obviously a fake. Anyway I answered for some reason and there was a guy with a strong Indian accent telling me he was calling from the "U.S. Government Grants Department" and that I'd won a $12,000 grant for being a good taxpayer. The only catch was that I must not use the money to do anything illegal. This is a new scam on me, so I asked him a bunch of purposefully silly questions while I googled "government grants scam". I gather from the results that the idea is to get your bank account number for a supposed wire transfer to the account, and they just arrange the wire to go the other way. Surely enough, my phone friend soon asked me for information about my bank and I rang off. I hardly suppose I need to warn anyone about these guys, but for what it's worth...

[Uche Ogbuji]

RobertBachmann stopped by #atom to mention that he'd tried to run an Atom file on the non-normative RELAX NG for the Atom RFC draft (I haven't seen an RNC for the final RFC itself). It failed because he used xml:lang in an atom:name child of atom:author. This contradicts the Atom spec, which says:

Any element defined by this specification MAY have an xml:lang attribute, whose content indicates the natural language for the element and its descendents.

The RNC did not specify this attribute in a couple of cases. The RNC is non-normative, but in this case there is no reason for divergence from the spec. I whipped up an atom.rnc that fixes the bug. Here's the diff from the version I found on-line.

This did set up a discussion between Anne van Kesteren and me. I feel that xml:lang only makes sense for some Atom elements, and that perhaps allowing it on all of them could be confusing. What, for example, does it mean to have xml:lang on the atom:uri child of atom:author? I suppose an outlandish (pun intended) interpretation could be references to localized sites, but that's really the province of the likes of XHTML's hreflang attribute. Moreover, I'm a bit puzzled by the bit from the Atom spec that seems to support my leaning:

The language context is only significant for elements and attributes declared to be "Language-Sensitive" by this specification.

So if it's not significant, why allow it? I think maybe there should have been a split in attribute sets between atomCommonAttributes and a atomCommonLanguageSensitiveAttributes, where the former would omit xml:lang.

Also, I'm used to the convention where xml:lang is used with content models that allow a language-sensitive element to be repeated, providing for multiple language versions in the same document. There are many cases in Atom where this would not be possible. For example, you could not have an English atom:title and a French one within the same atom:entry element. You could get tricky with by using a single atom:entry with type="xhtml" and multiple language versions within the xhtml:div, but this feels a bit constricting.

Anne doesn't mind xml:lang everywhere, and pointed out that xml:lang="" is an option for specifying no language context (rather than language context inherited from parent). I think in the end I could go either way on xml:lang everywhere.

This discussion also made me think of xml:space. This special attribute might get a mention right in the XML spec, but that doesn't mean it doesn't have to be addressed in XML applications. Even in the case of DTD, the spec says

In valid documents, this attribute, like any other, must be declared if it is used.

The same goes for RELAX NG, the conventional schema language for Atom. There is no xml:space to be found in either the normative RFC or non-normative schema, but the rules for Atom undefinedAttribute do allow for this attribute (as well as xml:id and just about any other XML or 'global' attribute). I assume that the intention is for applications to treat this attribute using the suggested semantics in the XML 1.0 spec. I do wish Atom had been explicit about this as is, for example, the XSLT 1.0 spec.

[Uche Ogbuji]

Dare laments Firefox's lack of text area content savings upon crashing. At first I found this strange because Firefox does save text area content in my experience. Then I remembered that I always install SessionSaver. I suspect that's where I might be getting my protection from. It did make me wonder whether XForms content is similarly protected. These days I like to use Chime's XForm document with the FireFox XForms extension to post to copia, and I should test how it handles crashes.

But the main point of this entry is to make a related rant and lazyweb request about Thunderbird. I learned the hard way that unlike Evolution, Thunderbird does not auto-save messages you are composing. That means that my habit of starting drafts and then switching to another task is very dangerous. If I do not manually save the draft and Thunderbird crashes, I lose my work. This is stupid. Evolution would save all compose window content in files named ".evolution-<opaque-id>", and would offer to restore these windows upon restart. If I can't find an extension along the lines of SessionSaver for Thunderbird, I'll have to ditch it. Do any of my LazyWeb friends know of such an extension? Googling and other searching turned up blanks.

[Uche Ogbuji]

Updated based on user experience.

Recently a user mentioned having trouble freezing an Amara app. This question comes up every six months or so, it seems, so I decided to make sure I have a recipe for easy reference. I also wanted to make sure that successful freezing would not require any changes in 4Suite before the next release. I started with the most recent success report I could find, by Roman Yakovenko. Actually, his recipe ran perfectly well as is. All I'm doing here is expanding on it.

Recipe: freezing 4Suite or Amara apps

Grab cxFreeze. I used the 3.0.1 release, which I built from source on Fedora Core 4 Linux and Python 2.4.1). Updated: I've updated freezehack.py to work with cxFreeze 3.0.2, thanks to Luis Miguel Morillas.

Grab freezehack.py, which was originally put together by Roman. Add it to your PYTHONPATH.

Add import freezehack to your main Python module for the executable to be created. update actually, based on Mike Powers' experience you might have to add this import to every module that imports amara or Ft.

Freeze your program as usual. Run FreezePython.exe (or FreezePython on UNIX).

See the following sample session:

$ cat main.py
import freezehack
import amara
diggfeed = amara.parse("http://www.digg.com/rss/index.xml")
print diggfeed.rss.channel.item.title

$ FreezePython --install-dir dist --target-name testexe main.py
[SNIP]
Frozen binary dist/testexe created.

$ ./dist/testexe
Guess-the-Google - Simple but addictive game

In order to share the executable you have to copy the whole dist directory to the target machine, but that's all you should need to do. Python, 4Suite, Amara and any other such dependencies are bundled automatically.

Now back to the release.

[Uche Ogbuji]

Redfoot recently had 'native' scuttering capabilities added to it's kernel. The original motivation was as a testbed to determine some reasonable parameters for a 'scuttering protocol'. That document was prepared in haste, but for the most part, the load function on Redfoot has been extended to provide built-in scuttering capabilities - using that protocol as a guide.

Redfoot provides a framework for loading (and persisting) remotely hosted chunks of executable code (redfoot programs – the current ones are mostly written in Python / KID). The most common context in which scuttering is discussed is the interpretation of FOAF graphs (social networks). However, I found the idea of a network of executable code with the dependencies (on applicaton 'data' and other third party 'bits' of funcionality / code) expressed via rdfs:seeAlso and rdfs:isDefinedBy very appealing as an alternative subtext to the whole 'Semantic Web' idea.

The main point of the scuttering protocol above is the use of a provenance graph (RDF graphs which contain statements about other RDF graphs) which uses a vocabulary to express (and persist) the HTTP headers of remote RDF graphs.

The cached HTTP headers are used to automate subsequent content-negotiated requests to the same resources to effectively mirror an HTTP network of RDF graphs (where the links are expressed by rdfs:seeAlso, owl:import, and rdfs:isDefinedBy) in a local store – applying RESTful best practices.

Below is an example of the statements added after a fetch of the URL http://metacognition.info/profile/webwho.xrdf.

For every URL fetched, the 'scutter links' (rdfs:seeAlso,rdfs:isDefinedBy,and owl:import) are traversed recursively (up to a system-wide maximum recursion depth). Links that do not resolve to an RDF graph are marked in the local cache as “non-rdf” to avoid redundant fetches to URLs known not to be RDF.

Chimezie Ogbuji

I found a nice mapping from Universal Feed Parser to RDF (SKOS,DC,AtomOWL), that Emeka will employ:

Each entry is an instance of (atomOwl:Entry,rss:item)

• The URL of the feed -> an instance of atomOwl:Feed
• Feed - atomOwl:entry -> entries
• entry (link or id as URI) - rdfs:label,skos:prefLabel,dc:title -> entry.title
• entry - dc:description,atomOwl:summary,rdfs:comment -> entry.summary
• entry,feed - dc:creator, foaf:maker -> foaf:Person
• entry.author_detail.name -> foaf:name
• entry.author_detail.email -> foaf:mbox
• entry.author_detail.href is the URL of the author
• entries.tags -> skos:Collection
• entries.tags.label -> skos:prefLabel
• entries.tags.scheme + entries.tags.term (URI resolution) -> URI of skos:Concept
• entry - dc:created,dc:date,atomOwl:published -> entry.published

Chimezie Ogbuji

"Scripting Flickr with Python and REST"

In his latest Agile Web column, Uche Ogbuji shows us how to use Python to interact with Flickr as a lightweight web service.

This Agile Web installment is fairly straightforward. I look at the several Python libraries for accessing Flickr from programs. They range from low level, thin veneers over the official Flickr API to the one higher level, more Pythonic library. And of course there's the obligatory package I just can't get to work.

[Uche Ogbuji]