Pywikipedia and author identifiers in Wikipedia

I'll admit that sounds like a strange combination of topics, but I'll explain.  I recently saw a mention of VIAF identifiers being added to Wikipedia articles.  That's interesting because VIAF is a union catalog of the world's major libraries' "authority files" (their term for a list of names they control) and using a VIAF id would allow you to bridge to all the constituent catalogs.  It's also one of the identifiers associated with Freebase topics.

When I looked into it, I discovered that Wikipedians had already been adding Library of Congress identifiers, so the VIAF id was just an incremental improvement.  The VIAF additions were supposed to have been done in August, so I wanted to see how many there were compared to the older LC identifiers.

In the past I've written little custom Python programs to query information like this, but I recently came across pywikipedia (aka PyWikipediaBot) which is perfect for tasks like this.  It's got a program which will count template transclusion (ie pages that include a template) as a standard component.  You give it the name of a template, tell it whether you want a list or just a count and it'll query the Wikipedia API to get your results.

$ python templatecount.py -count -namespace:0 Authority_control

Getting references to [[Template:Authority control]] via API...

...

Number of transclusions per template

------------------------------------

Authority_control: 5183

Hmm, that's not as many as I'd hoped.  We selected namespace 0 to restrict our count to the main articles as opposed to talk pages, user pages, etc.  If we replace -count with -list, we can get a list of all the articles. The first time you run any of the pywikipedia tools it'll ask you a few questions to establish defaults for wiki family (wikipedia, wikitravel, etc), language, username, etc, but these can all be overridden on the command line.

The tool allows you to qualify a template name with a parameter name, so we can look at the breakdown between VIAF and LCCN.

$ python templatecount.py -count -namespace:0 \  

    Authority_control/VIAF Authority_control/LCCN

Authority_control/VIAF: 3569

Authority_control/LCCN: 4122

So it looks like there are roughly equal numbers of each and, based on the total count, most templates probably contain both.

One of the things that I noticed when looking at the Template:Authority_control documentation is that Normdaten is an alias for it and looking at the counts shows it's actually used.

$ python templatecount.py -count -namespace:0 Normdaten

Normdaten: 1227

That's interesting.  I wonder what the story behind that is?  Naturally the mind immediately wanders to German Wikipedia.  I wonder if that template is used there and, if so, how frequently.  Fortunately for us, the tool can query a different Wikipedia with the flick of a switch by adding -lang:de.

$ python templatecount.py -count -namespace:0 -lang:de Normdaten

Normdaten: 254890

Wow, a quarter million identifiers! That's more like what I was hoping for.  German Wikipedia is much further ahead in adding strong identifiers to their articles.  They started with a big push in 2010 and have been steadily adding them ever since as you can see from this graph.

Strong idenifiers in German Wikipedia

Next up -- how to actually retrieve template parameter values...

Citizen Curation of Smithsonian Metadata

Museums and cultural institutions have started releasing their catalog metadata under more liberal licenses recently.  Culturally this is a big change for them because traditionally they are hoarders and "guardians of the truth," so it's hard for them to give up control.

There are enormous benefits to sharing their metadata though and, as they're gradually coming to realize, they'll receive back more than they give.  One of the first museums to do this was the Powerhouse Museum who's collection metadata was used as an example for the Free Your Metadata project.

Two big benefits of having your metadata publicly available are that 1) it can be linked to other sources to find interesting connections in other collections, archives, etc and 2) you can get other people to help you clean it up and correct it (bonus!).

The Cooper-Hewitt National Design Museum, part of the Smithsonian, is one of the latest to make its collection data available online for anyone to reuse under a CC0 license.  A couple of weeks ago they released a concordance of their people matched up with entries in other data sets such as Wikipedia, Freebase, the Library of Congress Name Authority File, etc.

They describe it as a pre-alpha release (we'll see why in a minute) and explicitly ask for help in cleaning it up, so let's take a look at how we might help out as volunteer citizen metadata curators.

The looking at the people, we can see that it include "corporate persons" (e.g. IBM) and has not only artists, but clients, manufacturers and anyone who might have a role in the creation or provenance of a design artefact.  There are 4,869 entries in their database, but the concordance only contains the 1,507 which have been matched against Wikipedia.  There are 725 links to Freebase, 222 to the Museum of Modern Art, 113 to the Virtual International Authority File (VIAF, a union catalog of authors by the world's libraries), etc.

Let's look at an example page on their web site.  The page for Frederic Edwin Church includes links to both Wikipedia and Freebase.  If we follow the Freebase link, we see that it has additional links including not only Wikipedia, but the Library of Congress, New York Times, VIAF, etc.  It also has a large amount of structured data in both human and machine-readable form including his works, exhibitions of his work, books about him, his teacher and a student.  It is this network of data that we're connecting to when we attempt to link Cooper-Hewitt entries to this web of links.

Down to work!  We're going to use Refine (aka Google Refine née Freebase Gridworks).  You can download it for Windows, Mac, or Linux here.  When you start it, it will open a page in your web browser where you can create a new project.  We're working with public data here, but it's a desktop app and all the data stays local, so if we were working on private data, we wouldn't need to worry about it escaping into the cloud.  Create a new project from the CSV file and turn off Parse cell text into numbers, dates, ... so that the identifiers with leading zeros don't have them changed.

Let's do a quick analysis of what columns are populated.  The spreadsheet uses the signal value 0 in many of the columns to indicate something is which doesn't have an ID, but uses blank for this in the Freebase column  Let's get convert all the 0s to blanks to make them the same. Select Edit for the first cell in the viaf:id column and delete the 0, then click Apply to All Identical Cells. All the 0s in the column are gone!  Repeat the process for the other columns.

We'll look up the Wikipedia IDs in Freebase to see what they refer to.  Because Freebase includes all the Wikipedia IDs, we can do the lookup by ID instead of name.  Here's how:

• Select Reconcile->Start reconciling...
• Choose Freebase Query-based Reconciliation
• select the a Freebase key in option
• select the this namespace option
• fill in the namespace identifier /wikipedia/en_id which represents the numeric identifier associated with the English Wikipedia article

When everything is filled in, click Start Reconciling.  In less than a minute, you'll have all 1507 IDs linked to their corresponding Freebase topic.  There is another namespace with the ID /wikipedia/en which uses the title from the Wikipedia URL if you ever need to match that way.  Very handy for enhancing someones list of Wikipedia links with additional information!'

There are now two new facets on the left side of the screen.  Looking at the facet labeled wikipedia:id judgement, we can see that not all IDs were matched up.  If we click none to select only those entries which didn't match and check out a few of them on the Cooper-Hewitt site, we see that they are things like disambiguation pages, redirects, etc.  Freebase doesn't include these because they don't represent a single thing.  We don't know for sure, but perhaps the matching process used by the museum only took into account name matches without checking the type of the thing being matched.

Let's delete all these bad matches.  It's possible that the correct link is on that disambiguation page and could be found by a human, but for our purposes it's easier to just put all of these back in the to-b-processed bin and deal with them later.

• Make sure the none entry is selected in the wikipedia:id judgement facet and nothing is selected in any other facet
• in the wikipedia:id  menu select Edit cells->Common Transforms->Blank out cells

As a sanity check, let's make sure that the given Freebase IDs all match their corresponding Wikipedia IDs.  Now that our wikipedia:id column is linked to Freebase, we can look up any other value that Freebase knows.  Let's add a column containing the Freebase Machine ID.

• Select Edit column -> Add columns from Freebase...
• In the Add property input field type Machine ID and choose /type/object/mid
• confirm that the preview display shows the MIDs and then click OK

Now we'll configure our facets so that we can see if the values in our newly added column match those that existed in the concordance.

• In the freebase:id column menu, choose Facet->Customized Facets->Facet by blank
• Select false in the newly created facet (ie non-blank values)
• In same column, choose Facet->Custom Text Facet... and use the expression: grel:value==cells['Machine ID'].value

We see that all 782 non-blank values match.  So far, so good.  Click Remove All to clear the facets.

You may have noticed that we gained an extra 263 rows when we added the new column.  In Freebase a topic can have multiple MIDs if it was merged during its lifetime.  This allows existing references to continue to work.  It doesn't matter which one we pick and since Cooper-Hewitt appears to have chosen the first one, we'll delete the extra rows with others.

• Select (blank) in the wikipedia:id judgement facet.  This column was fully populated before, so this selects all the new rows.
• In the All column menu choose Edit rows-Remove all matching rows

Now let's add a some value. There are 462 entries which have Wikipedia IDs, but no Freebase ID.  Let's fill those in.  

• Select Edit cells->Transform... from the column menu for the freebase:id column
• Use the expression cells['Machine ID'].value
• After confirming the results we can remove the Machine ID column Edit column->Remove this column

As a sanity check, let's see if the names of the Freebase topics match the names in the ch:name column.  We'll use a Custom Text facet on the wikipedia:id column (which has been reconciled against Freebase) with the expression:  cell.recon.match.name.unescape('html')==cells['ch:name'].value

to compare the values in the two columns.  This takes the name of the matched Freebase topic, unescapes the HTML entities (a workaround for a display quirk) and compares the result to the value in the other column.  We've got 898 exact matches, 346 mismatches, and 263 topics which aren't in Freebase for some reason.

If we take a look at the mismatches, we can see a variety of types of differences.  There's Création Baumann for Création Baumann AG and Honeywell instead of Honeywell, Inc.  because the Freebase standard is to not include suffixes like Inc., Co., AG, SARL, etc.  There are minor capitalization differences like Jean le Pautre for Jean Le Pautre and variant spellings like Wenceslas Hollar instead of Wenceslaus Hollar.

Other differences seem more significant and probably deserve closer review.  For example, Henry Dreyfuss Associates clearly isn't the same thing as Henry Dreyfuss himself (although in the world of Wikipedia they could easily be discussed in the same article).  Ditto Chermayeff & Geismar as compared to Ivan Chermayeff.

One way to tackle this is to attempt to reconcile the 600 or so mismatches and non-matches to see if we can match them up with a Freebase topic.  We'll just use the name as given by Cooper-Hewitt and see what we get for matches.  We can then compare the two sets of IDs to see where the problems lie.

• In the wikipedia:id judgement  facet, select both false and (error) (click include in the hover menu to select the second entry)
• In the ch:name column, choose Reconcile->Start reconciling
• Select the Freebase Reconciliation Service
• Select no particular type (another possible strategy would be to first reconcile against Visual Artist then Organization, etc, but we'll take our chances and try to do them all at once)

When we're reconciling against Freebase, we may occasionally run across duplicates such as Edward Fella and Ed Fella.  We can flag these to be merged on Freebase.  This will queue them up for voting by the Freebase community and as soon as three people agree, the merge will be processed.  Of course, we have to make sure that we don't confuse something like the book Edward Fella with the person.

Freebase isn't limited to topics which have articles in Wikipedia, so we'll find some additional matches such as Ronan or Erwan Bouroullec who have an entries with their Netflix and IMDB IDs because of their appearance in the movie Objectified.

Freebase also allows us to create entries which might be missing.  For example, there's an entry for Donald Deskey, but none for his firm Deskey-Volmer.  We can not only create it, but also link the new entry to both founders.

Of course the fact that the names match doesn't necessarily mean that we've got the correct entry.  The singer Beck didn't manufacter the lens for this old camera and Elizabeth I wasn't the 12 year old who embroidered this sampler in 1834.  We've done enough for now, so we'll come back in a future installment and try to clean up more of this.

The step for now is to publish our improvements so that others can take advantage of them.  We'll export our improved data as a CSV from Refine and commit it to our fork on Github.  If you're rather start from where I left off, feel free to clone my copy of the data and improve it some more.

In future posts, we'll look at some additional cleanup and enhancement strategies as well as examine some interesting things that we can do with the interconnections that have been created.

You can discuss and provide feedback on Google+

First thoughts on Google acquisition of Metaweb

Yesterday Google acquired Metaweb, owners of Freebase, for an undisclosed price in a cash transaction which has already closed. The sixty or so employees moved out of their old offices Friday afternoon and will be starting in the Google SF offices on Monday. I'm sure everyone is relieved to be staying in San Fran rather than having to trek down to the Googleplex.

This follows Google's acquisition of ITA for $700 million at the beginning of July which will not only bolster their capabilities in the travel vertical, but also includes the Needle database and Thread query language technology as well as some back end web scraping technology to harvest data to feed it. (I should do a separate post on Needle based on my notes from their presentation at the Cambridge Semantic Web meetup.) It'll certainly be interesting to see how these two new acquisitions fit together with existing efforts like Google Squared (which already uses Freebase). See for example these views of Kurt Vonnegut's books on Squared and Freebase.

Google's director of product management for search, Jack Menzel, wrote in the Metaweb announcement that they are interested in enhancing search through a "deeper understanding" (i.e. "semantics") of queries and web pages. Of course the Semantic Web folks immediately claimed the news as validation of their decade of work, but I don't think it's that simple. It'll be some time before it's clear what Google was after with this acquisition and how they'll use it.

What are some of the things that Google might have been interested in?

People - Metaweb has some bright engineers working in a variety of areas include their proprietary graph store ('graphd'), data mining, machine learning, semantic web, alternative UIs, etc. They already hired one of the graphd engineers a few months ago and may have decided to get the rest of the engineers in one go instead of piecemeal.

Technology - There are a number of interesting technology components, some visible and some not:

• graphd - their home-grown graph database
• Metaweb Query Language (MQL) - a JSON-based query-by-example style query language
• Acre - a server-side Javascript application development environment and hosting service
• Wikipedia import pipeline - extracts data from infoboxes and text from articles
• entity reconciliation - backroom Hadoop based technology used to reconcile data sets and do graph merges

Patents - Metaweb has a number of patents and patent applications which could be of interest to Google. This post contains a list of some of them. They range from early Hillis patents covering the concept of a "meta" or "knowledge" web to more recent ones on graphd technology.

Bing chaos - Microsoft bought Powerset a couple of years ago and uses the technology in Bing. At the time Powerset used Freebase data. Perhaps messing with prime competitor in search held some attraction for Google.

Freebase - Freebase is Metaweb's collaboratively maintained data wiki which was bootstrapped with Wikipedia data, but now also includes information from MusicBrainz, Open Library, and a number of other public domain data sources as well as cross-links to less liberally licensed databases like IMDB, NNDB, NY Times, etc. Although much of the data is available in their data dumps, not all of it is and many interesting analyses can only be done on the full data set.

There are some interesting views in the comments posted on Techcrunch article. Read/Write Web and GigaOM also has a pieces. I agree with the view that this was likely a relatively cheap deal that went at a low multiple of the $57 million that Metaweb had raised. It's a good deal for both parties because Google got good people and good technology at a cheap price and the VCs got an exit for a company that had yet to figure out a business model without having to pump more money in to sustain them until they did.

Time will tell what impact this will have on Freebase and, more generally, open data and the semantic web communities. Google said that it plans "to maintain Freebase as a free and open database for the world" as well as "contribute to and further develop Freebase," but this could be done at broad range of investment levels with a corresponding range of outcomes.

From a personal point of view, I'd like to see Freebase survive not only because I've contributed 1.4 million facts to it, but because I think its model of collaborative schema development and strict reconciliation has some advantages over the distributed "anyone can say anything" model which is more popular in the academic/W3C Semantic Web space. I also think the combination of machine-based and human reconciliation has huge potential that Metaweb had only barely begun to scratch the surface of. If Freebase withers, it'd be tempting to recreate it. The barrier to entry is much lower with today's technology than it was when Metaweb was first starting.

I've got a lot of ideas for synergy among Google, Metaweb, and ITA as well as some thoughts on the implications for current Freebase app developers, but this is long enough, so I'll save those for separate posts.

Thoughts on Metaweb business strategy

Metaweb hasn't announced its new strategy yet, but supposedly will soon, so I'm writing down my suggestions in advance, so we can compare and contrast when it appears. Just to be clear, this is not based on any insider knowledge of any kind and does not represent the views of Metaweb Technologies Inc.

The Metaweb (or Freebase) business strategy has always been a bit of an enigma. They said they were building "The World's Database" and would charge for something later, although it hasn't be clear what.

So what would I do? Here are some thoughts (on how to develop the strategy, rather than the strategy itself):

• Hire (or promote) a Director of Product Management - Not because that's what I do, but because, while they've had good product management in individual areas like their custom app dev environment, they've been hugely stovepiped and don't appear to have an overall product strategy. The product strategy is clearly going to be driven by the executive team and board in a startup, but someone has to be in charge of focusing the discussion in a way that will produce a concrete and implementable strategy, implementing that strategy, and then revising it based on real world customer feedback.
  
• Focus - They've done everything from their own database engine and query language (arguably a competitive differentiator), to their own bulletin board system (definitely not!) to a complete development environment with its own version control. A startup can't afford the same expansive vertical integration strategy that an IBM or HP pursues.
  Focus is key. They need to focus only on those things which are absolutely critical to success and survival. The generous initial funding ($57M to date with a $42M tranche two years ago), may have actually been a curse in this regard.
• Holistic view - Metaweb appears to consider their various software components, their data integration efforts, the resulting data, their volunteer community, and their (potential) commercial customers as independent things which can be optimized separately when they're all inextricably linked, to one degree or another, to each other. It doesn't matter how pretty widgets are if, when I link to Boston from my family-oriented site, the default page shows it as the filming location for the porno flick Slave Workshop Boston.
• Customer Engagement - The only place to tell whether you're winning, losing, or standing still is in the marketplace. More customer involvement is critical. Both to refine product & service requirements as well as to generate design wins that can be used for marketing.
  
• Developer Ecosystem - A vibrant developer community is critical to success. Building this means not only providing the right libraries and tools, but recruiting the developers, training them, and making them successful. This doesn't mean huge corporate machinery is required, but it needs to be a dedicated, ongoing goal for someone. If you look at successful developer programs, non-code assets and processes are at least as critical as the raw developer tools. The business side can't be ignored either.
• Evangelism - Most or all of the marketing staff was apparently let go in late 2008/early 2009 and marketing seems to have been an occasional, part time effort of people with other jobs since then. That doesn't work. Metaweb is, at its core, an engineering company and most engineers have a severe allergy to marketing, but, having done a lot of both marketing and engineering, I know each is critical. They have a technical product set with new concepts in an emerging market, so it's going to be a very technical sell, but it's still marketing. Someone needs to have it as their real job (and get measured on it).
• Standards strategy - Metaweb has never said anything about what their standards strategy is or how they see their technologies relating to thos of the W3C. There's certainly a lot to dislike about some of the W3C choices, but an ugly standard is still a standard. Metaweb did implement RDF publishing support last year, but they need to say more about their long term strategy.
• W3C/Semantic web community - Perhaps the W3C is just naturally opposed to any type of commercialism, but establishing a better relationship would be useful to both parties. Having someone of Tim Berners-Lee's visibility diss you at a venue as prominent as TED 2009, where he completely glossed over Freebase's role as one of the largest publishers of linked data, isn't good.
• Open Source - The company has a number of open source projects, but doesn't talk much about its open source strategy. At the very least, it should claim credit for the things it does and have an easily accessible list of open source projects it contributes to.
• Brand - They've finally realized just how misguided the choice of Freebase was (it's the only Google Alert where I need to add -c*caine to the search terms) and appear to be backing away from that brand name, as well as its associated garish orange livery and flag waving rhino logo. While there's a good case for using a single brand for both a startup and its products, I'm not sure Metaweb is the right brand since it has generic meanings and usages as well. I'd investigate establishing a new brand for the product family.
• Human/machine synergy - I put this last, because it's not a short-term thing, but it represents huge potential for the future, in my opinion. It's an area that Metaweb is uniquely positioned to exploit, which makes it all the frustrating that they haven't made more progress on this front. The synergy between machine-based data reconciliation processes and crowd-sourced processes could create a virtuous feedback loop where machines do the drudge work and humans decide the edge cases, in the process providing training data to refine the classifiers and info extraction algorithms. They've only taken the smallest baby steps so far, but I believe this area has huge potential for those who learn to exploit this synergy effectively.

Freebase Gridworks data curation and cleanup tool

I've been alpha testing the Freebase Gridworks tool from Metaweb, but haven't been able to talk about it until now. Since they just announced it, I guess it's no longer a secret.

Research scientist David Huynh has been interested in collective data operations since his days at the MIT CSAIL Simile project. You can see collective editing in this 2007 Potluck screencast. Jon Udell called this "stunning." After David moved to Metaweb, his 2008 Parallax demo showed the power of collective operations for browsing Freebase data (and UCG's DERI group forked a SPARQL version called SParallax).

The Gridworks tool is another riff on that same collective operations theme, but this time focused on data cleanup and reconciliation rather than mashups or browsing. There's a lot more to it than what you see in the screencasts (and, naturally, some limitations which are glossed over as well), but while it's still in testing I'll reserve any detailed discussion of features. Suffice it to say though, that the anticipatory buzz in the Twitter-sphere is justified. What remains to be seen is how well they'll follow through on completing the tool, as well as integrating it with the various types of data sources & sinks which are of interest to users.

From a selfish point of view, I'd like to see people use tools like this to contribute to the availability of cleaned up public data sets rather than just using it to clean their private data silos. Of course, convincing people to do that is a much bigger problem -- one which the whole Linked Data / Semantic Web community has yet to come up with a compelling answer for.

The Ellerdale Project - new semantic search/trends

The Ellerdale Project (@ellerdale) has just emerged from stealth mode with a couple of discreet tweets. Art van Hoff (@avh) and collaborators have been working on this for a while now and I've been very curious to see what they come up with.

What they've revealed so far is a semantic search engine which builds on Freebase to search for topics instead of keywords. If you haven't heard of it, Freebase is a semantic general knowledge database that takes Wikipedia and makes it more structured allowing easier automated processing (as opposed to human reading). Zing, van Hoff's previous gig, used Freebase , so he's no newcomer to its capabilities. It'll be interesting to see what he does with it.

Ellerdale is indexing the web too, but a lot of their focus seems to be on Twitter. They map hashtags to topics, keep track of trending topics, and show a real-time stream of relevant tweets. No hint as to what their business model might be, but one might guess that it'll be advertising based.

In addition to their web app, Ellerdale has a simple RESTful API which exposes some of the inner machinery for reuse in mashups. For example, here's everything they know about Angelina Jolie. If you look at the JSON results, you can see links back to Freebase, Wikipedia, and the New York Times, as well as a bunch of categories which appear to represent the union of Freebase types/properties and Wikipedia categories. The API covers the basics, but that's about it. For example, there's no way to twidle any of the knobs and dials that control how it determines topics are related. No API key required. No word about quotas.

The Ellerdale IDs look a little like GUIDs, but they're more like serial numbers. They start at #1 with the basics like English Noun, Verb, Adverb, Adjective, then progress on to Person, Male, Female, Date, String, Hashtag, and Category.

Beginning at 0000-1000, they've loaded all the words from the Princeton WordNet corpus including adverbs, verbs, adjectives, and nouns, although the API doesn't expose the synsets, if they've got them loaded. Whatever natural language processing that WordNet is being used for is not exposed in any native way through the API -- just its results.

After that, we've got the full hierarchy of Wikipedia categories from 0010-0000 to 0019-0001. Around 0080-0000 the topics/concepts themselves begin including some rather esoteric stuff like postal codes from Freebase which don't exist in Wikipedia and after that are the hashtags like 00fa-0001 #mala

Although I'm guessing that they get most of the Wikipedia content by way of Freebase, they appear to have some type of side channel to get fresh topics directly because they have things like the Motorola Droid (#droid), but without any link back to Freebase (or Wikipedia for that matter).

That it's it for a first look. I'd love to hear more about the project from anyone who's got info to share.

Breaking the 1 million barrier

I finished loading the updated National Register of Historic Places database into Freebase last week. In addition to containing the latest data released by the National Park Service, combined with the latest Wikipedia articles, this run created new topics where Freebase didn't have existing ones. You may remember that the initial run focused solely on reconciling existing Freebase topics.

Freebase should now have a complete copy of all National Register of Historic Places entries which are of International, National, or State significance. The Local significance listings still used the old strategy of only reconciling existing topics.

Below is a summary of the before and after counts. We picked up 4,535 entries which had either been added to Wikipedia, added to the Register, or both. On top of that we created another 20,553 entries, bringing the grand total to over 35,000 listings.

	Starting Count	Existing Topics Reconciled	New Topics Created	Ending Count
International	0	1	10	11
National	2010	699	4386	7095
State	2423	1121	16065	19609
Local	5978	2627	92	8690
TOTAL	10434	4535	20553	35518

Each topic contains a fair amount of information, so the entire load amount to about 750,000 "facts" (or "triples" in RDF-speak), bring the total number of facts that I've written to Freebase to over 1.1M. Unfortunately, their "tallybot" which does the nightly updating of totals has been broken for a while, so I'm only getting credited with a paltry 300K.

The one remaining loose end is to try and do a better job of reconciling the architects/builders and what the Park Service calls "significant people" associated with the listing. This will require human vetting of a queue of tasks, so it'll require some additional infrastructure to be put in place before I can set people loose on working on it.

Featured Freebase app & base - US National Register of Historic Places

Speaking of Freebase, they've featured some work of mine that I never mentioned, so I suppose I should talk briefly about it.

Back at the end of 2008 I decided that after year of casually following Freebase that it was getting interesting enough to invest some time in learning it in a little more depth. Of course the only way to do that is hands-on, so I needed a project. I didn't want to start with an idea that had commercial potential (they're secret!) and I've got an interest in old places through my genealogy hobby, so I decided to load up the U. S. National Park Service's Register of Historic Places database. The source database is in dBase format, so grabbed a Python module to read it and started playing around with loading it into Freebase. Data reconciliation between two slightly crufty databases is a non-trivial issue, so I played around for quite a while on Freebase's sandbox before I was happy with the results and was ready to load it up on the production database.

Of course shortly after I got it all loaded the NPS released a new version of the database, so now I need to go back and update everything. That's OK though, because the first time around I'd only used the data to add types and properties to existing topics in Freebase (still over 10,000 topics with 100K+ facts). I hadn't created any new topics from scratch. This will be a good opportunity to load the entire database, at least to some level of significance (perhaps National and State, but not Local).

Another little project I did for Sunshine Week 2009 was add the Congressional Biography IDs (aka Library of Congress THOMAS IDs) to all the U. S. politicans. This ID is use in the online versions of all the bills that go through Congress, so is an important unique identifier.

Finally, another project which was just mentioned in the Freebase blog is my very first, very primitive Acre app, Untyped which can be used to find topics containing a specific keyword in their name which have no type assigned to them. Freebase is working hard to get as many topics as possible typed, so this tool can be used to help with that. Most of my other Freebase work has been done in Python, but this uses their new hosted templating engine. It's still a little rough around the edges, but has been improving a lot. Because it's hosted, you don't need to worry about running things on the Google App Engine or another hosting service.

More fun Freebase stuff in the pipe... Stay tuned!

Freebase Hack Day - Sat. July 11 San Francisco

Just two weeks until the Freebase Hack Day that Metaweb is running at their San Francisco headquarters. It's free and will feature unconference style discussions/presentations as well as general hacking. Read more about the goings on in this blog post. Although it's free, space is limited, so you'll need to register with Eventbrite (when it comes back up from its upgrade).

This is the second Hack Day they've held and I'll be in attendance for this one, so if you're going, give me shout.

Google Summer of Code 2009 (GSoC2009)

If you know any students who are interested in open source software, the Google Summer of Code is a great opportunity. Encourage them to apply. The application period is open now and ends April 3.

One thousand students will be paid $4500 by Google for a summer of working on open source projects and will be mentored by experienced open source developers. To my mind, the experience and mentoring is almost more valuable than the case (although obviously that varies greatly depending on the economic situation of the student).

If you look at the list of projects, you'll see that there's something for every taste. Projects range from low-level bit banging in C on bare iron to bioinformatics to games to a wide variety of so-called "social" apps in a wide variety of different programming languages. Students and mentors come from almost one hundred different countries as well, so there's an enormous amount of diversity on that front as well.

I've been a mentor for three of the four years the program has been in existence (2006, 2007, 2008) and last year had the satisfaction of seeing one of my original students become a mentor himself. It's a lot of work, but very satisfying. Unfortunately my project won't be participating this year due to a combination of cutbacks at Google (about 10%) and a desire to rotate in new organizations, but the ArgoEclipse team would still love to mentor any new folks, students or other, who are interested in getting their feet wet with open source development.

Freebase, open government, and enumerations

I'm preparing a short series of articles about Freebase, but Raymond Yee had a question about something I was working on over the weekend, so here's a quick hint to help him along.

What he calls "keys" are called "enumerated properties" in the Freebase documentation and there's an article on how to set them up. Unfortunately, the schema editor was broken when I was working on the National Register of Historic Places database schema, so I had to resort to reverse engineering things from the Explore view (accessible by pressing F8 on any page and scrolling to the bottom of the page) and then modifying the schema's property type by hand using their MQL query language. You can see the end result in the schema where item_number is typed as an enumeration.

There's also a good article on how to create a URL template that I used successfully to link to the original application submissions. For the Congressional Bioguide, it can be used to link back to the original biography.

Coincidentally and independently from Raymond's project, I was actually working on loading up all the Congressional Bioguide ID's last weekend because they are used in the XML form of legislation on THOMAS, which is run by the Library of Congress. I decided to take a slight detour to write a little name parser and Freebase name queryer in Python, so haven't actually gotten around to loading the IDs yet. One of the biggest problems in working with Freebase is reliably resolving personal names. They typically only have the main name that was used as the Wikipedia article name. There's really no telling what name form the article's editors will have chosen and even though the full name and some aliases are often identified in the opening sentence of the article, Freebase doesn't import this information from Wikipedia.

Mobile Internet World was tiny!

I swung by the Yankee Group's Mobile Internet World show in Boston yesterday to check out what was going on. Google's Rich Miner, the head of the Android effort, was doing a Q&A at 2 pm, so I figured I'd see what folks were asking him and check out the vendors on the show floor.

The show was tiny! Boston has two convention centers and when I arrived at the BCEC, things were so quiet that I was worried I was at the wrong one. I eventually wended my way to the back corner where the show was being held and found the show floor. There were only a couple of dozen vendors present and only three of those had anything other than the tiny pipe and drape style displays. I bumped into Derek Speed who I knew from Digital Semiconductor showing off the Moblin SDK for Intel, which was fun, but other than that it was pretty disappointing.

There were perhaps 80-100 people at the Google Q&A (or about 25-30% of the turnout that they had for their developer day in Cambridge last year). In addition to Rich Miner, they also had Justin Mattson from their developer program there to answer questions.

Nothing earthshaking was revealed in the Q&A. Rich said that the early reviews from the media were at the high end of the range of what he thought was possible. He was also pleased with the T-Mobile pre-sales. As for futures, he thought they could do a better job with integrated search (phone + web) and multi-modal search (voice), but didn't say anything about when/if that would be improved.

One thing that I missed before was that the App Market popularity would be computed on both downloads and usage, so they must be uploading usage stats. Rich said there was no plan to expand the app market to content types other than apps because it wasn't a profit center for them.

If they have the show again, I'll definitely look more carefully at the exhibitors list before deciding whether it's worth even an hour or two.

Building Google's Android on 64-bit Linux

This week Google released the sources for all the open-source components of Android including the kernel, the frameworks, the SDK, and most, but not all, of the bundled apps.

My laptop is a dual boot system that mostly runs Windows, but Linux and OS X are the only supported Android build environments, so I fired up Linux (64-bit Ubuntu 8.04 Hardy Heron). Things don't quite build out of the box for 64-bit Linux, so here are some tips to get you going.

In addition to the dependencies listed at http://source.android.com/download#TOC-Linux you'll also need the following:

$ sudo apt-get install ia32-libs g++-multilib gcc-multilib lib32z1-dev lib32ncurses5-dev

There's an additional problem with a missing softlink to the 32-bit version of the X11 client library that you can resolve as follows:

$ cd /usr/lib32
$ sudo ln -s libX11.so.6 libX11.so

It really ought to be included in one of the standard packages, but doesn't seem to be on Ubunto (the shared library itself is included in ia32-libs however).

Follow the rest of the directions on the Android site to install their repo tool (and git if you don't have it), fetch sources, and then do

$ cd ~/mydroid (or wherever you put things)
$ make sdk

to build your own shiny version of Android from scratch. This took over an hour on my laptop, so be patient. When it's finished, you should be able to change to the results directory and running the freshly built emulator

$ cd out/host/linux-x86/sdk/android-sdk_eng._linux-x86
$ tools/emulator

You should see an Android emulator window pop up on your screen running your freshly built code.

What's wrong with UML?

The catalyst for finally starting a blog was Kenn Hussey's post about What's Wrong with UML... I wanted to contribute to the discussion, but my comments would have been way too long for that little tiny comment box.

Apparently the Object Management Group (OMG) has asked its members to come to the next meeting with a list of their top three things which are wrong with UML. Wow! Where to begin?

Without meaning to criticize Kenn, because I think he's done more than almost anyone to help open up UML and the OMG process, I think his list is made from much too much of an insider's perspective. Here's his list:

1. Un-intended inheritance
2. Undefined namespaces for standard stereotypes and primitive types
3. Inability to create a usable XML schema for UML

Let's take a few giant steps back here to get a little bit better view of the problem from the point of view of a UML user. As someone who's spent years working on an open source UML editor where I actually have to deal with users who are affected by this problems as they try to do their work, let me give you my list:

1. Lack of Interoperability
2. Incompatibility
3. The OMG

Let's take a closer look at each one.

Interoperability - The dirty little secret of all the UML tool vendors is that there simply is none -- and no one cares. I could go on about XMI this and MOF that, but let's start with the basics. UML is a visual/graphical modeling language. There is absolutely no way to interchange the graphical representation. WTF? With a little bit of luck, a consultant who's got a bag full of XSLT scripts, and two tools that don't stray too far from the spec, you might be able to get the semantic contents of your model across from one tool to another, but what about all those diagrams that you spent days and weeks slaving over? All completely gone. You'll need to redraw them from scratch. Sure, a spec for UML Diagram Interchange was promulgated (six or seven years after the initial UML specs were released), but no one's implemented it and the Eclipse folks have said that they consider it unimplementable so they're going to work to get their own standard adopted instead.

Even if we just look at the semantic contents of the model, as serialized in an XMI file, interoperability is very dodgy. Different tools have different bugs in their XMI reading/writing code which don't get fixed, vendors use proprietary extensions, and different tools implement different versions of the spec, but the biggest problem is that there's no systematic focus on improving interoperability. It's been broken so long that people don't expect it to get fixed. Each customer is own their own to fight with their own specific vendor and the vendors believe they have no incentive to fix the problems since it helps guarantee vendor lock-in. (They're wrong in their belief that this is to their economic benefit, but that's another story).

Compatibility - I mentioned different tools implementing different versions of the spec. Why is this important? Because the OMG has paid absolutely zero attention to compatibility between versions. UML 1.5 is the only version which was a pure superset of the previous version. In every other release, they've made incompatible changes at multiple levels of the specification. Not only have they changed the language itself in incompatible ways, but they've incompatibly changed the metamodel that's used to define it and the file format that's used to serialize it. Two tools which both implemented UML 1.3 wouldn't be compatible if one serialized using XMI 1.0 and the other serialized using XMI 1.1, let alone if one implemented UML 1.3 and the other implemented UML 1.4

This has an obvious negative effect on interchange/interoperability, but it has impacts throughout the modeling ecosystem. Vendors waste engineering effort reimplementing things just to conform to the new specification. Customers have to upgrade tool chains in lock-step and may not be able to upgrade at all if even a single vendor lags. Downstream standards groups working on modeling standards for bioinformatics, finance, or health care have to either freeze on an old standard as their basis or let the incompatibilities ripple downstream to their consumers.

UML 2.x deserves a special mention here because it is such a massive dislocation in the time/space continuum. In addition to the normal changes in the layer above (MOF) and the layer below (XMI), fundamental concepts of the UML itself were completely thrown out and reinvented for UML 2.x. These changes were so extensive that the OMG didn't even attempt to produce a change-barred spec or keep a comprehensive list of them. Instead entire sections will have a tiny little note at the end that says something like "This concept from UML 1.x has no equivalent."

Object Management Group (OMG) - This is probably going to seem harsh, and since they're local, I may even have friends working there, but I believe both of the problems above, and many of UML's other problems, can be traced back to the OMG and how it works. I'll reserve detailing the problems for a separate post, but they can be summarized as:

• Industry trade group, not a standards body
• Don't eat their own dog food
• Closed to participation by individuals, (most) users, and open source groups
• Too slow
• Too fast (ie not careful enough)

More on the OMG in another post.

As far as UML goes, there are any number of problems which could be enumerated, but if the interoperability nut could be cracked, they could build the momentum to address the other problems.

UML has been around since 1999 and its whole raison d'être is to be the lingua franca of the modeling world. Why not make it a goal to actually achieve this aim before the 10 year anniversary rolls around?

Obligatory first post

I put up my first web page back in the 90s, but I've resisted blogging for years, partly because I'm anti anything that smacks of being hip and trendy, and partly because I was put off by much of the style of the so called "A-list" bloggers -- you know, the ones who post six times a day, consider their words to be equivalent to those engraved on the tablets handed down from the Mount, and write articles which consist solely of things like:

"This is cool."

Cool in what way? Rad windsurfer cool or nerdy cool or script kiddy kewl or ...? You really couldn't write five words to describe why I should follow that link? Having been a software architect for much of my career, I'm pretty familiar with the whole "Terse == Knowledgeable" theme, but I don't buy into it. To be fair most of the tech blogs, at least the ones which aren't about blogging itself, do better than the standard A-list genre.

Dan Bricklin's blog is one that I've followed for years and is pretty close to what I aspire to. He only writes when he's got something to say and the posts are spaced out enough that I can keep up without having to check three times a day. If I've got real work to do for a day or three, I won't have to catch up on eighteen new posts. It helps that he's interested in a similar range of topics -- software engineering, the business of software, intellectual property, open source, photography, etc. Brough Turner has a similar blog in different technology space, principally focusing on mobile, but more generally all types of communications. One thing that they both do well is tie technology to its business and social context so it's not a pure geek fest.

If you catch me backsliding by posting drivel or using oh-so-trendy terms like blogosphere, feel free to send a cyber-slap my way. My one nod to blogging culture is to choose a faux-edgy blog name which is a play on my initials and the popular engineer's retort RTFM (Read The Fine Manual) from back in the days when users had manuals printed on dead trees. Not a very original idea, of course. That URL on Blogger was taken ages ago and hosts, like so many others, a place holder blog.