The Catch-22 of Open Format Adoption, Part 3: Office Documents

Having looked at music formats and instant messaging protocols, this final installment of a short series on open formats covers what may be the most ubiquitous of digital file formats: office documents. Spreadsheets, presentations, desktop databases, and the common text document hold most of the business information of our age.

In North America, at least, most of this information lives inside a set of patent-protected, binary (which makes them difficult to reverse-engineer), and undocumented file formats. The Microsoft Office formats, the most well known of which is the Microsoft Word format, are used to store millions (billions?) of documents, from personal journals to government legislation.

For those creating these documents, the problem is inherently disguised. If you create a Microsoft Word document, then you must have access to Microsoft Office and can therefore open, read, and modify the document. The problem arises when you don’t have access to a copy of Microsoft Office. This may be due to financial limitations, or it may be because you are running on a platform that is not supported by Microsoft. No matter how much money you have, you can’t buy a copy of Microsoft Office for Linux.

The frustration of receiving Microsoft Word documents as email attachments led Richard Stallman, founder of the Free Software Foundation, to write a brief manifesto covering the perils of this proprietary format.

The essential problem with a proprietary document format like Microsoft Word is that a private corporation owns the ability to access the works you have created. While it’s not likely that Microsoft is going to deny you access to your Microsoft Word-formatted love letters and chili recipes tomorrow, they do theoretically hold that right.

Confusion and Optimism

The closed binary format in Microsoft Office has been enormously broad in its reach. However, the life of this format is limited. Microsoft recently announced plans to move to a documented format that could potentially be accessed through non-Microsoft means.

The meaning of this announcement has yet to be truly understood. Some see this as the end of the proprietary Microsoft format and a great victory for freedom and openness, as millions of documents will be created in an openly documented format. Others are more cynical, citing licensing issues that will limit what people can do with the formats.

It seems clear, though, that while the legal issues around the new Microsoft formats remain disputed, their technical architecture (basically XML in Zip files) will be much more easily accessible regardless of whether access is endorsed by Microsoft or not.

I don’t clearly understand the issues around this yet myself. The Microsoft community/weblog site, Channel 9, posted a video interview about the new Office formats with Jean Paoli. Watching this video shows the Microsoft engineer’s obvious enthusiasm for openness. However, the video ironically requires proprietary Windows Media technologies for playback.

Alternatives and Workarounds

As Ogg Vorbis is to MP3, and as Jabber is to MSN/ICQ/AIM, so OpenDocument is to Microsoft Office formats. OpenDocument is a new set of standard office file formats for text documents, spreadsheets, presentations, and charts. This open and standard format is the default format in the forthcoming OpenOffice.org 2.0 office suite, but could theoretically be implemented by other applications as well.

Saving your documents in the OpenDocument format means that no one owns the ability to access your works. While the specifications aren’t perfect (I was dismayed to here complaints about the spreadsheet component), it remains a critical standard.

As is the case with instant messaging protocols, the move from proprietary to open office file formats can be eased with the help of transitional software. The OpenOffie.org suite (both the 1.x and upcoming 2.0 versions) can open, edit, and save the main Microsoft Office formats quite well. Using OpenOffice.org, I can easily open any Microsoft Word attachments I might get in my email.

For those that are still stuck with Microsoft Office as an overall platform in their organization, but are looking to move away from Microsoft Windows, there are more promising options. The Wine project is a compatibility layer for running Windows applications on Linux. Especially when packaged in the Codeweavers CrossOver Office product, it is surprisingly easy to actually run Microsoft Office on Linux. This is obviously only a transitional aid, and not a long term solution, but it is helpful.

Conclusion: Freedom Should Be On By Default

The core idea behind this series on open formats and protocols is that you should not be limited in access to what you have created yourself, regardless of the tools you used to create. No one would buy a pen that produced writing that could only be read through special glasses sold by the same company. Even more so, no one would allow their governments to publish documents created by this crippled pen.

Being locked out of content that should be free or that you have legitimately purchased is bad enough. I have to use illegal software to watch DVDs (that I have bought and paid for) on my laptop. However, it is even worse when you are locked out of content that you have created yourself.

If your mom buys a computer, writes you a letter, and emails it to you in the Microsoft Word format, you have to pay Microsoft to read the letter. Of course, your mom doesn’t have to use Microsoft Office, but if it is the default word processor on her new computer, she may not realize the issue.

If you have your wedding video-recorded and it is given to you by the production company in DVD format, you can’t make copies for your family or as a backup. Again, the production company doesn’t have to use the proprietary DVD format, but it is the only one that will play in everyone’s home DVD player. [UPDATE: Several people have corretly pointed out that the proprietary DVD encryption (CSS) is optional and need not be used on personal DVDs – good point.]

For these reasons, it is not good enough that freedom be available as an option. Freedom must be on by default.

The Catch-22 of Open Formats mini-series
  1. The Catch-22 of Open Format Adoption, Part 1: Music
  2. The Catch-22 of Open Format Adoption, Part 2: Instant Messaging
  3. The Catch-22 of Open Format Adoption, Part 3: Office Documents (you are here)
 

Free and Open Source Software at silverorange

I’ve written a short article about how we use free and open source software at silverorange, the web development firm where I work.

Over the past few years at silverorange we have become more interested in and more aware of free and open source software. The interest started on the server side, where the open-source revolution was well underway with the growth of the Apache/Linux web server and then of PHP and other web scripting languages. Since then, however, our interest and experience in open source software has spilled over on to our desktops and laptops.

Since open source software doesn’t have a PR firm to tell the world that a small business like ours can run linux on a majority of desktop machines, I thought we could share a bit of information about the open source software projects we use and follow.

Read the full article over, Free and Open Source Software at silverorange, at our silverorange labs weblog

 

The Catch-22 of Open Format Adoption, Part 2: Instant Messaging

I wrote last week about the catch-22 of open audio formats. Online music isn’t the only domain in which open formats are emerging, nor is it the most significant. The world of instant messaging (IM) is another case where open protocols have emerged to compete with their proprietary predecessors.

Chances are if you’re online to read this article, you use one of the major commercial IM services. MSN, AIM (which now includes iChat users and ICQ), and Yahoo’s IM services are all enormously popular.

In this case, Jabber is the free/open-source alternative. While the end-user experience is basically the same, Jabber has a different significantly architecture from the dominant IM.

I’m not an expert in this area, so I’ll keep my description as basic as possible. The Jabber protocol is similar to the email infrastructure, where anyone can setup a server that clients connect to. The server then relays messages to other clients on that same server, or to other servers to reach other clients. Like with email, then, you can setup your own server – though this isn’t practical for most people. The more likely scenario involves companies or internet services providers (ISPs) setting up Jabber servers much like they do with email. This differs from MSN, Yahoo, and AIM in that each of these services have their own central server systems that are controlled by the company that owns each network.

As with music file formats, most people don’t care about instant messaging protocols. They just want to chat with their friends and co-workers. You have to use the same protocol/service as your friends, or you’ve got no-one to talk to.

There are some key software applications that can help bridge the gap between proprietary protocols and the open Jabber protocol. Several instant messaging client applications, including Gaim, Trillian, and Adium, allow you to connect to all of the major IM networks. You can have contacts from MSN, AIM, and others, all on the same contact list as your Jabber contacts.

These multiple-protocol clients help ease the transition to open protocols. If I were to switch entirely to Jabber today, I would no longer be able to talk to many of my friends. However, using Gaim, I can use Jabber whenever possible, but still maintain contact with those of my friends still using proprietary protocols.

Of the 20 to 30 contacts I have in my instant messaging client, a little more than half of those are using Jabber. The remain contacts are either AIM, ICQ, or IRC (I’ve managed to drop any MSN contacts).

This is likely a higher ratio of Jabber-to-proprietary contacts than many. This is because at the small business where I work we have our own Jabber sever setup that allows us to have secure (and free) instant messaging (both for one-on-one chat and for group chats). The open-source nature of Jabber allows our company to easily control and manage our own instant messaging server. I would encourage other businesses to do the same. It has been a great tool for us.

The Catch-22 of Open Formats mini-series
  1. The Catch-22 of Open Format Adoption, Part 1: Music
  2. The Catch-22 of Open Format Adoption, Part 2: Instant Messaging (you are here)
  3. Part 3: Coming soon
 

The Catch-22 of Open Format Adoption, Part 1: Music

We’re all familiar with the MP3 file format. As far as most people are concerned, the format implies free music. The software required to play MP3 files is usually free as well. That said, neither of these things necessarily follow from use of the MP3 format.

What most people don’t realize is that the MP3 format itself is not free. If you want to create a device or a piece of software that plays back or creates MP3 files, you have to pay Thomson Consumer Electronics for a license to do so.

Companies that sell products that support MP3 are paying Thomson for each sale. That means Microsoft Windows, Apple’s Mac OS X and iPod. This even applies to software that is free for the end-user to download, such as Apple’s iTunes or Winamp. That’s right – Winamp had to pay for a license for every copy of the player that all of us got to download for free.

What the heck is Ogg Vorbis?

There is an alternative format that is a technical match to MP3 that is not encumbered with patent or licensing issues. The Ogg Vorbis format is as good or better than MP3 and is completely free (both in terms of price and licensing).

Why, then, hasn’t Ogg Vorbis taken off? I see a few reasons for this. The first is that the term “MP3” became a brand name associated with free music. Companies involved in music-related products and services wanted to be able to say “MP3”.

Apple had the opportunity to make the move when they introduced iTunes and the iPod. Both are compatible with MP3, but the default format is AAC, something Apple presumably used for the ability to control playback (“digital rights management”).

So What?

The most significant reason that Ogg Vorbis didn’t overtake MP3 is that MP3 did what people wanted. End users weren’t paying the license fee. Winamp was free, iTunes was free. Why change?

We see the real problem with a non-free file format when free/open-source software starts to become more prevalent on the rest of the desktop. Completely free/open-source desktop Linux distributions cannot include support for MP3 playback, because they would have to track (and pay for) each download. As a result, Fedora, Ubuntu, Debian, and other popular Linux distributions can’t play a simple MP3 file out of the box. This is because the file is simple, but the lisencing is not.

Why Not Just Use Ogg Vorbis, Then?

What then, is a music publisher who cares about free and open software to do? I produce an amateur radio show made available for download in MP3 format. I would like to publish in Ogg Vorbis format, but a lot of my potential listeners would have to jump through hoops to be able to play the show.

Consider John Q. Listener. He’d like to listen to my radio show on his new iBook. However, iTunes doesn’t support Ogg Vorbis playback by default. A plug-in is available, but is he really going to go install it just to listen to my dorky little show? Even if John does find a way to play the Ogg Vorbis files on his laptop, he won’t be able to play them on his iPod. The same goes for overwhelming majority of computer users who are running Windows.

I want to support the open file format, but I also want people to listen to my show. I want people walking around with my favourite songs on their iPod. What am I to do? I have three options:

  1. Publish only in MP3 – This works for everyone but a small number of Linux users (most of whom know how to get MP3 playback for their Linux computers anyhow). Free file format be damned, John Q. Listener is walking around with my show on his iPod.
  2. Publish only in Ogg Vorbis – People learn about the format. Some may even find the software required to play it on the Mac or Windows PC. However, many (most?) people won’t bother listening since they don’t have support for the file format already installed. No one with an iPod can listen to it (unless they convert it to MP3, and it’s not that good of a show to be worth that kind of time and effort). The world is a better place, but I’ll never get famous this way…
  3. Publish two versions, one in MP3 and one in Ogg Vorbis – many online media outlets go this route with streaming formats (publishing simultaneously in Real, Windows Media, Quicktime formats) to ensure the widest possible reach. However, this adds a new layer of complexity for my John Q. Listener. Now, instead of just downloading and listening, he has to choose between two formats. He doesn’t care, he just wants to rawk. Also, much of the benefit of publishing in Ogg Vorbis is now lost, since only those who are already familiar with it and using it will bother choosing it as a format.

I went with option #1. I’m still not entirely comfortable with this, but I want as many people as possible to hear by show. Obviously, I’m not writing here with a clear recommendation. Rather, I hope to highlight the benefits and issues surrounding the move to free/open formats.

I’ll be writing more about the catch-22 of open formats, and with a bit more optimism, in the coming days.

The Catch-22 of Open Formats mini-series
  1. The Catch-22 of Open Format Adoption, Part 1: Music (you are here)
  2. The Catch-22 of Open Format Adoption, Part 2: Instant Messaging
  3. Part 3: Coming soon
 

Video from GUADEC, the GNOME User and Developer European Conference

GUADEC logo

The GNOME User and Developer European Conference (GUADEC) is underway this weekend in Stuttgart, Germany. For those, like myself, who are unable to attend, the fine open-source media company, Fluendo is providing live streams and archived videos of the talks.

The talks from GUADEC day one are now available for download. The videos are encoded in the free/open-source Ogg Theora format. On Linux, anyone using a Gstreamer player, Helix, or VLC can play the videos. Windows users can install the plugin for Windows Media Player, and Mac users can use VLC

Thanks to Fluendo and the conference organizers for providing the videos.

Update: The talk by Ralph Giles of Xiph.org (the organization behind the Ogg Vorbis audio codec and the Theora video codec mentions that Yahoo is shipping the DirectShow (Windows Media) plugins for these open-source formats with their Yahoo Music service download. In simpler terms, this means that people who have download the Yahoo Music player can automagically play Ogg Vorbis and Theora files in Windows Media Player. Nice.

 

Roadster: Mapping on Linux

I spent last week in New England visiting some clients, friends, and geeks. The geeks part was at the Linux World Expo and Fedora Users & Developers Conference. I’ve posted some photos from the trip.

The Linux World Expo was comprised mostly of creepy salesmen at booths with microphones giving out penguin toys made by children in Malaysia (I presume). It was definitely aimed at corporate middle-management types. There was much talk of “deploying comprehensive integrated solutions on a L.A.M.P. stack. All about the stacks. It was worth visiting for a few hours if only to witness the sheer spectacle of it all.

I then visited the Fedora Users & Developers Conference (cleverly named FUDcon) at Photonics Center at Boston University. There were some interesting talks and some more interesting people. The contrast from FUDcon to the Linux World Expo was quite striking (salesmen talking loudly into cellphones vs. geeks, doing stuff). Video of the FUDcon talks will be posted on the FUDcon website soon.

Screenshot of Roadster
Screenshot of Roadster mapping Boston. See the full-size version.

The trip was also an opportunity to meet up with Ian McIntosh, who I had met at the fall Boston Gnome Summit. He is working on a project that I am very excited about. Roadster is a street mapping application for Gnome/Linux much like Microsoft’s Streets & Trips.

Ian is just getting going on the project, but it is already quite promising and has some key potential contributors interested. Carl Worth, maintainer of Cairo, the graphics package that Roadster uses to render map, was quite enthusiastic about the project and eager to help improve the rendering speeds (it’s pretty slow right now). Eric Raymond, who maintains the GPSD package for GPS tracking devices on Linux was also interested. My own friend and co-worker, Nathan Fredrickson is also helping out.

It was fascinating to see a young project spark enthusiasm and participation from people like that. Several people told me that Microsoft Streets & Trips was the only reason they still keep a Windows partition on their laptops and would love and alternative.

Roadster is still a young project, but there is a Roadster website, a development wiki, and a roadster-devel mailing list if you are interested in helping out.

 

Consistency in Save Confirmation Dialogs in Gnome

For slice of readership to whom this type of thing is relevant, I have compiled an overview of of the inconsistencies in file-save confirmation dialogs in Gnome. There is also a page with screenshots of the dialogs in question. I’ve posted the overview to the Gnome desktop development mailing list for discussion.

If you don’t know or care about (or any combination therein) what I’m talking about, I thank you for your patience.

 

Gnome Outliner v0.1 Released

Gnome Outliner v0.1

The Gnome Outliner project was dreamed up and got started here on this weblog last May. Gnome Outliner is a simple application to write and edit outlines for the Gnome desktop environment for Linux.

There was some nice buzz when the project got started – lots of people were proposing ideas and started writing code in several different languages. After the buzz died down, things slowed on the project for a few months. In the past week, though, we’ve gotten a series of patches and things are starting to get rolling again.

Following the great open-source motto to “release early and release often”, we’ve done our first release: Gnome Outliner version 0.1. It’s not quite ready for every day use, but there is a good base there and I’m anticipating some nice progress in the coming months. Thanks to all who have contributed so far.

 

How I Became a Free Software Zealot

It started innocently. I started using a free/open-source web browser, then called Phoenix (now Firefox). It seemed to me to be faster, easier, and generally better than Internet Explorer on my Windows XP powered laptop. Soon after, having been unhappy with the performance of Microsoft Outlook with my large volume of archived mail and unwilling to pay for or pirate the promising new (2003) version, I switched from Microsoft’s Outlook to the Mozilla Thunderbird mail application.

Soon after, I made the switch from Trillian to the free/open-source instant messaging client, Gaim. It then dawned on me that the three applications I use most, my web browser, email client, and instant messaging client, were all free/open-source software. Not only were these applications free software, they are also available across multiple platforms (Windows, Linux, and some for Mac OS X).

Armed with the realization that I was close to platform independence, I intentionally sought to weed out the few remaining Windows-only applications in my arsenal. I made the switch from Microsoft Office to OpenOffice.org. This left some games, and Photoshop/Illustrator as the last key proprietary/closed-souce applications in my regular use.

Driven primarily by curiosity and technical/professional interest (particularly in alternative user-interface design), I decided to exercise my newfound platform/operating-system independence and switch my primary laptop to Linux. This was over a year ago and I’ve been relatively happy, given some challenges, since.

All along, these decisions and similar decisions at the company-level where I work were pragmatic. Open-source software is good, but it’s really about using the best tool for the job, be it free/open or proprietary. While the free/open systems were generally beating out the proprietary systems, the decisions were still driven primarily by a (relatively) qualitative comparison of features/quality, rather than any philosophical best about licensing.

I had always been turned off by Microsoft-hating Linux-zealots that play into my stereotype like a bad gay sitcom character. Microsoft has many problems, but they also develop some remarkably good software (I’m a fan of the Office suite and love Microsoft Streets and Trips). What bothered me more than the last-computer-conference-I-was-at-Tshirt attire was the fundamental belief that free/open-source software was better than proprietary alternatives.

However, having lived in a mostly free/open-source software environment for over a year now, I am starting to drink the kool-aid.

I had been running Mac OS X on a old iMac for browser testing purposes and following the development of OS X for a while too. It dawned on me that one of the primary reasons keeping from using OS X as my primary operating system (in addition to not wanting to replace my working hardware), was due the licensing, control, and ownership of the software and platform.

Mac OS X, since it is built largely upon open-source components and protocols, carries with it many of the pragmatic benefits of open-source software. However, since a significant amount of the Mac OS X system is not free/open-source, you do not have all of the freedom you would have on a completely free/open-source system. You are, in the long term, still at the whim of a private corporation (Apple Computer).

To many people, quite understandably, this will not matter – especially since Apple is producing some great software lately. That’s fine with me. I’m not at a point where where I want to force everyone to use free/open-source software. The idea of forcing something in the name of “freedom” just doesn’t sit well with me.

That said, I am starting to think that governments and other public institutions might have some kind of moral and ethical (though not, at this point, legal) obligation to share software it develops or has developed on its behalf.

While I’m not about to start burning the Windows XP license that came with my laptop in the street, I’m starting to take into account licensing and freedom to use and control software in the decisions and recommendations I make for myself and those around me.

 

Cleaning Your Bedroom and Improving the World

When I was growing up, my bedroom was always a mess. Every week or two, my parents would get me to clean my room. I wasn’t interested in cleaning my room, but I had to get it to at least pass a quick visual inspection my mom or dad.

To make sure I passed the informal inspection with minimal amount of work necessary, I would stand at the door of my room, where my parents would stand, and look in. I would scan the room and make note of the first thing I noticed that was out of place – pants on the floor, the unmade bed, or whatever was the most visually obviously out of place. Once this was taken care of, I would go back to the door, have another look, and pick the next thing I noticed. I would repeat this simple process until things were looking good enough.

This process has stayed with me ever since and often proves to be useful way to decide what to work on first. Over the past year, I’ve tried to apply this bedroom cleaning prioritization technique to help improve, in a small way, the open source desktop computing experience.

Last fall, I looked at my desktop computing environment and took note of the first thing that didn’t feel right. At the time, it was the ugly old Firefox icon (then called Phoenix). That helped, in small part, get the process started that culminated in the redesign of the visual redesign of all things Firefox and Thunderbird.

More recently, now having a beautiful web browser, I returned to my bedroom cleaning technique and took another look at the desktop and see what bothered me next. This time, it wasn’t visual. Rather, it was the sounds used in the Gaim instant messager application that were the most prominent rough edge.

I set out a few months ago to improve these sounds. Like with the Firefox visual work, I didn’t have all of the skill needed to do the work myself, so I looked to others for help. This time it was Brad Turcotte, a musician (aka Brad Sucks), that came to my aid. He and I bounced sounds that he created back and forth for a while until we had something that sounded right.

These new sounds have now been accepted by the Gaim developers and will be included in a future release.

Now, I’ll have to head back to my bedroom door and take a look around to see what rough edge I notice next.