Widgets 1.0 (v2)

Today the W3C published the Second Public Working Draft of the Widgets 1.0 Specification. It’s been nearly a year since we published the first public working draft (11 Nov, 2006) and much has changed and been added to the spec (…and it still has a long long way to go yet before it will be finished!). The most notable addition to this version of the spec are in the attempt to standardize a subset of the Zip specification and support for digital signatures using XML Digital Signatures. Unfortunately, a lot of exciting things that are under discussion by those participating in the standardization effort have not made it into this latest draft. For example, we are still trying to work out a nice model for automatic updates, but we should have something drafted up fairly soon.

The main problem I’ve been working on over the last two months is trying to specify a subset of Zip that should be used by widgets. My goal has been to define a subset that is interoperable across all platforms and devices in such a way that it also ensures longevity. As you might imagine, this has proven to be quite a challenge…

The issues with Zip

The Zip file format is what is commonly referred to as a de facto standard: it is not formally specified by any standards body, but of it is so widely implemented that it is interoperable across OSs and devices. This seems great on the surface, but when you try to standardize it, it becomes quite a nightmare. The main issues are these:

  • There are competing Zip specifications and there are many versions of each of the Zip specifications.
  • Different version of the Zip specification are implemented across different platforms and OSs.
  • There are many features in Zip that are desirable (eg. UTF-8 support), but are not widely implemented.
  • Zip is not an “open standard”, it is the property of PKWARE.
  • Zip is periodically updated and PKWARE does not provide any links to previous versions of their specs.

Competing Zip specifications

There are essentially two Zip Specifications that applications make use of: the “official” PKWARE Zip Application Notes and the “unofficial”Info-Zip Application Notes (mostly on Unix). The unofficial notes basically take whatever PKWARE has officially published, and gets modified, or otherwise clarified, by the guys at Info-Zip. In this sense, much of what one finds in the Info-Zip specs is identical to the PKWARE Zip spec. But, because PKWARE actually maintains the official spec, the PKWARE spec is always more up-to-data than what Info-Zip has on its website (for instance, the latests version of Info-zip covers version 6.2.0 of the official Zip spec (26 April 2004); the latest version of Zip is version 6.3.2 which came out in September 2007!, so InfoZip is three years behind PKWARE!).

Problem: Info-zip contains details that pertain to how info-zip works and may not be compatible/interoperable with the PKZip Spec. For example, Info-zip contains details about how to handle Unix permissions, while PKWARE’s Zip spec does not. This might not make the file formats incompatible, but it does make them physically different. You can try this out yourself: zip up a file using Info-Zip’s zip implementation and then zip up the same file using Windows’ Compressed Folders. The results will be different, but you should still be able to decompress the Info-Zip file using Windows’ native Zip implementation.

Different version of the Zip specification are implemented across different platforms, OSs, Specs

Another significant issue form a standardization perspective is that packaging formats are making use of either some Info-Zip spec or some PWARE spec. Significant examples include:

Java/JAR (including WAR and EAR) :
Info-ZIP Application Note 19970311
Open Document Format (ODF):
Info-ZIP Application Note 19970311
Open Office XML – Open Packaging Convention (OOXML-OPC):
PKWARE Zip Application Note (version 6.2.1), but with a bunch of clarifications.
OEBPS Container Format 1.0:
PKWARE Zip Application Note (no explicit version, but at least version 2.0 needed to extract and version 4.5 needed to extract Zip64).

I still have little idea as to what version of the Zip specification is actually implemented on each OS, let alone on mobile devices (information that seems to be quite difficult to come by!). As a result, and after some discussion with Jon Ferraiolo of IBM, I decided to base the Widget Spec on the OEBPS-OCF’s conformance requirements for Zip packages. I was tempted to make the widgets specification conform to the OOXML-OPC spec (put away your tomatoes!) because, in my opinion, the container aspects and conformance requirements are well specified (even if the rest of OOXML is “evil”).

Desirable features in Zip (6.3.2)

There are a number of really cool features in Zip that would make specifying a container format for widgets much better. They include:

  • Strong Encryption (using x.509 digital certificates): basically solves the digital signature problem, I think.
  • UTF-8 support: solves a significant part of the internationalization problem.
  • Zip64: future proofing.

To require widget engines to actually support these features puts a fair bit of strain on makers of widget engines. At this point, we have required that implementers support UTF-8 and Zip64.

Zip is not an open standard

The fact that Zip is proprietary might be something that comes back to bite us on the ass. I’m no lawyer, but there of patents/IPR issues surrounding Zip. I’m also not sure about how PKWARE will feel about WAF specifying a subset of their specification. I’ve emailed PKWARE and informed them of what we are doing and requested that they review the spec. They have responded and said that they will look into it.

Where to from here…

Looking forward, I’d really like to get all the physical and logical packaging stuff done. That includes:

  • Anything Zip related
  • The inter-package addressing model
  • How to handle decompression
  • How to name files in ASCII and UT-8

I’d also really like to nail down the auto-updates model and make sure that the manifest language we are specifying is covers all the common use cases. The security model is the elephant in the room 🙂 No one wants to touch it at this point; but we know its a massive issue. Another massive issue is the APIs… but that’s not something I want to get into now. A big issue for me is internationalization. I’ve been blocked a number of times when I’ve proposed doing internationalization using folders… every widget engine except Opera does it, so I think we should do it too.

W3C stops standardization of the declarative format for application and user interfaces (about time!)

Yay! the W3C has canned the work on the Declarative Format for Applications and User Interfaces (DFAUI), putting an end to something that had no way of ever finishing. Of course, you probably have never heard of the DFAUI because the WAF WG never published any documents about it. The idea was to standardized an XML language similar to XAML or Openlaszlo…. but instead, what the WAF-WG got was an input from Nexaweb called XAL. Anyway, the people that were supposed to be editing the document never got very far, and as far as I am concerned, the work they produced was of fairly low quality (that’s not to say my work doesn’t suck!).

These are my random thought on how I think the DFAUI should have been standardized…and why it failed….

Continue reading W3C stops standardization of the declarative format for application and user interfaces (about time!)

July-August wrap-up…finally confirmed

Wow, I can’t believe it’s September and how slack I have been about keeping this blog up-to-date…

July was mainly taken up by my PhD confirmation and August with the widgets spec, visiting Oslo for a WAF face-2-face, and a brief visit to the W3C office in Canberra to attend a standards symposium.

Confirmation dramas

My PhD confirmation went well enough… about 10 people were nice enough to come along. I basically presented the issues around widgets and then argued about why it was valid to do my research their standardization. I got a bit of grief from the academic panel about my resistance to include the academic community in the research. I basically tried to argue that academics are out of touch with what is happening in this area and there would be little point in me including them (as I also have very little idea about what is going on in this area!;)). Still, they insisted that I produce at least three academic papers out of my research and make that my PhD… I agreed.

I will now produce three papers for my PhD (I’m doing my PhD by publication, which means I just publish papers instead of writing PhD thesis that no one will read):

  1. The first will be about the issues around widget engines and the problems that could be addressed by standardization.
  2. The second will be on the design decisions that have gone into the Widgets 1.0 Spec. This paper would cover, for instance, the reasons we are using XML for the manifest format instead of JSON… or why we may or may not have a namespace for the widget manifest format.
  3. Third, would discuss the Widgets 1.0 Test-suite… particularly any interesting design decisions. There is not much literature out there on web-based test suites for specs. Well, not much that I could find as part of my research.

Anyway, I was asked to resubmit my confirmation document and list the above three publications. To be a smart-ass, I submitted the document with the following footnote (mostly to test if they would read any of my changes):

It is my position that producing peer-reviewed publications for journals and conferences is a waste of time (according to an article published CiteSeer in Nature (Lawrence, 2001), the average citation rate of a journal paper is 2.74, and articles freely available online are more highly cited. For greater impact and faster scientific progress, authors and publishers should aim to make research easy to access. Something that is not possible when papers are published for profit on the medium of paper. Also QUT’s Creative Industries faculty finds it difficult to deal with the concept of harnessing collective intelligence (O’Reilly, 2005) as it undermines what academics do as they can no longer claim authorship over works/publications. This is both unfortunate, sad, and a very archaic way of thinking. It is obvious than harnessing the collective intelligence, as it done in open standardization efforts, will more often yield higher quality of output than those produced by pretending that everything is written in isolation.

That footnote did not go down too well, so I was asked to remove it (hehe, yes!! CENSORSHIP AT QUT, IT’S ALL TRUE!!!!)… anyway, I removed it resubmitted a few days ago, so I am now finally be confirmed. Glad that part of my life is over and done with.

WAF F2F in Oslo

Morning in Oslo...

The Web Applications Format’s (WAF) face-2-face meeting in Oslo went very well. It was held at Opera Software‘s HQ in Oslo. We most talked about Widgets and decided to finally dump the Declarative Format For Applications and User Interfaces (DFAUI). (I’ll talk more about the widgets stuff we discussed in a bit). The implication of dumping the DFAUI was that our working group charter is monumentally out of date! In fact, the current WAF charter does not even mention widgets. The WAF working group will have its charter reviewed in November (as the group also expires in November, 2007). I expect the group to be rechartered for another 2 years.

The discussions around widgets focused mostly on the editor’s drafts of the widget requirements and the widget spec. As I’ve written about previously, I’ve done a lot of work recently on the requirements. The requirements document is starting to become quite stable. However, the widget spec is still in a bit of a mess. I mostly blame myself for that so I’ve started working on restructuring it. The reasons I think the widget spec is a bit of a mess is because:

  • I still don’t have a complete overview of the overall problem space (particularly APIs).
  • I really want to make sure that a proposed solution is as compatible as possible with existing implementations, and also follows W3C principles. This means really understanding how the current market-leading widget engines actually work. This involves a lot of research, development, and testing on each platform, so it is quite time consuming.

Regardless of my document structuring inabilities, I think that the meeting was quite successful in at least nailing down what we need to make a priority in the short term. Namely:

  1. Manifest (language and processing)
  2. Packaging (bits of Zip that we want to use)
  3. Auto updates (a HTTP and XML based model)

I recently made a post to the WAF public list about automatic update based on the discussion the working group had at the face to face meeting. In the coming days (weeks?), I’ll write a blog entry on current approaches to automated updates. I’ll mostly just focus on FireFox, Yahoo! Widgets, and compare that to a pure HTTP solution proposed by Mark Nottingham.

Standards Symposium, Canberra

The W3C Australia office and NICTA jointly organized small standards symposium, which was held at the W3C office (CSIRO) Canberra on the 26th of August. Overall the event was very interesting covering a wide range of topics relating to standardization of various technologies (mostly built on XML).

One interesting talk was given by Anne Cregan about some work she is involved with regarding the Semantic Web. As someone who was once into the rhetoric of the semantic web, I was quite fascinated to hear that there is a working group trying to create what Anne described as an English-like expression of OWL. I then proceeded to ask Anne why there was no collaboration between the HTML WG and the Semantic Web working groups, given that, the semantic web is supposed to interact with the Web (AKA, HTML+tag-soup)? She mentioned GRRDL, which made me giggle as I can’t think of a more useless technology.

My conclusion from Anne’s talk is that the Semantic Web and HTML groups are still working completely separately of each other, which is probably why the Semantic Web movement got so badly pwned by the Microformats and Tagging communities. At least it’s good that they have started dumping all the XML syntax in favor of more human readable alternatives. If the alternatives prove to be usable, then there may be hope yet to integrate them on the Web… particularly if any new semantic web language does not rely on XML… maybe someone will come up with a nice JSON alternative 😉