Andrew Savikas over at the TOC blog has been lamenting the recent AAP letter to the IDPF, and rightly so. Publishers need to stop thinking of digital books as low-fidelity copies of print originals, and they need to stop thinking about reading as an act of solitude. Reading, whether done in solitude or with a group, always ultimately connects us to the larger world, and the ways we make that connection are changing quickly.
Savikas also points out the lack, in the ePub specifications, of frameworks for social connection, and I think this hits on a bigger issue, which is that the ePub specifications are completely lacking as a Web-based framework. In other words, they seem to have been developed, quite ironically since they are based on Web technologies, without a sense that they might one day prove very useful in the context of the Web itself. ePub, for all its merits, is not browser-friendly, search-engine friendly, or at all ready for the next wave of Web-based social utilities such as annotation, presence, instant communication and more granular linking. Why is this?
First, let me review what an ePub is. It’s essentially a directory of images and/or XHTML documents, just like a web site. The directory is called the OPS, or Open Publication Structure. Minimally, it must contain one file with an extension of ‘.opf’–referred to as the OPF file. This OPS directory is also compressed into an archive format (this is like what WinZip or Stuffit do to your folders). Thus it becomes a single file referred to in the ePub spec as the “container.” It’s what we refer to when we say “EPUB file” (the official style is all-cap).
Now that the acronyms are flying, I want to address the two big reasons that ePub is Web-challenged right now:
- The recommended container format, OCF
- The proposed way of extending the core structure, OPS “XML islands”
I’ll address these one at a time, and try to explain what I mean without getting too technical.
ePub needs a non-opaque container
On the first point, the Open Container Format that defines an “EPUB file” is a format completely incompatible with browsers, opaque to search engines, and unreasonably difficult to link into from other documents. The good news is that in the specs, it’s only a “recommended” method of containing an Open Publication Structure. The bad news is that the tools and apps coming out are just assuming that Open Publication Structures must be contained in the Open Container Format. For an example, a valid Open Publication Structure (which remember is just a directory or a folder on your desktop), can’t be opened in Digital Editions, nor can it be validated with the ePub validation tool called epubcheck. Digital Editions and epubcheck only accept zip files.
The directory that holds the Open Publication Structure is compliant with IDPF specs, it’s just not in the expected container format. It is itself a container, though, and you could drop that container onto your web browser and easily browse all of its content. In its zipped form, you couldn’t do that, but you could read it Digital Editions. So what’s wrong with the picture?
What’s wrong is OCF, the Open Container Format. I don’t think ePub books need a container format at all, they just need a container, and the perfect container is a plain old fashioned directory. An ePub directory would be named with the ‘.epub’ extension, would contain a valid Open Publication Structure, and would appear to users as a single file. The questions are: 1.) How do you transfer these structures, and 2.) how do you protect them from accidental or intentional tampering by end users?
OCF was supposed to answer these questions, but in doing so, it cut the Web out of the picture. And the main alternatives to OCF are so-called “web archive” formats, none of which work across all of the major browsers. So the consensus is starting to seem like: “we’re stuck with OCF.”
We’re not stuck with OCF
The No Container Format approach has many reference implementations. One is the concept of Bundles. I call it a concept, because the implementation of a Bundle is really just an uncompressed directory of files, often a regular directory just like any other directory, but with an extension and structure which positively identify it as a bundle. Browsers can access it, systems can index it easily and links can point into the XML structures it contains, but it appears to users as a single “package.”
The implementations are often vendor-specific, because they’re tied to operating systems. Think of Digital Editions, and how when you install it your OS suddenly puts an ePub icon on files with the ‘.epub’ extension. The methods for associating icons with file extensions are often the same methods used to identify bundles.
As far as transferring bundle containers, the process is often completely transparent to the user. Email a bundle to a publisher and it will be zipped in the transfer without either party knowing it.
A bundle container would not cut out any current developers or manufacturers, and would add the benefit of linking ePub, literally and figuratively, to the Web. It’s worth consideration, and if it’s already been considered and dismissed, at least renewed discussion
ePub needs extension mechanisms more than extensions
I don’t think the second issue can be solved without first tackling the container problem, but assuming the IDPF does propose linkable, searchable, browseable containers, extensibility needs to be tackled next. Extensibility is simply a techy term for the potential of a format to be extended. XHTML’s extensibility, for example, is represented by the ability for anyone to create and use new tags.
The other two specs of the ePub triad, OPF and OPS, are based on XHTML, but they’re not designed to leverage its extensibility. The specs don’t recommend ways to link, annotate or interleave with Web-based social utilities, but more importantly they don’t recommend how we might specify those ways either. This is a much more pressing requirement. The closest the specs get is the concept of “XML islands” which are “non-preferred vocabularies” of tags in EPUB content documents. This is a step in the right direction. It does allow other kinds of XML to be used in ePub books, as long as there’s something basic to fall back on. However, it’s also an open invitation for vendor-specific approaches, and that’s not good at all. Good frameworks are known for their freedom within limitations, not for declaring open season on corporate invention.
A model for extensibility
A great example for the IDPF to follow is the XMPP standards organization, formerly the Jabber Foundation. They have developed an XML-based “document stream” specification and an extension mechanism to go with it (XMPP and XEP). Vendors develop extensions alongside the standards body, with the goal of finalizing them as XMPP Extension Protocols (XEPs). Extended functionality can be implemented gradually, and in ways that are guaranteed to be compatible with other vendors, but it can also leverage corporate innovation.
As a case study of an extension mechanism for ePub that falls outside the spec, take Adobe’s use of XSL-FO. This is an XML-based templating language geared toward paginated publications. It’s a nice and useful extension to ePub, but it’s Adobe’s, not everyone’s. We need to extrapolate from this, imagining dozens of companies applying their own page layout extensions outside of the specs. Maybe some of those extensions become incorporated into the specs, maybe not. Either way, the purpose of the specs has been defeated, because innovation is driving the spec, rather than the other way around. EBabel has spilled into ePub. This is an outcome the IDPF should strive to avoid.
I hope the community will consider these points seriously. My aim is not to attack ePub but improve it and empower it to evolve.
Moderator: Just a reminder—the ePub logo is unofficial. Meanwhile many thanks to Aaron for his thoughtful critique. Whether you agree with him or not, it’s important that this standard benefit from intensive scrutiny and civil debate; that’s what open standards should be about. For a different perspective, see responses from Hadrien Gardeur and Jon Noring. – D.R.
Thanks, Aaron, for your thoughtful article – very much needed. There’s a number of technical issues that should be discussed, which I won’t discuss in this comment. Rather, I need to clarify my position regarding EPUB and its underlying specs before I continue to provide technical commentary.
Although I appear to many to be the most visible EPUB technologist, evangelist and apologist (David Rothman probably has me beat on the evangelism side, though), let me note that I have also been a strong critic of some of the specific features and framework design of OCF, OPS and OPF.
For example, I fought hard against the “Islands” feature in OPS, not because I don’t believe in content document extensibility including support for alternate vocabularies, but because it was premature to implement since we have a number of issues to weigh before we support new and untested features that will not be that widely used in the short-term.
What are these issues? They are many that Aaron brought up, such as interactivity, referencing and citation, deep linking, etc., along with a few others. Once we more carefully discuss these issues and new requirements, only then can we add new features. Once we add a new feature, we are stuck with it (this has been the problem with the implementation of HTML since the mid-90’s), so it is imperative we add new features carefully, deliberately and thoughtfully, not hurried by any short-term agenda of a particular commercial interest.
I also developed the OpenReader suite of specs, a next-generation digital publication framework, similar in many respects to OPS and OPF. And a few of us have proposed the Generalized Container Format, which resolves several problems with the current OCF. (GCF is ZIP-based, so it still falls under Aaron’s criticisms – note that one driver for the design of GCF is that it could be used to bind a “web site structured publication” similar to what MHTML does for binding together a web site.)
I could go on with other technical initiatives I have been and am now working on that relate in some fashion with EPUB, but I want to make it clear that of the contributors who were heavily involved in the technical development of OEBPS, and then of the recent OCF, OPS and OPF specs, I am proud to say I have been the most fiercely independent, with no motives other than what I believe is best for all the stakeholders which comprise the Digital Publication universe (DPU), and this includes the end-user, the most important stakeholder of them all.
Since mid-2000 with the demise of Yomu due to the dotcom bust, I have not, and currently do not, represent any commercial interest which is building authoring tools or reading systems (hardware and software), or has any other commercial interest in a particular outcome for the design of the EPUB spec. (I work closely with DigitalPulp Publishing, a small e-book publisher, and not-so-small e-book retailer. DPP only wants to see an intelligently-done reflowable and open digital publication spec that benefits all stakeholders in the DPU, and they leave the details to me.)
And I’ve made sure to represent the requirements and needs of the little guys in the technical working groups. I’ve always sided with the accessibility community (e.g., I was the one who forced the issue to require the NCX in EPUB), the librarians, the smaller indie publishers, and the consumer of course, to name a few. I also keep in close touch with some folk in the larger publishing houses as well.
Anyway, just a clarification of how I personally relate to EPUB and the associated specs, before I provide any technical commentary in this and related threads.
I don’t believe that the container format is much of a problem. It’s very easy to read a zip file, any browser could implement this.
I would much rather side with Jon Noring on this issue. We need ways to package more than a single book (or single version of a book) rather than remove the container format.
XML-islands are not necessarily a bad thing, as you’ve pointed, you need to have a proper fallback whenever you’re using an XML-island to make sure that the content is always compatible with a standard reading system (although it may not be optimized for it).
Adobe’s XPGT is a different issue… We really need ePub to handle paginated publications, something that XSL-FO can do. CSS3 is also going in that direction, but using completely different methods. With the current support for CSS2.1 in ePub, the IDPF must be following very closely the drafts for some CSS3 modules. But CSS3 is far from complete and won’t be ready any time soon. IDPF must choose between using XSL-FO now (very powerful, but quite complicated) or wait for CSS3 (not as powerful yet, fairly easy to use). Jon, could you provide us with additional insights on this matter ?
Hadrien, to answer your question (as best I can) regarding page styling support in a future version of OPS/OPF, I currently see three options:
1) Do nothing for the time-being and see how things shake out as EPUB support grows and matures. (And to watch how CSS3 matures.)
2) “Bless” support for the CSS3 properties.
3) “Bless” support for the XSL-FO approach, currently being championed by Adobe.
Now which approach I support, I’m still uncertain (although if we implement anything it should be CSS-based since we already support CSS). Thus, I tend to fall into the #1 camp, followed by #2. (Yes, I am aware that the CSS3 properties of interest are not yet finalized, which is a problem – I’m checking on how quickly these properties will or can be finalized.)
I’d really like to see more than just one implementation of a fairly-complete EPUB reading system (the only current one that meets this level of EPUB support is Adobe’s Digital Editions) so we can get more balanced developer feedback.
I have contacted a couple people I know who are well-connected with the CSS working group and familiar with XSL-FO as well. Hopefully they are currently evaluating Adobe’s proposal.
I would like to hear comments from others after they have studied Adobe’s proposal.
Is EPUB actually compatible with web browsers? If I unzip (say) Frankenstein by Mary Shelley from feedbooks I get three dozen .xml files which might display ok individually, but how do I read the book in a web browser? The information on how to do so is in fb.opf and fb.ncx but neither one is web compatible. It seems we need a epub2html utility that adds a web-compatible index.htm to the directory.
If the directory was web compatible then perhaps the .epub container would be unnecessary. On the other hand, the lack of a container was a serious disadvantage of the original OEB spec. It lead to multiple incompatible OEB-like formats. In the end, FBReader had to declare that an OEB document zipped together and renamed .oebzip would be treated as an OEB ebook. FBReader could read the .opf file directly (so the directory was in principle a sufficient container) but readers wanted a single file per ebook.
Alan, in a recent TeleRead blog article I discussed an EPUBtoWebSite converter. Look for the light gray highlighted section in the article.
It should not be difficult to write such a converter. I’ve scoped out what the converter has to do.
@Alan I’ve got a working prototype right here of a web application that reads an epub file, renders a TOC and displays the content on-screen. Currently it looks at an uncompressed epub directory; I’ll add zip support tomorrow. Since I started and finished it in the same afternoon, I’d say yes, epub is very compatible with web browsers, with a little bit of code to parse all the metadata.
But this application requires that the epub file live on the same web site as the reader. What people seem to want is a cross-browser plug-in (or native browser support) that would render any epub file found anywhere, similar to the way Acrobat is implemented in the browser, but hopefully with less loading time.
I’m not a browser developer so I can’t help with that, but if you want to put pressure where it might be most effective, I’d say get in touch with the Firefox team as a first start, or a group that is interested in developing a mature Firefox extension.
@Liza is right. Unpacking ZIP files will really not be an issue for browsers or for search engines, so I don’t think that criticizing OCF today makes much sense (ZIP “just works”). That said, perhaps there would be a way to define a better container in the future (but I’m not yet convinced of the motivations for that).
I do think that Aaron is right to point out that tools and validators should accept the unzipped container. Also publishers and vendors should consider using an unzipped format for any content that should be web-discoverable.
Just to note that DTBook documents are also supported in EPUB, and I’m currently (re)studying the DTBook DTD to ascertain how web-browser-compatible DTBook is.
From my understanding, it is CSS-compatible (not like TEI which has structures that are not compatible with the HTML/CSS paradigm), and so one should be able to apply a default browser style sheet extension. I expect that very few EPUB’s the average book reader will obtain (in the short-term at least) will include DTBook documents. But in the long-term we may see more DTBook. It is a quite good markup vocabulary (XHTML with extensions) that forces structure on the author, which is Good™
@Liza, most ebook readers work on local files. So that is a good start.
I was thinking more of a stand-alone utility, rather than a web application, but either works as a proof of concept. Note that format shifting is a recognized method of supporting EPUB, so EPUB to EPUB+WebFiles would seem to meet that criteria and once the WebFiles are created they should work remotely.
Yes, it is important to understand that the OEBPS working group (and the PSWG which preceded it) has always considered, for decision-making purposes, that OEBPS/OPS may be used for both:
1) Direct rendering in capable reading systems, and
2) Conversion into other formats. Who does the conversion (including by the end-user) is immaterial.
(The original view of OEBPS being an “exchange format” was essentially publicly stated for “political reasons”, but we in the working group always considered both possibilities above in all decisions we made.)
So obviously an EPUB–>”local web site” converter fits into the general scheme of things which the OEBPS Working Group considers as viable/approved/encouraged/etc.
Let a thousand flowers bloom!
One specific: Google almost surely can handle zipped texts. If Google can do it, probably other services can or will. Thanks. David
@alan – an index.html file in an OPS structure would certainly be good idea. I’m not sure why it was not included in the spec as a fallback for the OPF file, except that EPUB was designed mainly for “e-book” devices.
@keith – I doubt that the folks who work on the major browsers are going to make OCF browsing a huge priority anytime soon. Likewise, until there’s a good reason, search engines are not going to do zip indexing (it took them long enough to index PDFs and Flash files).
Many here seem to have missed the first point, which is that there is room for another type of container more inclusive of browsers as reading agents.
There are good reasons that the W3C never proposed wrapping web sites in zip containers, and they have to do with linking, annotation and searching–all things we expect from digital books. For the same reasons, we don’t develop web sites as collections of PDFs. Search engines DO NOT index the contents of compressed files, and there is no standard scheme for linking into those contents. This has serious implications for any document format that relies on compression.
Aaron, we actually designed the container in such a way that it is possible to build an IRI scheme to point inside of it. It could be something like this:
epub:/OPS-Pub-Identifier/content-doc-filename/#idOr even use the more elaborate XPointer schemes when there don’t exist id’s at the points in the documents we’d like to point to.
@Jon: Vendor-specific IRI schemes already exist to do this sort of thing (jar:), but unfortunately new IRI schemes are unlikely to be officially approved, and less likely to be uniformly adopted by the browser manufacturers. Xpointer is even more of a stretch, and has few implementations yet. It would be better to have a way right now to use the standard http: scheme, since it was specifically designed for HTML filesystems and would make EPUB instantly accessible to the Web.
I do not understand the ins and outs of these issues very well, but it has been my impression with what I have seen of EPUB that some sort of internal linking is needed in the archive. This has been my principle frustration with FBReader. It is nice that it opens archives and all, but none of the local links work (with FBReader, this problem is not limited to archives — no local links function).
Web browsers handle local links well, but there is no mechanism for this in archives. How would a browser, or any program for that matter, follow a link from one document in an archive to another in the same archive?
My program generates multipage HTML files from a single Project Gutenberg text file. If I cannot link between files, what is the point of separating them in the first place? And what is the point of putting the files in an archive anyway as opposed to one big fat text file?
Yes, for the epub: IRI to stick, we’d have to get approval, which IDPF is certainly capable of doing (with Adobe backing us up we could probably run the gauntlet.) Of course, there’s the required application modules for this IRI which needs to be developed.
Certainly the kitchen sink http: could be used, and I assume it could be tweaked to carry something equivalent to the epub: IRI. But I’ll let the web developer experts chew on this. It’d still require special applications.
Standing back and looking at the bigger picture, we must have a “single file” for distribution, sale and archiving. Just like a book is a single object, an e-book MUST be a single digital object. I can’t imagine anything other than this, at least for a long-time. Even computers have what’s called “file systems” where the single file is still king.
Now this brings up what to do if not ZIP for this single file container.
Interestingly, one thing brought up is to place all the OPS files into a single XML document. Unfortunately we quickly discovered that Microsoft holds a key patent of putting multiple files representing a document/publication into a single XML document. This is the reason OpenOffice documents are a jar file containing multiple files. The OpenOffice folk originally wanted to encase all the files into a single XML document, but the patent issue stopped them. (This is what I was told, anyway – I’ve not verified this from other sources.)
Even with the XML solution, there’d be strong pressure to use compression, so again we are back to the issue of compression, which seems to be what the real issue in this discussion.
Btw, IDPF is considering something like the epub: IRI, and would certainly get it blessed so it is not “vendor specific” – just to clarify my prior comment.
LuYu, not sure what you meant by your comment, but EPUB certainly supports internal hard-wired links where one may jump from one content document to another independent of the spine.
From my understanding (and this is not a knock since I understand the difficulty of building reading systems), FBReader’s support of EPUB is pretty sparse.
That is, from my understanding the current FBReader cannot be considered a conforming OPS Reading System per the OPS spec. Over time it may reach the level of OPS support where it conforms to all OPS reading system requirements, and I hope it does.
Let a thousand flowers bloom!
LuYu, EPUBs support standard hyperlinks just like web pages use. The problem arises when a link from outside the book needs to point into it, or when a link needs to point from one book into another book. This is a problem that can be solved with good software, but not one that web browsers can currently handle well.
Jon, can you share where the impetus for compression is coming from? I presume that device manufacturers and publishers simply want to maximize the number of books that can be contained on one storage medium?
If compression is a political hotpoint, it would be good to consider requiring compliant EPUB reading systems to recognize two different containers, one compressed, the other not. For example a directory bundle with a .ops extension, and the .epub compressed version.
For my platform I needed to make the distinction between a web-friendly directory of standard HTML, an expanded epub directory that the software should render to the browser after some parsing, and a traditional epub file with a .epub extension that the browser should download and hand off to a reader like Digital Editions.
Just internally, I decided on “.epubx” as the epub-expanded extension, but I wouldn’t ultimately expose that to the end-user. It’s unsatisfying for a number of reasons: for one, I don’t want to ad hoc invent new formats or practices. But it’s also lousy because a directory/folder shouldn’t have a file extension — I’m not sure what’s the right approach here.
Aaron, I’ll have to recheck the OCF spec, but I believe that OCF does not require compression for any of the files. If compression is used, though, it must be the classic Flate (or whatever it is called).
Nevertheless, at least with regards to ZIP, a ZIP is a ZIP whether compression of individual files is used or not.
The issue of compression will always be political. But I agree that with today’s higher bandwidth and cheap ultra-high density disk drives where space is no longer an issue (both server and client sides), looking at the necessity of compression of text-encoded files can certainly be revisited. As noted before, Bill Janssen is a strong proponent of a MHTML-like approach. One can certainly imagine containing an OPS Publication within MHTML. As I’ve argued over the last few years, it’s really not the container which is the issue – it is the creamy filled center, the OPS Publication.