I often receive queries like:

I am converting the Word file of my book to a PDF (a bit later in epub). To be read on as many devices as possible, is PDF or PDF/A better? PDF/A-1a or PDF/A-1b?

This may seem like a rather nit-picky question, and the bottom-line answer is straightforward: stick to PDF/A to maximize portability, and the lower conformance level “b” is fine. But some interesting strategic points are illustrated by the details underlying this answer. …

Technorati Tags:
Adobe, e-book, e-books, ebook, ebooks, Paul Biba, PDF

As well, my work at Adobe these last three years has been centered around a relatively prosaic objective: establishing open standards that enable multi-channel/cross-device distribution of eBooks. For all intents and purposes, this work is done: epub is now firmly established as the industry standard for reflow-centric eBooks. That took a considerable effort, on the part of many people, and I’m really proud that we did it. But… that was the easy part: essentially migration of print to digital. epub does take portable documents to the next level – breaking past beyond PDF’s paper-replica model. But that’s only the beginning of the fundamental reinvention of the book that digital content and the Web will enable. In other words: now it’s really going to get interesting. I expect my future work, and this blog, to focus on this transformation.

Among various projects, Bill and his team have been involved with Adobe Reader Mobile SDK, Adobe Content Server, Adobe Digital Editions and Adobe InDesign. Bill sits on the board of the International Digital Publishing Forum, the e-book industry’s main trade group, and has played a major role in the ePub standards movement.

To put it mildly, Bill and I haven’t always seen eye to eye, but the “best of luck” is far from mere politeness. One of TeleRead’s main missions, as I see it, is to encourage open standards—so that people can own e-books for real, without worrying about new formats and DRM. Not all our contributors are as fervent about it as I am. But that’s my priority, at least, in addition to the fight for a well-stocked national digital library system. How has Bill figured in this? By corporate standards, he has been a positive phenomenon despite our differences.

When Jon Noring and I set up OpenReader, Bill was smart enough to realize that the IDPF needed to get more serious about open standards before we or others preempted the group. The result? ePub. Dozens of e-book devices use ePub or will use it—for example, the Sony Readers—and major publishers are standardizing on it. You can thank Bill more than a little for that. Similarly, it was Bill who had the courage to call for social DRM, the use of which would make it far, far easier to own e-books for real than the usual DRM would.

About Bill’s future hopes, he looks “forward to the opportunity to push for open standards and interoperability from a completely neutral perspective. I can be reached at whmccoy ‘at’ gmail.com.” Great priorities, Bill. Again, good luck!

Meanwhile I’ll be interested in knowing who ends up as Bill’s replacement on the IDPF board, and how that person feels about issues such as open standards and DRM. I suspect a special election will come up soon.

Bill writing on his continued involvement in the industry—and the need for vigilance: “…there is no doubt that I’ll continue to be involved in the future of digital books, especially where that future intersects with web standards and open source. I believe that Adobe will continue to play a critical role as an enabler of interoperable solutions, but I also believe that the community needs to stay vigilant to ensure that for-profit corporations don’t just talk the talk about being open, but also walk the walk. I’ve certainly tried my level best to do this from the inside, but look forward to the opportunity to push for open standards and interoperability from a completely neutral perspective.

Detail: Although Adobe is laying off people, Bill writes: “While my transition comes as Adobe is restructuring, it’s really more about Adobe creating a new, expanded organization focused on digital publishing media monetization. My team is the nucleus of this group, and the additional wood that Adobe is putting behind this arrow should be great news for our customers and business partners. We’ve got a lot of exciting stuff on our roadmap, and it’s been an exceptional honor to have worked with such an incredible group of people. But, it was a logical time for me to move on.”

Adobe’s interests are far more aligned with the adoption of open formats that we can address with our authoring tools and services. We see e-book DRM as an enabler for a larger market, one that we can address with our tools and services on a level playing field vs. proprietary silos that give a choke-hold to players who have a strong position in the book distribution value chain. We don’t forecast revenue from e-book DRM even in the best case ever itself being a large material business for Adobe—large enough to pay its way and allow us to sustain and enhance the solution, but quite small relative to our other tools and services.

I’ve said before that Adobe’s open to evolving towards a non-proprietary ubiquitous DRM standard, even as we see obstacles to getting there in the near future (especially around IP enablement). What we don’t want to see is one proprietary solution taking over control, or fragmentation of multiple proprietary solutions. Publishers have already voted with their feet, so to speak, by requiring/letting Amazon deploy DRM. If the only reasonable cross-platform alternative for eBooks was to go DRM-free, then some publishers would distribute only through Amazon, leading to everyone else getting "iTunesd". So to me it’s pretty obvious that cross-platform eBook DRM that works with EPUB and PDF is necessary to ensure that the open, cross-platform alternative wins. But if every publisher were to choose to go DRM free, using PDF & EPUB but not DRM, hey I’d have absolutely no problem with that outcome.

Adobe has just released, under BSD license, EPUBGen, a Java library that generates EPUB.

To quote from our digital publishing developer blog:

EPUBGen is a Java library that demonstrates EPUB generation from a variety of document formats, and which may be a useful starting point or reference code for other EPUB generation needs. That is to say, it’s an effort to promote the development of a variety of tools and workflows.

EPUBGen has both a set of back-end code generation modules and front end format importer modules. The back-end modules generate EPUB and illustrate more advanced functionality, including font subset embedding with obfuscation

The code itself can be found on the epub-tools Google Code site, which includes other sub-projects witg Python/XSLT scripts for generating EPUB from DocBook and TEI XML. For more of the gory details on font embedding with obfuscation (aka "mangling"), which illustrates the recently published IDPF Tech Note about same, see this related blog post.

Contrary to misunderstandings, for example, the IDPF’s format standard for e-books does support inline charts and graphs well.

In particular, EPUB supports SVG which is, not coincidentally, a very close analog to PDF page contents.

So anything one can represent on a PDF page or portion thereof can be represented in  EPUB, with real, selectable text and scalable vector graphics. 

A progressive academic journal provider, Hindawi Publishing has developed an innovative workflow that results in EPUB with inline SVG for mathematical figures and illustrations. For example, if you open this example in Adobe Digital Editions, you will see that the equations are zoomable and that text within the equations is live and zoomable. Check out other Hindawi examples.

Visual richness in other respects

Beyond SVG, visual richness in EPUB is further enabled by embedded font support, CSS styling, and the ability to embed rich media objects (including Flash where supported).

As well, there is an experimental extension to EPUB for page templates based on XSL-FO. Dynamic conditions, for example, can enable things like customizing the decision points for when to shift between 1/2/3 columns, etc.

Since the Association of American Publishers and others have requested IDPF work to further enhance EPUB in the area of page layout support, I suspect that some means of doing this will become part of the base standard at some point.

Good reflowable text: Main focus—but options exists

That being said, the raison d’etre for EPUB was primarily to transcend the final-form page limitations of PDF, and better enable smaller-screened mobile devices.

So the current priority, especially around authoring workflows, is relatively basic reflowable text.

And, to maximize accessibility it’s a good thing to consider that every publication should have, insofar as it’s possible, a completely structured and reflowables/alter text-based representation. In other words, all the visual richness stuff should be considered optional enhancement/alternatives.

But what if you want to go beyond optional? You most certainly can. EPUB isn’t and won’t be just for basic viewing when sophisticated hardware is in use.

Granted, a number of so-called EPUB viewers out there don’t handle SVG. But that’s going to change very rapidly.

Bill McCoy has played a key role in the development of e-book standards at the IDPF. In a different form, this appeared as a post to the Reading 2.0 list. We’re running it with Bill’s permission. By the way, Bill prefers “EPUB” over “ePub.” We’ll go along here, but the TeleRead style will remain “ePub” since it’s easier on the eye. I hope that Adobe and the IDPF, both of which use “EPUB,” will consider “ePub” since it will look better in material aimed at consumers—so often the ultimate drivers of standards. – D.R.

Steve’s book starts far too slowly. But like a train racing along in the open countryside after a pokey departure from a downtown station, this one really picks up. I can’t wait to read what happens to Steve’s hero, Stanley Branford, the victim of the ID theft.

The novel is a Kindle-format book written under the name of Steve Holt. And if I owned a Kindle II E Ink gizmo, I could go on an e-walk outside and automatically resume where I left off on the Kindle app on my LCD-lit iPod Touch. Caught up in the suspense of a courtroom scene, I wouldn’t have to waste time searching through the book for same “page.”

Time for open syncing capability

I’d love to see the synced-autobookmark feature in many other e-readers—with interbrand capability and the standard ePub format, and ideally without DRM to muck it up. That way, I might start in Stanza or another app on the Touch and go on to enjoy Say My Name on my Stony PRS 505, which, like the Kindle, has sunlight-friendly E Ink.

This scenario of open synching would be a win for most everyone, vendors and readers; even Amazon would benefit in the long term since e-book would be less of a hassle. The Lexcycle people, the developers of Stanza, loved the open sync idea when I discussed it with them over lunch at Tools of Change. The key is to have the right standards for syncing and reading among different apps and devices, and meanwhile, if Steve can release an ePub version of Say My Name, then so much the better.

Another friend of the sync idea: Bill McCoy at Adobe

Now the sync idea has yet another supporter, IDPF board member Bill McCoy of Adobe (who, you’ll recall, has joined Lexycle and others in championing another good idea—open distribution, formally proposed by Lexcycle).

In an e-mail responding to my query, Bill has actually gone beyond me, and I like where he’s coming from:

When you say "find their old places", do you mean stores/repositories or digital publications? Anyway, I would say "absolutely" to either.

The latter is a feature that I like to call "bookshelf in the sky.” Amazon Kindle now has a closed proprietary version of this with their iPhone app (and the experience as you upgrade from one Kindle to another), as well as a separate silo in their "Amazon Upgrade" [sic] browser-based reader.

Part of this is up to publishers. Forward-looking publishers realize that if they sell a digital book to someone they should enable them to read it wherever and however they like.

The former—finding preferred sources—is analogous to favorites in a browser. Makes sense and not particularly hard.

Whether it’s via a bookmark arrangement or maybe an interlibrary or interstore virtual "locker," e-bookdom would benefit from the wherever-and-however concept. Toward this end, both open distribution and open synching have important roles to play.

Update, 11:57 a.m.: Tim O’Reilly likes the open sync idea.

Disclosure: Steve not only is a TeleRead contributor but also said nice things, in his newsletter, about The Solomon Scandals (he decided on his own to write up my book). Judge for yourself, but I think that once you’re past the start of Say My Name, you’ll get caught up in Steve’s own book. I originally was planning to mention Say only at solomonscandals.com.

Stanza, the leading iPhone eBook software, includes an excellent online catalog system that enables users to seamlessly acquire free and commercial content from within the application.

The Lexcycle team built this system in an open, extensible manner using Atom. Adobe and Lexcycle have been working together on Adobe PDF and ePub eBook support, and now we are deepening that collaboration in working together, along with the Internet Archive and others, to establish an open architecture enabling widespread discovery, description, and access of book and other published material on the open web. The Open Publication Distribution System (OPDS) is a generalization of the Atom approach used by Stanza’s online catalog.

I’m grateful to the Lexcycle team as well as my friend and colleague Peter Brantley for their efforts on behalf of open access and interoperability.

This work is at a relatively early stage, as evidenced by the “DRAFT” notice on the specification wiki and the intentionally lightweight process (i.e. not yet involving a de jure standards body). We are taking a page from the way Atom itself was nurtured in the early going. If you are a reading system provider, or a distributor of free or commercial digital publications and are interested in signing on as a supporter and contributing to the definition of OPDS, get in touch with Peter, myself, or Marc Prud’hommeaux at Lexcycle.

On-ramp for online stores and other sites

I believe this effort has the potential to be a critical enabler to the growth in access to and adoption of digital books. For consumers, OPDS will deliver seamless integration of convenient acquisition from many sources, on any device or reading system, without lock-in to “One Store to Rule Them All.” For content distributors, ODPS will enable reaching consumers across multiple reading systems and devices: not as a replacement for online Web stores, but as a valuable supplement and on-ramp. For reading system developers, OPDS will make your device/application more useful and valuable. Stay tuned for more soon about how OPDS will be utilized within the Adobe digital publishing solution set.

Moderator’s note II: Please note that Feedbooks has been working to allow its catalog to appear within various e-readers, via APIs; and in fact FBReader has this capability. In the past, Feedbooks and Lexcycle have cooperated, and I suspect that the Lexcyle idea is building on these past efforts. Significantly, everyone involved here supports ePub, and I can see terrific synergies. Content shouldn’t just be easily found; it should be able to be easily read, in the technical sense.  – D.R.

(Originally posted under a different headline in Bill’s blog. The current headline over this post is mine, not Bill’s. – D.R.)

Now, let me say up front that I don’t think ebook DRM is "good good good" any more than I think that of taxation, standing armies, or the proliferation of nuclear technology. But although one may dislike taxation, one may dislike even more the likely consequences of eliminating taxes (diminished schools, roads, law enforcement, …). Peter’s post focused on negative attributes of DRM in isolation. But to me, the important thing is to look at likely outcomes given various scenarios, and to consider what these outcomes would mean for the principal actors involved (authors, publishers, and readers). Not whether something is good or bad but whether it’s better or worse than the likely alternative.

To me, it’s pretty clear that the establishment by the industry of a broadly adopted cross-platform ebook DRM system should lead to a significantly better outcome for all concerned than if no such platform ends up getting established. "DRM" is a somewhat loaded term: to clarify, by "ebook DRM" I mean a relatively lightweight means of limiting and/or discouraging copying and use beyond publisher-permitted limits, intended more to "keep honest people honest" than to totally prevent copying. After all, a book can be scanned and digitized, or even re-keyed, with only a middling level of difficulty — so aiming for "ironclad" DRM is not warranted, even if it were feasible.

Your thoughts?

Peter, executive director of the Digital Library Federation, argues that DRM  isn’t worth messing with. Bill undoubtedly will be talking up DRM as useful technology for libraries and other loan-related apps. I just hope he won’t be as gung ho about DRM for retail purposes—DRM is a sales toxin that penalizes honest book-buyers without stopping piracy. Bill himself has written the following in the past: "I would like nothing more than to have DRM technology just fade away." He didn’t say Adobe should stop doing DRM for publishers wanting it, but even he recognizes its major negatives.

Small publisher vs. DRM

Others go further. My publisher, Lida Quillen of Twilight Times Books, successfully urged eReader.com not to DRM-hobble her books. And that reflects the commonsense of many a small publisher. I just hope the big boys will wake up about DRM’s downside, especially in retail. People want to own books for real, a point that Peter himself has eloquently made.

You can read Peter’s first shot, in the O’Reilly TOC blog, and Bill’s reply will follow there. Ahead is one of the best arguments in the Brantley post:

DRM systems do not age gracefully. Because they are tightly coupled systems — a series of protections that must all be aligned in order to permit proper functioning — DRM is particularly sensitive to technical obsolescence. Thus the value of content secured by DRM is inherently reduced compared to its unrestricted use state, because its anticipated future value is less. In turn, that reduces the ability of a DRM protected system to maximize revenue generation over the content’s lifecycle — and for books, the potential value should last more than my lifetime, but be able to be handed down to my daughter, now 7 years old, and generations beyond that. Libraries are not just about access, but preservation for the ages. DRM breaks libraries.

To Bill’s vast credit, he has acknowledged that social DRM could be a useful alternative to traditional DRM. From his past writiings:

Adobe is committed to continuing to supply DRM technology in our solutions as required by our publisher partners. We will continue to work to make DRM as seamless as possible for end users, while also protecting rights holders from piracy or unauthorized use, and we are poised to deliver some innovative new capabilities in this regard.

Yet, I would like nothing more than to have DRM technology just fade away. After all the main challenge we have in digital publishing is to get it adopted by mainstream consumers. And the main challenge 98% of book authors and publishers have is to get people to be aware of their books, not to prevent piracy. So my challenge to print publishers and authors: why not support ‘social DRM’, rather than heavyweight DRM? If that’s a direction you are willing to go, Adobe will back you up, 1000%.

Hmm. Isn’t it possible that Bill should be encouraging Adobe, the IDPF and its corporate members to check out social DRM for real? And what happened to idea of the IDPF doing a logo for DRMless ePub? Let’s not forget the main show, e-book standards. If the IDPF even loosely links ePub with DRM, it will be a setback for a far-more-important cause.

At one point I was reluctantly open to interoperable DRM, but more than ever these days, I wonder if the IDPF shouldn’t spend its limited resources on more useful activities. Even without DRM, e-book standards over the long run will be a challenge to maintain. DRM, given its vulnerability to technical obsolesce, just adds to the complexity. Do publishers really want to spend millions over the years on a so-called DRM standard? I’m a writer, and even a "standard" could mean some money out of my pocket—not just indirectly in the cost of the technology but in lost sales to DRM-hating readers. Forget all this hype about "seamless DRM." Sooner or later, given DRM’s complexities and inherently closed approach, there’ll be ugly gotchas for publishers and consumers.

Apple is unlikely to be able to pull an iPod here, but not because Amazon has any kind of insurmountable lead in e-book selection. I don’t think that’s the case, not at all.

First, the selection of commercially relevant e-books at the Kindle Store is still very thin. Less coverage of what really sells in trade than a decent airport bookstore. Much of the “vast” Kindle Store selection is filler eDocs. Some major publisher lists are MIA. This has already been discussed on the list so I won’t belabor the point. But the race to get everything that sells in digital isn’t over, it isn’t even half over. When you get beyond U.S. market, it has barely begun, and Kindle is not the leader.

Secondly, the aggregation of e-books by Ingram and others includes not only the content but the commercial relationships that enable multiples downstream retailers. To map to physical book value chain, they are not just distributors, they can act as wholesalers. In this model the publisher retains more control over the pricing (vs. a retailer being able to impose their own will), but that’s not necessarily a bad thing.

Thirdly, these integrators already have close to the level of commercially relevant content as Amazon already—in some respects more content—and this will quickly evolve from the less-mobile-optimal PDF to also being available in ePub. As publishers make content available to channels it’s going to be in ePub, and Amazon will not only not have any exclusive, but will have work to do to convert it into its proprietary format (potentially with mixed results with regard to quality).

Last but not least, retailers can pull from multiple aggregators to maximize selection. I know already of several online retailers getting "feeds" from more than one, and I expect this to continue. Major retailers can go direct to get unique deals from publishers but will have as a backstop essentially all the commercially salable content via the union of what’s available from multiple aggregators.

Related: ePub for the Kindle—and here’s why. – D.R.

I’ve long since given up massively collecting the “old 78’s”, and today have only kept a few favorites. One favorite I kept, a quite rare classic jazz recording from late 1928, is shown to the right. [note 1] My experience collecting older sound recordings has given me some unique perspectives as it relates to media, e-books, copyright, conversion, archiving, formats, etc.

In a seemingly unrelated story (yes, I will tie all this together to e-books!), last weekend I was looking at my family’s fairly large VHS movie collection, wondering what to do with it. I’ll probably just give it away since who will buy it? I reflected that a lot of money was spent over the years to acquire these analog video tapes, and now they sit on the shelf, gathering dust, taking up space, never to be viewed again by my family.

Nor will I digitize these video tapes (because of the time and cost to do it right, and because they are of limited NTSC-grade fidelity) since we have moved on to DVD and probably soon will switch to high definition. So instead of converting these analog VHS movies, we’ll probably spend more $$$ to simply replace some of them with high definition digital versions – essentially buying the same movies twice.

Sound familiar?

Tie to e-books: Obsolescence

What do these two small tidbits in my life have to do with e-books and digital publishing? And what about ePub? Lots, really.

The common thread that runs through my two stories is media obsolescence:

1. Obsolescence (as it relates to difficulty of playback and transfer) of the artifact containing the audio/video recordings, and

2. Obsolescence (lower fidelity) of the underlying audio/video information.

With e-books as digital files which we can download (or can trivially retrieve from CD-ROM and DVD-ROM), we solve the type 1 obsolescence since we don’t have to carry the digital bits on any particular, difficult-to-transfer artifact. Transferring them with “no loss” is as easy as typing “copy” – no need to spend hundreds or thousands on special equipment to “transfer” the publication.

The second type of obsolescence, though, requires a little more discussion, delving into some of the themes I wrote about in a 2003 TeleRead article: OEBPS: The Universal Consumer eBook Format?

“Flexibility” and “High-Fidelity” are important as affirmed by AAP

Very recently the AAP, whose membership includes all the major American publishers, released an open letter with a strong endorsement of IDPF’s ePub. The following two points in AAP’s letter are germane to this article:

1. AAP sees retailers selling ePub directly to consumers (refer to AAP’s clarification on this point), as well as selling derivative formats converted from ePub. Publishers understand the great flexibility that ePub provides.

2. AAP uses the phrase “high-fidelity” to describe ePub. This mention means presentation quality is important to AAP, and thus should be important to everyone else in the e-book industry. It also acknowledges that indeed ePub is “high-fidelity.”

It is clear that publishers consider “flexibility” and “high-fidelity” in e-book formats important, for themselves, for the rest of the industry, and for consumers. And ePub is a format that meets these requirements.

The case for ePub as a consumer format, and its similarity to web content

A couple e-book commentators have slammed using ePub as a consumer format, arguing that it is not “digested” enough (my phrase) to be read on limited resource handheld devices (maybe “dumbed down” is a better phrase?) In their view ePub should only be used as a “master” that the consumer never buys nor handles, and from which the publisher/wholesaler/retailer converts to proprietary formats.

What they really are advocating is that the consumer should only purchase a digested “low fidelity” format optimized for their particular handheld device, a device that is most likely here today and gone tomorrow. (Just like low fidelity 78 RPM records and VHS were “digested” and tied to their particular playback devices – where are they today?) If the consumer gets a different handheld device, they will need to buy the book again in a different, proprietary format. [note 2]

As just noted, AAP itself rejects this limited view of ePub and is supportive of repurposeable, high-fidelity ePub being sold directly to, and read by, consumers.

After all, there now exist ePub reading systems (for example, Adobe’s Digital Editions), and since the specs underlying ePub are completely open and unencumbered, anyone can build their own ePub reading system (or “user agent”.) End-user flexibility in using the e-book is important to AAP.

(Not to mention the possibility with DRM-free ePub that end-users could use tools to convert ePub to some other temporary format as they need, whenever they need it. This possibility has not been discussed.)

Furthermore, today’s “limited hardware” is actually quite close to being able to directly render ePub (filtered to lower fidelity as needed by the device) since ePub is essentially close to a packaged web site. The hard work in rendering ePub is parsing XHTML, decorating the “parsed tree” with CSS (techies here know what I mean by this) and splashing the character glyphs on the screen, which is exactly what web browsers have to do.

Many handheld devices today include web browsers. In essence, when web browsers become ubiquitous in most handhelds, then directly rendering ePub on all these handhelds will be very easy to support. That is, if the handheld can display web pages, it has the requisite horsepower to natively display ePub.

With AAP’s strong endorsement, and the release of thousands of books in the ePub format in the near future, I see more interest in modifying web browsers (and it won’t take much) to directly read ePub, as well as specialized ePub readers. Let a thousand flowers bloom!

DRM: The monkey wrench in the gears

Obviously, DRM throws a huge monkey wrench into the end-user convertibility of DRM-protected ePub. But then DRM throws a monkey wrench into all format discussions, so what’s new?

It is hard to predict if the e-book industry, as it matures, will fully embrace DRM encryption, or will decide to forego using it (or only use what Bill McCoy at Adobe calls “social DRM”, where the book copy is unencrypted but identifies the individual buyer.) It does appear the music industry may drop heavy-duty DRM. I hope the major publishers will do likewise.

As I wrote about a while back, I believe DRM encryption is not needed by publishers to maximize their profits and maintain sufficient control over the use of their content. “Social DRM”, combined with digital signatures, is probably more than adequate protection for published e-books and other types of digital publications.

Summary

I see ePub (and its compatible successors) being the reflowable, high-fidelity e-book format that consumers need to only buy once, and have confidence the format will not become obsolete anytime soon. And ePub is designed for convertibility – when a new, high-fidelity, open format replaces ePub in the future, conversion of ePub to that new format, without loss of information, will be possible.

Even if today not all handhelds have the required horsepower to directly render ePub, they soon will, mainly because there’s strong consumer demand for handheld devices to include web browsing capability (even if rudimentary.) ePub is essentially close to a “web site” packaged in an ordinary ZIP file, an important fact which a lot of people strangely seem to miss.

If I have the choice of buying a digested, low fidelity, proprietary e-book format which can only be read on a particular device, or a high-fidelity, open, repurposeable format that can be read on all devices (both natively and converted as needed to other formats), now and into the future, which would I choose? Which would you choose?

I do not want to buy the same book over and over again in order to read it in some needed proprietary format, and to always be stuck reading my e-books in “low fidelity” whenever I have a device capable of higher-quality presentation.

Yes, Bill: there’s hope in .epub. It’s the DRM-standards part I’m worried about: this is no small detail. Meanwhile perhaps you can return to your interest in social DRM, which could help address interoperability problems. Why isn’t the e-book business getting serious about SDRM? Or maybe watermarking? Random House seems pleased so far with watermarking for audio books and is ditching DRM requirements in most cases. Why are e-books so bleepin’ different?

Question of the day: Who’s Bill referring to when he writes: “Tellingly, one major eBook retailer, despite promoting their own proprietary format, has quietly begun accepting EPUB submissions from publishers”? Amazon? Fictionwise? Another? Of course, in Hachette’s case, retailers haven’t any choice. Laudably, .epub is now standard distribution format from Hachette, which owns Warner Books. Oh, for consumer formats to catch up—ideally with traditional DRM not cluttering up the works!

You’ll be able to tie your books to your personal ID, not suffer the usual machine-linked approach. The ML approach is torture if you own a whole bunch of e-book-capable machines—or when a hard drive goes south, as they’re all likely to do in time.

Perhaps with that in mind, eReader has been using an ID-based approach for eons by way of encrypted credit card-related information.

Oink! Oink!

Adobe’s plans are Good News even for us DRM haters, but they’re still just lipstick on a pig. Even Adobe concedes the “inconvenience” of “the user ID and activation processes” associated with Named Activation. And yet we know people want e-books on many devices, which eReader allows, via its credit-card-linked approach. Will Adobe Digital Editions, too, rely on card-linked IDs? My hunch is no. What I can say, however, is that I still hear LOUD oinks.

As I’ll show later in this post, even Adobe’s new DRM (as in “New Nixon”?) could be a long way from “iPod simple” if you include the registration process.

Two positives…

Meanwhile here are two positives:

1. Adobe Digital Editions can automatically convert already-bought, machine-tied books—Easy Activation books, that is—to the human-tied Named Activation.

2. If Digital Editions makes it as expected to the Sony Reader, then perhaps you can use portability of the DRM to be able to read the same books on your other machines—ideally in the IDPF’s .epub standard, toward which Adobe made many valuable contributions.

…but a pig’s still a pig

Even so, the lipstick won’t squelch the oinks or cover up the snout or stench. I wonder what device limits if any might exist. An Adobe FAQ mentions up to six desktop machines and six PDAs (numbers applicable to Digital Editions?), but as reading devices proliferate, even that might not be enough. I’d hope that DE 1.5 wouldn’t have a limit.

While I appreciate Adobe’s new flexibility, as long as it’s making us register, why can’t we use the social DRM approach that Adobe’s Bill McCoy so wisely talked up some months ago? Much better would be no DRM. But at least the social DRM approach would let you read your purchased e-books forever. So how come Adobe hasn’t provided shoppers with that option in cases where publishers would allow allow it?

No, Adobe isn’t going to vanish tomorrow, but even the largest corporations tend as a rule to fade away eventually, and I want my books to be outlast both the company and me. Permanence—isn’t that one of the ways in which cardboard-and-ink books differ from most other media? Why can’t e-book catch up, at least a little, in that regard? And speaking of DRM and “forever,” is there a grandchild provision? Is Adobe providing for ways to bequeath personal libraries?

Too Rube Goldbergish even for Adobe

As much as I’m an e-book-lover, I remain an e-skeptic. DRM is no small reason why.

Technology keeps changing, and as hard as Adobe may commendably try to simply matters, DRM makes e-books horridly Rube Goldbergish. Just show this page from Fictionwise—an innocent bystander—to your techophobe friend.

On top of that, when I Googled up an Adobe activator site for recent information, the “Sign up for an Adobe ID” feature wasn’t working. At least on my Firefox brower, the “select a country feature” didn’t function when I was checking out the latest wrinkles of the registration process. A new URL? If that’s the problem, then we have one more illustration of an inherent flaw of DRM—the fact you need stability for it to work, the very antithesis of what technology is all about.

My experience with the activation site, current or discarded, is not good news either for Adobe or e-books as a whole.

Imagine, too, how I feel when I see statements such as, “If you are contemplating buying a new machine, please consider keeping your old machine intact unless you bought all of your books with Acrobat or Reader using Named Activation.” Even if your disk crashed?

Perhaps this is yet another lesson in the follies of relying too heavily on the vagaries of bits and bytes.

And maybe others have learned it better than Adobe. If DRM’s so great, why is it that Apple, the very company whose audio players supposed are “iPod-simple,” wants to move away from the technology?

Dialogue sought

Far from being an Adobe hater, I hope the company will regard this as constructive criticism and go on to simplify the activation process if it insists on DRM—and experiment with social DRM, which at least would make books permanently readable without any gotchas. Hey, Bill. You were on to something good in talking up social DRM. Follow up with some action, both at Adobe and within the IDPF! I’d welcome responses from Bill, DE’s Peter Sorotokin or others at Adobe. Just why can’t Adobe give social DRM a shot, in line with Bill’s suggestion?

Detail: Social DRM, as far as I’m concerned, could work with or without registration. Even the supplying of a mere e-mail address would remind people that this was copyrighted material. No utopias expected. Via the right business models and systematic efforts to build communities around books, publishers can help keep honest people honest.

Related: A different perspective from Alex at MobileRead.

Technorati Tags: Adobe , DRM

At the same time, however, Greg writes on a Gutenberg list that he needs convincing evidence that .epub will indeed be an open, honest standard without gotchas coming in from Adobe or any other company. He’ll also need the right software tools—free and open source.

“On the fly” explained

But first, what does “on the fly” mean? It means that Gutenberg would treat .epub as it now does Plucker.

You’d type in a number to identify the e-book file, then wait while the conversion gears ground away and generated .epub from another format such as HTML or .txt. This isn’t an optimal solution, but it’s a good start, especially if Gutenberg also uses direct links to sites with ready-to-go .epub.

Catnip for consumers, if IDPF doesn’t play games

The benefit for Gutenberg visitors would be for future Sony Readers—expected to come with Digital Editions, Adobe’s software that can read .epub, not just PDF—to be able to read .the IDPF format without conversion hassles at the human readers’ end. The same could happen with Bookeen‘s forthcoming Cybook Gen3; in fact, an entire generation of E Ink machines with .epub-reading capabilities, whether or not they originated from Adobe software, which apparently won’t happen in the case of the Cybook.

Adobe funds the IDPF, whose executive director, Nick Bogaty, is about to start a job there. While the public domain community will benefit from .epub and mustn’t walk away from the possibilities by ostracizing the IDPF just because Adobe’s involved, we also need verifiable assurances that no one will compromise the integrity of the standard. Integrity is the key to many different brands of commercial software and hardware—not to mention open source freeware and shareware programs—working with .epub from Gutenberg and other sites.

How public domain sites could help the standard

If the public domain community embraces .epub, even less than fully, such as through the on-the-fly approach, it would be a significant victory for the standard. Many more more people probably read public domain books than those in the DRM-infested proprietary formats from large publishers.

The cranes and wrecking balls would finally be at work tearing down the Tower of eBabel, or at least we’d be closer to this long-awaited event than we are now. That would help everyone, from Project Gutenberg and self-publishers to Random House, Simon & Schuster and HarperCollins. We need to make e-books as easy to use as CDs; nontechie consumers are sick, sick, sick of eBabel.

Greg’s wise conditions

While showing flexibility, Greg also insists on conditions, wise ones as I see it. For example, he doesn’t want DRM inflicted on Gutenberg and he demands free, open source software that can assure that an .epub file is really an .epub files, as opposed, say, to something with PDFish elements hidden inside.

Amen to that! I want there to be provisions for checking the files spewed out by .epub related tools, as well as a systematic way of guaranteeing that e-reading software is truly compatible. Otherwise such programs—both the creation and reading varieties–shouldn’t qualify for an .epub logo. Furthermore, the tools should be able to be vetted by third parties, independent of Adobe and even the IDPF, to make certain no one has compromised them.

Other open source .epub tools needed

The goal as I see it should be to allow the existence not just of commercial software for creation and reading but also the free, open source variety—a much-needed alternative to the present mess where individual corporate interests come ahead of those of e-bookdom and society at large.

In Gutenberg’s case, there’ll need to be linux software to create .epub from another format, once someone punches in the number. Might Adobe, per chance, be willing to pay the costs or otherwise facilitate this? It’s a member of the Open Content Alliance., and Adobe’s Bill McCoy has portrayed himself a friend of the public domain. Helping Gutenberg, without strings attached, would be a great act of goodwill. Or what if a foundation such as MacArthur or George Soros’s Open Society Institute–without any commercial interests—provided the funding? Or how about Brewster Kahle, the philanthropist behind the Internat Archive and related projects, including OCA, which could benefit mightily from a standard, reflowable format fit for cellphones and PDAs, not just desktops? Could he finance open source software useful to OC and Gutenberg alike?

Or, as a funder for conversion software, not just for Gutenberg in particular, how about the American Library Association or another large library or university group as a funder? Or maybe even a grant from one of these groups to the Digital Library Federation, as an overseer?

Volunteer programmers might also be the solution for Gutenberg and similar groups. Anyone from the MobileRead programming community care to participate?

But what about format validation and related precautions—a prerequisite for Greg to be interested? The IDPF apparently has no such animals right now. I’d welcome a roadmap with an ETA and assurances that the tools will be free and open source. Given Adobe’s participation in the Open Content Alliance, would the company be willing to spin off the job to a project overseen by someone like Brewster, who is far more trusted in the open source and public domain communities than Adobe is?

The rewards of openness: Avoidance of proprietary hell

David Moynihan’s struggle with the use of Mobipocket at Munsey.com shows the difficulties of working with the alternatives to openness, proprietary systems. I want an open approach and ready-made solutions for small guys like David. WordPress hasn’t done too bad a job getting the masses online with blogging. We need an open source WordPress equivalent for book publishers, including the “self” variety. The big boys can still sell their CMS systems and offer services, too, not just software. But WordPress is there for the masses.

By the way, I’m wondering if none other than the WordPress people or the Drupal people could do .epub-creation modules. Plus, an OpenOffice plug-in would be nice. The .epub standard is too new for powerful open-source tools to exist now.

If the IDPF uses gotchas from Adobe or other companies to make open source impossible, yes, I’ll join Moynihan in his grumpiness. But let’s give the IDPF a chance rather than coming up, as his Reg commentary did, with a pack of distortions and lies. Consumers are begging for an end to the Tower of eBabel, and the public domain community shouldn’t try close off the options, as long as they’re not just Trojans from Adobe or elsewhere. I take it for granted that software companies will attempt Trojans–hence, the need for well-vetted validation and other precautions.

Adobe bastardizing .epub already?

In that vein, let me quote from Aaron Miller of Book Glutton, a small company that is eager to use .epub while at the same time insisting on its integrity.

“It makes me nervous to have Adobe so prominent in the IDPF. They seem committed to the standard, but they’re using XSL formatting objects for their XHTML .epubs, and it would be very difficult to get a web browser to gracefully use XSL-FO.

“If they decided to lean heavily on that, to the point that .epubs without an .xpgt file (their XSL stylesheet) became unreadable in DE, then we would have a problem.”

Hello, Adobe and IDPF? What’s your response? Can you assure Aaron that you’ll keep .epub and Adobe-specific things separate? Meanwhile he also reports he can’t even get DRMed filses to work in Digital Editions.

Different issue from the specs

That said, keep in mind that these are Adobe issue, not those with the actual specs, which a bunch of other companies have vetted. Let’s not repeat the same confusion that David Moynihan did in his nutty commentary in the Register.

What will count is close monitoring of the IDPF and Adobe to make certain that we don’t get a de facto Adobe-owned standard.

This is why the open source community needs to embrace .epub and get the right tools out there—so that plenty of people will notice if notice if proprietary gotchas creep into .epub, either as a spec or in the world of implementations.

A warning to the IDPF and Adobe

In Adobe’s place, I would take fast action now–to regard to validation-related tools and other safeguards, as well as either helping Gutenberg or encourage for others to do so. I also suggest that Adobe encourage creation of free open source tools in general, plus go ahead with the phased-in logo plan I’ve suggested, so as to decouple the format and DRM issues.

Procrastination will harm the standard. I jumped out of OpenReader in part because the Consortium didn’t do implementation well (not entirely OpenReader’s fault since it lacked sufficient support). Adobe’s Bill McCoy earlier was joking about a Cargo Cult mentality—the expectation that the software would just materialize for the OpenReader standard, simply because OpenReader deserved it.

Same applies to .epub itself, Bill. If you sincerely want it to be a standard, an admirable legacy for you and Adobe alike, then you need to look beyond the current commercial solutions and nurture open source as well, for the sake of .epub’s long-term credibility and the cause of e-books as a serious, trustworthy medium.

Who knows? The open source community might well come up with concepts that would show up in commercial software; potential synergies abound. IBM hasn’t done too badly by reaching out to and encouraging the open source community, and by moving somewhat from a product-focused approach to a services-oriented one.

That just might be a rather attractive role model for Adobe.

History vs. the current and tangible: .epub-related code

Such actions would help Adobe overcome a rather understandable fear of the company in the public domain and open source communities. Some years ago a Russian programmer visiting the U.S. went to jail for circumventing Adobe’s DRM. The company has also worked to make use of its standards a legal requirement for those dealing with the U.S. government. David Moynihan raised these issues in the Register, and I certainly can understand why.

What counts, however, in the end, won’t be history but something current and tangible, at least on computer monitors—.epub-related code. I hope that Adobe and the IDPF will listen and learn from from OpenReader, which was valuable in prodding the IDPF to get, but which failed as a real-life standard.

Although major publishers will be releasing .epub books with help from translation houses, let’s not forget the smaller publishers and the public domain sites whose e-books have probably found many times more readers. Their use of .epub will great speedily its adoption, thereby benefiting both Adobes and small-fry alike.