From a post on TUAW by Erica Sadun:
I spent a good bit of time this morning taking a peek under the hood at iBooks Author and the files it builds. By request, here is a quick summary of some of the information I gathered on the topic. I warn you that this is going to be a non-generalist post, so do feel free to skip ahead on the site if this kind of info isn’t really your thing.
The iBooks format appears to be an EPUB-like variant specific to Apple. Like EPUB, it’s a zipped up file that contains an archive of the materials that make up the book. Inside, you find an Open Packaging folder and a META-INF Open Container Format folder, with its container.xml file. Unlike EPUB with its application/xhtml+xml mimetype, .ibooks uses application/x-ibooks+zip.
There are numerous other small differences. For those I defer to Jim Dovey, who tweeted expertly on the subject this morning.
If you re-name .ibooks files to .epub, they are just close enough to EPUB that you can read them into Adobe Digital Editions and Calibre. From Calibre, you can then export to EPUB although my tests show that you lose many of the fine details specific to Apple’s extensions. It’s so easy, however, to export directly to an iPad running iBooks 2, that you may not need to use this approach to recover EPUB files.