Comments on: 5 Years of Distributed Proofreaders http://www.teleread.com/public-domain/5-years-of-distributed-proofreaders/ News & views on e-books, libraries, publishing and related topics Tue, 14 Feb 2012 21:55:20 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Garson Poole http://www.teleread.com/public-domain/5-years-of-distributed-proofreaders/comment-page-1/#comment-19872 Garson Poole Tue, 25 Oct 2005 10:22:13 +0000 http://www.teleread.org/blog/?p=3747#comment-19872 Branko, Thanks for your volunteer work and thanks for the link to the discussion of the DjVu format and XML. Several years ago I visited the website of the "Million Book Project" when it was active at Carnegie Mellon. Unfortunately, the project seemed to be stalled and inactive when I visited. It is good to see that Brewster Kahle and the Internet Archive are moving forward together with Canadian libraries. I agree that retaining typographical information when proofreading and converting text into a markup language would be highly desirable. The HTML versions from Distributed Proofreaders that preserve some typography are great. Preserving precise information about publishers, edition numbers, print runs etc is also important as David Rothman has emphasized on this blog. The digitalization of millions of volumes is such a massive task that I think human proofreaders will only be able to perform markup on a subset of the volumes. For many books only limited formats will be available, e.g., scans, OCR data, and DjVu. I hope a widely deployed free reader with an openly specified format will be available for these books. Branko, Thanks for your volunteer work and thanks for the link to the discussion of the DjVu format and XML. Several years ago I visited the website of the “Million Book Project” when it was active at Carnegie Mellon. Unfortunately, the project seemed to be stalled and inactive when I visited. It is good to see that Brewster Kahle and the Internet Archive are moving forward together with Canadian libraries.

I agree that retaining typographical information when proofreading and converting text into a markup language would be highly desirable. The HTML versions from Distributed Proofreaders that preserve some typography are great. Preserving precise information about publishers, edition numbers, print runs etc is also important as David Rothman has emphasized on this blog.

The digitalization of millions of volumes is such a massive task that I think human proofreaders will only be able to perform markup on a subset of the volumes. For many books only limited formats will be available, e.g., scans, OCR data, and DjVu. I hope a widely deployed free reader with an openly specified format will be available for these books.

]]> By: Branko Collin http://www.teleread.com/public-domain/5-years-of-distributed-proofreaders/comment-page-1/#comment-18731 Branko Collin Mon, 24 Oct 2005 08:57:33 +0000 http://www.teleread.org/blog/?p=3747#comment-18731 Garson, having quality and completeness levels, and including scans, would certainly seem to be useful, especially if it will enable you to lift a document to a next level. This should allow you to release early and often, as the FOSS people like to say. In a sense (a very vague and loose sense) this is already happening; many projects release tens of thousands of documents in scan form with uncorrected OCR, lots of scientific projects release a few heavily marked-up documents, and Distributed Proofreaders seems to be somewhere in between, with typographical characteristics often preserved in HTML versions. (HTML versions are not obligatory, but by my last count approximately 75% of the ebooks coming out of DP are accompagnied by an HTML version.) As for pointing to locations on a scanned page, <a href="http://www.archive.org/iathreads/post-view.php?id=25169" rel="nofollow">I know it is possible</a>, but it needs to be implemented. If it needs to be implemented at DP, it needs to be implemented in the entire system, which is going to be a lot of work. Personally, I'd prefer to work first on moving to a generic XML/SGML format for DP that retains most typographical information. Garson, having quality and completeness levels, and including scans, would certainly seem to be useful, especially if it will enable you to lift a document to a next level. This should allow you to release early and often, as the FOSS people like to say.

In a sense (a very vague and loose sense) this is already happening; many projects release tens of thousands of documents in scan form with uncorrected OCR, lots of scientific projects release a few heavily marked-up documents, and Distributed Proofreaders seems to be somewhere in between, with typographical characteristics often preserved in HTML versions. (HTML versions are not obligatory, but by my last count approximately 75% of the ebooks coming out of DP are accompagnied by an HTML version.)

As for pointing to locations on a scanned page, I know it is possible, but it needs to be implemented. If it needs to be implemented at DP, it needs to be implemented in the entire system, which is going to be a lot of work. Personally, I’d prefer to work first on moving to a generic XML/SGML format for DP that retains most typographical information.

]]> By: Garson Poole http://www.teleread.com/public-domain/5-years-of-distributed-proofreaders/comment-page-1/#comment-18541 Garson Poole Mon, 24 Oct 2005 04:57:07 +0000 http://www.teleread.org/blog/?p=3747#comment-18541 The work being performed by the volunteers at "Distributed Proofreaders" is magnificent. Yet, I think that it is time to consider a new supplementary strategy for making public domain books available. Instead of waiting for proof-reading each book why not release books in a preliminary form as a collection of scans together with the best current results from an automated optical character recognition analysis. This approach will allow for the release of myriad books. Each book should only require a few megabytes of storage using compression methods specialized for text. This is acceptable now because hard-disk and flash memories are capacious and relatively inexpensive. Also, storage costs continue to decline. I suspect that Google and Yahoo may take this approach because human proof-reading is slow and expensive. Will the “OpenReader” format support a collection of scans together with OCR text that can be used for searching? The OCR text would be supplemented with “index” type information that would allow a search result to point to a specific location on a scanned page. The work being performed by the volunteers at “Distributed Proofreaders” is magnificent. Yet, I think that it is time to consider a new supplementary strategy for making public domain books available. Instead of waiting for proof-reading each book why not release books in a preliminary form as a collection of scans together with the best current results from an automated optical character recognition analysis. This approach will allow for the release of myriad books. Each book should only require a few megabytes of storage using compression methods specialized for text. This is acceptable now because hard-disk and flash memories are capacious and relatively inexpensive. Also, storage costs continue to decline. I suspect that Google and Yahoo may take this approach because human proof-reading is slow and expensive.

Will the “OpenReader” format support a collection of scans together with OCR text that can be used for searching? The OCR text would be supplemented with “index” type information that would allow a search result to point to a specific location on a scanned page.

]]> By: David Rothman http://www.teleread.com/public-domain/5-years-of-distributed-proofreaders/comment-page-1/#comment-17876 David Rothman Sun, 23 Oct 2005 13:01:32 +0000 http://www.teleread.org/blog/?p=3747#comment-17876 That's a valuable mini-history, Branko--thanks very much for your lastest post. I hope readers will pick up on the fact that DP's contribution isn't <em>just</em> in quantity of texts, but also in quality. - David That’s a valuable mini-history, Branko–thanks very much for your lastest post. I hope readers will pick up on the fact that DP’s contribution isn’t just in quantity of texts, but also in quality. – David

]]> By: Jon Noring http://www.teleread.com/public-domain/5-years-of-distributed-proofreaders/comment-page-1/#comment-17423 Jon Noring Sun, 23 Oct 2005 05:31:06 +0000 http://www.teleread.org/blog/?p=3747#comment-17423 Congratulations to Distributed Proofreaders! Congratulations to Distributed Proofreaders!

]]>