Monday, June 14, 2010

ZODB Book Project

The Z Object DataBase (ZODB) has been in production for over a decade. It is very useful outside of the Zope application server. The ZODB offers low friction reliable object persistence. ZODB is not SQL (it is hierarchical and does not require use of an ORM layer). Unlike many of its NoSQL cousins, the ZODB supports reliable transactions (ACID). Arguably the ZODB is the most widely deployed object database in the world. ZODB ships with every Plone, Zenoss, Grok, and Zope deployment.

Unfortunately the lack of documentation for the ZODB does not reflect its maturity, robustness or feature set. The documentation is scattered. Most of the information is outdated. Many best practices have been learned over the past 10 years. There has not yet been a concerted effort to generate and maintain ZODB documentation. This initiative is set on fixing the first step, generating documentation. The following topical overview is a draft of what material may be covered:

  • Introduction to ZODB.
    What it is. Maybe some bits about the NoSQL craze, how the ZODB has been doing that for more than 10 years. When is the ZODB a good fit for your app?

  • Your first ZODB application.
    A short chapter dealing with installation and running the first app.

  • A more complex application.
    A bit more involved explanation of how it works and a more useful sample application.

  • Basic indexing and searching.
    The Catalog and indexes. I might use the repoze stuff, which is somewhat simpler.

  • A more in-depth look at the ZODB internals.
    A little more information about how the ZODB works. At least enough stuff to understand the later chapters about storages and debugging.

  • ZODB Storages.
    Details about the FS storage and discussion of RelStorage and maybe DirectoryStorage.

  • Other indexing and searching strategies.
    Other catalog implementations, third party indexes and using external indexing solutions, like Solr.

  • Advanced ZODB.
    Evolving objects and other problems...not really sure about this one yet.

  • The debugging FAQ: frequent problems and suggested solutions.
    General debugging strategies and then a FAQ with common problems.

  • Scaling.
    The ZODB cache, ZEO and replication services.

  • Maintenance.
    Packing, backups, etc.
This project is set on collecting money from the community to fund a seasoned ZODB developer and published writer, Carlos de la Guardia to complete a Creative Commons licensed ZODB book. The effort is significant and we are looking to raise a relatively large amount of money. A professionally written and edited book will be a huge asset for the ZODB community. It will enable developers to evaluate the ZODB quickly and understand the modern patterns of software design applied by seasoned developers using ZODB in production.

Update: Jim Fulton has volunteered to technically review the book!

Individuals and organizations can contribute and will be recognized for their contribution in the following ways:
  • A printed version available on Amazon will have their name and/or logo's in the book.

  • The ZODB website will provide links to their website.

  • Their contribution will be documented in the main ZODB documentation and will permanently reside in the documentation.
The following levels of support are available for sponsoring this project. The target goal is 10000 USD.
  • $20-149, Individual Name
  • $150-399, Company Name
  • $400-999, Company Name + Logo + URL; link on ZODB.ORG
  • $1000-2499, Company Name + .5 page "spread"; link on ZODB.ORG
  • $2500-, Company Name + full page "spread"; link on ZODB.ORG
Some other aspects of this project:
  • Provide a modern unique logo/design for the ZODB project

  • We hope to flush out and clarify the meaning of all public interfaces and methods in ZODB source docstrings.

  • A Creative Commons licensed PDF will be available.

  • Put the new book online at zodb.org
  • Possibly provide a hard copy published on Amazon.
The ZODB community gratefully appreciates your contribution. We hope to raise enough money to provide the advance to Carlos de la Guardia by end of July 2010. Once the advance is in place Carlos will start work. We will continue taking contributions. Depending on the amount of money we receive in total will dictate the depth and breadth of documentation. Several volunteers/participates have already signed up. We will be providing updates in the near future.

The next steps:
  • Register the paypal account to take contributions.

  • Send email to various mailing lists and companies using ZODB soliciting donations

  • Get Carlos writing!

8 comments:

  1. Some topic ideas:
    - Standalone ZODB (using it without Zope 3/Zope 2, just barebone ZODB)
    - ZEORaid

    ReplyDelete
  2. Not a topic, but one more:
    Conflict resolution and what's behind that.

    ReplyDelete
  3. I suggested a few topics in https://mail.zope.org/pipermail/zodb-dev/2010-May/013311.html but they are already covered.

    I only wish the book would focus exclusively on zodb since I do not use zope or plone and these systems are sufficiently documented.

    We are using ZODB with Django in our company. We are the developers of the django-zodb (http://triveos.github.com/django-zodb/) and make a small donation to the project (our company is small and we do not have revenue yet).

    ReplyDelete
  4. A topic that I'd like to see in the book is techniques to debug errors that occur on transaction commit: this kind of errors is a PITA to debug because when the exception triggers the bad code has been already executed and is thus hard to find. For instance if I attach a function object to a persistent object all goes fine until the transaction is committed, and a "TypeError: can't pickle function objects" exception is raised.

    ReplyDelete
  5. You know what would be nice is if you guys did something like optilude did for his laptop so we can see how close to goal you are. I think he used ChipIn http://www.martinaspeli.net/

    ReplyDelete
  6. Thanks for the feedback.

    Regarding scope:
    No talk of Zope, Plone, or any web application will be the focus of the ZODB book. At least that is my understanding. This will be focused purely on ZODB, persistent, transaction libraries.

    Debugging/Tips n Tricks:
    I think this is absolutely critical section. How to debug unexpected mutation of persistent objects. How to debug transaction failures. There are many others.

    Regarding donations:
    We are trying to reuse the infrastructure we have as a company to reduce overhead for taking donations.

    Solicitation to all experienced developers:
    If you are a experienced developer drop me an email. I will add you to this blog and you can write about your ZODB experiences. Post recipes. And help contribute. Hopefully your post will be inspiring or pulled into the book as a whole (and credit you for the work ;).

    ReplyDelete
  7. Please add a title to this post, feed validators are complaining ...

    ReplyDelete
  8. If the objective is to raise awareness and establish legitimacy among the minds of the NoSQL friendly who are not aware of ZODB or Python object databases, perhaps a brief mention of "other implementations" of the ZODB concept might be worth noting.

    There is at least one other - Durus, a ZODB work-a-like. Durus is a Python object database modelled after ZODB but minus some of its complexity. IMO that Durus was even written is a testament to ZODB.

    http://www.mems-exchange.org/software/durus/

    ReplyDelete