Falcon Code for Firebirders

Jim Starkey wrote on the Firebird-Architect mailing list:

Somethings in Falcon will look remarkably familiar to Vulcan hackers,
particularly MemMgr, JString, and SyncObject. I think a number of small
changes were made after the Vulcan code was split off, so somebody might
want to take a close look.

The class Bitmap is worth looking at. Like many things in Falcon, it
uses a fixed width, variable depth tree that meshes well with the memory
pool manager to avoid fragmentation. It is both faster and more memory
efficient that the Firebird analog.

The SQL parser is a very simple hard coded recursive descent parser. It
is used only for meta-data access in Falcon, however. User queries go
through the MySQL yacc/bison parser. They, too, wish they have a hard
coded recursive descent parser. Maybe Firebird should lend MySQL Arno.

People look far into the future for SuperServer might take a look at the
SerialLog. It’s a unified do/undo/redo log used both the push non-time
critical processing post-commit and for recovery. It works very well
for each.

Data flow in Falcon works like this:

1. Update statements create new record versions in memory. Index
updates go into per-transaction/per-index DeferredIndexes (in
memory fixed width, variable depth btrees).
2. Immediately pre-commit, updated records and deferred index nodes
are squirted into the serial log.
3. When everything else is done, a commit records is written into the
serial log and the log is flushed (group commits happen here)
4. Post-commit, a gopher thread move records and index nodes into
data page and index page images in the cache.
5. Once every 30 seconds or so, the page cache is flushed to disk,
the database file fsync-ed, a checkpoint record written to the
serial log, and the serial log flushed.

Deferred indexes are scanned during index scans using the same
transaction rules as record visibility. Deferred indexe hang around in
memory until the gopher thread is finished handling the transaction.

Falcon doesn’t contain the Netfrastructure JVM, though the hooks are all
there in conditional compilation. Netfrastructure still exists, but
non-MySQL code exists in a separate non-public tree. The Falcon code,
however, is common to both.

The EncodedRecordFormat should be of some interest to Firebird. It’s
denser than run length encoded records and probably faster to process.
There are some tricks, all of which are visible.

Somebody should look at the implementation of filtersets. Filtersets
are absolutely fabulous. The first guys to release a mainstream (i.e.
non-Netfrastructure) implementation will lead the industry…

Architecturally, Falcon is a clean sheet of paper relative to Interbase
/ Firebird. Index scans are highly similar, but that’s about that.
Much of the difference comes from Falcon designed as a single process
engine rather than the cluster-friendly Interbase model. Once you give
up the idea that the database must be sharable at the file / lock level,
all of the design considerations change.

I’d be delighted to discuss at length why I did one thing in Interbase,
another in Vulcan, and third in Falcon.

Oh, Falcon beats both InnoDB and MyIsam for retrieval operations. The
jury is still out on update operations (a couple of important
performance features aren’t in the Falcon alpha). I probably shouldn’t
talk about that, yet.

I’m told the public Bitkeeper tree will lag the internal tree by only an
hour or so, so if you consider database development to be a spectator
sport, feel tree to watch.