Replication

RDM uses files in a specialized format to store data. These files are referred to as pack files in this document. Changes to data in the pack files are made using a technique called “copy-on-write”. That means that these pack files are never updated in the middle but instead, changes are always appended to the end of the last pack file. This design has some advantages but also implies certain constraints.

The main constraints are:

• Everything has to be organized as some sort of a tree which rules out some data structures
• For changes to this tree where only an internal node needs to change, all the nodes up to the root of this tree have to be updated as well.
• No direct lookup of data. In the general case, data has to be looked up by walking the tree from the root.

Some of the advantages are:

• Great parallelism where many operations can be performed in parallel
• Very efficient write since we only append to the end of a pack file
• Data can be snapshotted with literally no extra cost with respect to memory usage and performance
• No need for a transaction log
• Open is very fast as no recovery is needed. The last pack file does need to be scanned to find the last committed transaction. This scan is very efficient as it utilizes periodic recovery markers and only reads a limited amount of meta data in the pack.
• Clients are able to cache content for a table even when changes have been made to that table by other clients
• We can relax ACID properties for speed by setting durability to consistent. When durability is set to consistent, pack files will not always be synced to disk, thus improving the performance. This can cause the loss of some of the last transactions. However, as long as there are no catastrophic system crashes (e.g. the OS goes down), no data will be lost. In any case, we are always consistent; data may be lost but there are never corruptions
• One way replication can be done by simply copying pack files from the source to the target as they are created on the source.

The last bullet point is not as simple as just copying the pack files from the source to the target. Some bookkeeping has to be done for the data to be made available on the target and pack files have to be deleted when they are no longer needed. Nevertheless, this method of replication can be made efficient and simple to use.