Documentation

Backups

FeatureBase

FeatureBase supports backup and restore options through the featurebase backup and featurebase restore commands. These commands transfer data between a FeatureBase cluster and a directory:

featurebase backup --host featurebase:10101 -o /path/to/backup/
featurebase restore --host featurebase:10101 -s /path/to/backup/

Both commands accept the --host option, which specifies a FeatureBase node by a hostname/IP and a port joined by a colon. Despite only requiring the address of one node, these commands act on the entire cluster (and thus require network access to all nodes).

The featurebase backup command accepts an output option (-o or --output) to specify the destination directory for the backup. This is not an incremental backup, so this directory is expected to be empty. If the directory does not exist, it will be created.

The featurebase restore command similarly accepts a source option (-s or --source) to specify the source directory (from a previous featurebase backup command). It should only be executed against an empty cluster. featurebase backup and featurebase restore should not be executed while a Delete call is in process.

Backups with Authentication

When authentication is enabled, TLS is also enabled. The host must use https for it’s scheme and a valid JWT must be provided using the auth-token flag. This token may be obtained by following these instructions. Only a user with admin access to FeatureBase will be authorized to perform a backup/restore.

featurebase backup --host https://featurebase:10101 -o /path/to/backup/ --auth-token <token>
featurebase restore --host https://featurebase:10101 -s /path/to/backup/ --auth-token <token>

⚠ WARNING: Attempting to restore to a live cluster populated with data may result in data loss and/or system instability.

NOTE: This process is only compatible with the RBF storage backend.

What is included in a backup?

A featurebase backup includes:

  • schema
  • key translation data
  • indexes
  • ID allocation state

It does not include node configurations or the state of other FeatureBase components.

When using an external lookup database, this tool will not back up the external database. The external lookup database must be backed up seperately.

The backup data is independent of the size or replication factor of a cluster. As such, it is safe to restore a backup to a differently sized cluster. This can be used to safely resize a cluster.

Restoring in Production

The ideal process for restoring to a production installation is:

  1. Stop ingest to the old cluster (if it still exists)
  2. Start a new cluster
  3. Use featurebase restore to apply the data to the cluster
  4. Verify that the cluster behaves as expected (run some queries and check for expected results)
  5. Redirect query traffic to the new cluster
  6. Start ingest to the new cluster
  7. Back up the old cluster again (if it still exists)
  8. Tear down the old cluster (if it still exists)

Consistency

The generated backup is not an instantaneous snapshot, as FeatureBase does not support that consistency level. Seperate components are backed up in a specific order, such that:

  1. If a record is created before the backup starts and is not modified, it will end up in the backup.
  2. If a record is created during the backup, it may or may not end up in the backup.
  3. If a record is modified during the backup, the backup will include either the original or modified state.
  4. If a record is deleted during the backup, it may or may not end up in the backup.
  5. All necessary key translation data will end up in the backup.
  6. IDs committed by the auto ID feature before the start of the backup will not be reused, but other allocated IDs may be reused.

TLS Configuration

In order to use TLS, the following options are required (for backup & restore commands):

  • --tls.ca-certificate: the CA certitificate to use for verifying the server certificate (defaults to CA certs installed to system)
  • --tls.certificate: the TLS client certificate to use for mTLS authentication
  • --tls.key: the client key corresponding to the client certificate (for mTLS)

For example:

featurebase backup --host featurebase:10101 -o /path/to/backup/ --tls.ca-certificate ca.crt --tls.certificate client.crt --tls.key client.key
featurebase restore --host featurebase:10101 -s /path/to/backup/ --tls.ca-certificate ca.crt --tls.certificate client.crt --tls.key client.key

Backup Concurrency

By default, backup and restore operations will transfer one file at a time. Using the --concurrency option, many files can be transferred simultaneously for higher throughput:

featurebase backup --concurrency 2 --host featurebase:10101 -o /path/to/backup/
featurebase restore --concurrency 8 --host featurebase:10101 -s /path/to/backup/

When backing up a live production cluster, this concurrency option should be set relatively low (or left at the default of 1). Setting this value too high may exhaust resources of the system and negatively impact applications using the cluster during the backup. The optimal tradeoff varies depending on cluster size, CPU speed, disk speed, and system load. In most scenarios, one per node is a reasonable conservative selection.

When restoring to a fresh cluster, this constraint is not applicable. The concurrency can be set fairly high, with several concurrent transfers per destination node. The exact number of optimal restore operations varies depending on network/disk speeds, and should be determined experimentally when doing a test-restore.

Storage Synchronization

By default, featurebase backup will wait for all backup files to be committed to persistent storage before terminating. This ensures that the backup will remain intact in the event that the system is forcefully shut down or loses power. This can be disabled in exchange for higher backup speed with the --no-sync option when the backup will be written somewhere else in a later step:

featurebase backup --host featurebase:10101 -o /path/to/backup/ --no-sync # back up FeatureBase without syncing the files to disk
tar -cvf backup.tar.xz /path/to/backup # put the backup in a compressed tar file
sync backup.tar.xz . # sync the tar file to disk and then sync the directory containing it to disk (both are necessary)
rm -r /path/to/backup

Testing a Backup

The most reliable way to test a backup is to attempt to restore it. This can be done by starting a temporary cluster, restoring to that cluster, and executing test operations (e.g. common queries) against it. This test can be done with a single local node, provided that the machine has sufficient memory and disk space.

When operating an on-premises FeatureBase cluster, we recommend that you restore backups to a realistically-configured cluster periodically. This ensures that the processes and tools for restoring in the event of a disaster are available when needed.

Best Practices for Production Backups and Restores

  • Back up frequently - recovering from an old backup means more data loss than a recent backup
  • Store backups on multiple systems (in case one fails)
  • Store at least one copy of the data off-site (in another datacenter or cloud)
  • Keep at least one tested backup at any given time
  • Backups should have at least the same level of security as the system being backed up - use appropriate access control and encrypt them (place them on encrypted storage and/or encrypt the files)
  • Prepare/maintain/test any necessary scripts/tools/plans for recovering from a backup such that they are available when needed

Comparison to Molecula v3.x Backup System

Molecula v3.x’s backup system worked by completely freezing writes, rsyncing the data, and resuming writes. The backup data could only be safely restored into an identical cluster. This option is not supported in Molecula v4.x.

The new process runs concurrently with writes, and is fully integrated into FeatureBase. It works by acquiring a read lock on each individual file while copying. Like the v3.x process it directly copies each database file in its original format for efficiency reasons. It is accessed as a single shell command so that a complicated multi-step process is no longer needed. The resulting backup data is independent of the cluster it was created from, and can be restored into a differently-configured cluster safely.

Something missing or incorrect?
Help improve this article or join us on Discord!

© 2022 Molecula Corp. (DBA FeatureBase). All rights reserved.