Add hybrid mnesia_rocksdb backup and restore

Provide mnesia_rocksdb:backup/1,2 and restore/1,2 that pair a Mnesia
checkpoint (schema and non-RocksDB tables) with RocksDB backup engines
for open admin and standalone databases.

Restore stages RocksDB into a sibling directory and redirects mountpoints
while Mnesia fallback purges the real directory, then relocates the data
and optionally restarts. Raise the minimum OTP version to 27.
This commit is contained in:
Ulf Wiger
2026-08-12 16:49:13 +02:00
parent 9ffb991c2c
commit 32384f76fe
8 changed files with 1262 additions and 26 deletions
+60 -1
View File
@@ -43,7 +43,7 @@ is provided.
* sext (included as dependency)
* Erlang/OTP 21.0 or newer (https://github.com/erlang/otp)
* Erlang/OTP 27.0 or newer (https://github.com/erlang/otp)
@@ -115,6 +115,65 @@ source code in `mnesia_rocksdb_tuning.erl` and `mnesia_rocksdb_params.erl`.
Contributions are welcome.
#### <a name="Backup">Backup</a> ####
`mnesia_rocksdb:backup(Dir)` creates a **local hybrid backup** of Mnesia
metadata/tables and RocksDB data. Unlike `mnesia:backup/1`, RocksDB-backed
tables are not dumped record-by-record through Mnesia; they are captured with
the [RocksDB backup engine](https://rocksdb.hexdocs.pm/1.9.0/how_to_backup_rocksdb.html).
Layout of `Dir` after a successful backup:
```
Dir/
MANIFEST % restore metadata (Erlang term)
mnesia.BUP % schema + non-RocksDB tables
rocksdb/<alias>/admin/ % backup engine dir for each alias admin DB
rocksdb/standalone/... % one dir per open standalone RocksDB table
```
Algorithm (local node only):
1. Activate a Mnesia checkpoint on **schema and non-RocksDB tables only**
(so large RocksDB tables do not get expensive checkpoint retainers).
2. For each open RocksDB instance (admin DB per alias, and standalone tables),
run `rocksdb:create_new_backup/2`.
3. Dump the checkpoint with `mnesia:backup_checkpoint/3`, excluding RocksDB
table data. Schema still describes RocksDB tables.
4. Write `MANIFEST`.
Options map (second argument to `backup/2`):
* `{verify, true}` — call `rocksdb:verify_backup/2` after each RocksDB backup
* `{mnesia_backup_module, Mod}` — override the Mnesia backup media module
#### Restore
`mnesia_rocksdb:restore(Dir)` restores a hybrid backup offline (local node).
Mnesia applies fallbacks by **purging** the Mnesia directory. RocksDB data is
therefore restored first into a **sibling staging directory** and mountpoints
are redirected until the fallback has finished:
1. Stop Mnesia (closes RocksDB handles).
2. Restore each RocksDB DB into
`"<basename(mnesia_dir)>.rocksdb_restore/"` (sibling of the Mnesia dir).
3. Set application env `mnesia_rocksdb.restore_mountpoint_root` so
`data_mountpoint/1` opens DBs from staging.
4. Install `Dir/mnesia.BUP` and start Mnesia (fallback purges the real Mnesia
dir; RocksDB tables load from staging with full data). During this window,
`delete_table` does not destroy staged RocksDB files.
5. Stop Mnesia, clear the redirect, move staged `*.extrdb` directories into the
real Mnesia directory.
6. Optionally start Mnesia (`#{start => true}`).
Common options for `restore/2`:
* `{stop, true|false}` — stop Mnesia before restore (default `true`)
* `{start, true|false}` — start Mnesia after finalize (default `false`)
* `{mnesia_dir, Dir}` — override target Mnesia directory
* `{skip_rocksdb, true}` / `{skip_mnesia, true}` — partial restore
#### <a name="Caveats">Caveats</a> ####
Avoid placing `bag` tables in RocksDB. Although they work, each write