Add hybrid mnesia_rocksdb backup and restore
Provide mnesia_rocksdb:backup/1,2 and restore/1,2 that pair a Mnesia checkpoint (schema and non-RocksDB tables) with RocksDB backup engines for open admin and standalone databases. Restore stages RocksDB into a sibling directory and redirects mountpoints while Mnesia fallback purges the real directory, then relocates the data and optionally restarts. Raise the minimum OTP version to 27.
This commit is contained in:
@@ -43,7 +43,7 @@ is provided.
|
||||
|
||||
* sext (included as dependency)
|
||||
|
||||
* Erlang/OTP 21.0 or newer (https://github.com/erlang/otp)
|
||||
* Erlang/OTP 27.0 or newer (https://github.com/erlang/otp)
|
||||
|
||||
|
||||
|
||||
@@ -115,6 +115,65 @@ source code in `mnesia_rocksdb_tuning.erl` and `mnesia_rocksdb_params.erl`.
|
||||
Contributions are welcome.
|
||||
|
||||
|
||||
#### <a name="Backup">Backup</a> ####
|
||||
|
||||
`mnesia_rocksdb:backup(Dir)` creates a **local hybrid backup** of Mnesia
|
||||
metadata/tables and RocksDB data. Unlike `mnesia:backup/1`, RocksDB-backed
|
||||
tables are not dumped record-by-record through Mnesia; they are captured with
|
||||
the [RocksDB backup engine](https://rocksdb.hexdocs.pm/1.9.0/how_to_backup_rocksdb.html).
|
||||
|
||||
Layout of `Dir` after a successful backup:
|
||||
|
||||
```
|
||||
Dir/
|
||||
MANIFEST % restore metadata (Erlang term)
|
||||
mnesia.BUP % schema + non-RocksDB tables
|
||||
rocksdb/<alias>/admin/ % backup engine dir for each alias admin DB
|
||||
rocksdb/standalone/... % one dir per open standalone RocksDB table
|
||||
```
|
||||
|
||||
Algorithm (local node only):
|
||||
|
||||
1. Activate a Mnesia checkpoint on **schema and non-RocksDB tables only**
|
||||
(so large RocksDB tables do not get expensive checkpoint retainers).
|
||||
2. For each open RocksDB instance (admin DB per alias, and standalone tables),
|
||||
run `rocksdb:create_new_backup/2`.
|
||||
3. Dump the checkpoint with `mnesia:backup_checkpoint/3`, excluding RocksDB
|
||||
table data. Schema still describes RocksDB tables.
|
||||
4. Write `MANIFEST`.
|
||||
|
||||
Options map (second argument to `backup/2`):
|
||||
|
||||
* `{verify, true}` — call `rocksdb:verify_backup/2` after each RocksDB backup
|
||||
* `{mnesia_backup_module, Mod}` — override the Mnesia backup media module
|
||||
|
||||
#### Restore
|
||||
|
||||
`mnesia_rocksdb:restore(Dir)` restores a hybrid backup offline (local node).
|
||||
|
||||
Mnesia applies fallbacks by **purging** the Mnesia directory. RocksDB data is
|
||||
therefore restored first into a **sibling staging directory** and mountpoints
|
||||
are redirected until the fallback has finished:
|
||||
|
||||
1. Stop Mnesia (closes RocksDB handles).
|
||||
2. Restore each RocksDB DB into
|
||||
`"<basename(mnesia_dir)>.rocksdb_restore/"` (sibling of the Mnesia dir).
|
||||
3. Set application env `mnesia_rocksdb.restore_mountpoint_root` so
|
||||
`data_mountpoint/1` opens DBs from staging.
|
||||
4. Install `Dir/mnesia.BUP` and start Mnesia (fallback purges the real Mnesia
|
||||
dir; RocksDB tables load from staging with full data). During this window,
|
||||
`delete_table` does not destroy staged RocksDB files.
|
||||
5. Stop Mnesia, clear the redirect, move staged `*.extrdb` directories into the
|
||||
real Mnesia directory.
|
||||
6. Optionally start Mnesia (`#{start => true}`).
|
||||
|
||||
Common options for `restore/2`:
|
||||
|
||||
* `{stop, true|false}` — stop Mnesia before restore (default `true`)
|
||||
* `{start, true|false}` — start Mnesia after finalize (default `false`)
|
||||
* `{mnesia_dir, Dir}` — override target Mnesia directory
|
||||
* `{skip_rocksdb, true}` / `{skip_mnesia, true}` — partial restore
|
||||
|
||||
#### <a name="Caveats">Caveats</a> ####
|
||||
|
||||
Avoid placing `bag` tables in RocksDB. Although they work, each write
|
||||
|
||||
Reference in New Issue
Block a user