mnesia_rocksdb/doc/online-encoding-migration.md

Online encoding migration (design note)

Status: design
Branch: uw-migration
Scope (MWP): change key/value encoding of an existing set table by copying into a new RocksDB column family while the table remains online.

Changing Mnesia type (set ↔ ordered_set) is out of scope for the first cut. For mrdb users, what matters for range seeks and key order is the rocksdb key encoding, not the Mnesia metadata type. Aligning Mnesia type via a custom schema transaction is possible later and needs separate design.

---

1. Motivation

Some tables were created with encodings that are awkward for production use. Example: integer primary keys with {term, {object, term}} encoding. The Erlang external term format preserves sort order for positive integers only up to 32-bit max; beyond that keys use SMALL_BIG_EXT and byte order ≠ numeric order. RocksDB iterators and seeks then mis-order heights.

Historically apps worked around this by creating a second table (gmmp_tallies → gmmp_tallies2) and migrating in application code. That is painful and easy to get wrong.

This feature moves encoding migration into the backend: dual-write to a new column family (CF), copy old → new, then flip the live db_ref and drop the old CF.

---

2. Goals and non-goals

Goals (MWP)

• Online migration of encoding for set tables (semantics set).
• Table remains readable and writable during migration.
• Safe under concurrent inserts/updates/deletes.
• Resume after process/node restart (without a durable RocksDB snapshot).
• Indexes (ordered) continue to work: updated on live writes as today; index CFs are not dual-copied for encoding-only main-table migrate.
• Explicit start/complete as admin/schema-level operations (not silent).

Non-goals (MWP)

• Bag tables (supported for completeness; poor fit for rocksdb dual-write).
• Changing Mnesia type (set / ordered_set / bag) in this pass.
• Rebuilding or re-encoding index column families.
• Multi-node coordinated migration (single backend instance first).
• Retaining a RocksDB snapshot across process restart.
• Record shape / arity / attribute renames (use mnesia:transform_table or app logic).

---

3. Existing building blocks

Piece	Role today
persistent_term meta map	Name → db_ref() via put_pt / get_ref / ensure_ref
CF create/drop	create_column_family / drop_column_family
Offline standalone→CF migrate	migrate_standalone/2: chunk select, insert new, delete old, put_pt
Snapshots	mrdb:snapshot/1, release_snapshot/1 (DB-instance scoped)
Hot write path	insert_ / delete_ encode with ref’s encoding

The new feature is an online, same-DB CF migrate with dual-write, not a reimplementation of standalone migration.

---

4. Design overview

4.0 Column-family naming (versions)

RocksDB CF names must be unique within a DB. The existing map is:

Logical resource	Physical CF name
data table Tab	"{d, Tab}" (gen 0)
index	"{i, Tab, I}"
retainer	"{r, Tab, R}"
admin	"{a, Alias}"

That mapping has no version. Encoding migration needs the old and new data CFs to coexist until cutover, so data CFs are versioned:

Generation	Physical CF name
0 (legacy / first create)	"{d, Tab}"
N ≥ 1	"{d, Tab, N}"

• tab_to_cf_name(Tab) still produces gen 0 for brand-new tables.
• Migration target = data_cf_name(Tab, LiveGen + 1).
• Live gen is stored on the db_ref as cf_gen / cf_name and durably as admin info cf_gen for the table.
• cf_name_to_tab/2 maps any data gen back to logical Tab.
• On open, if several data gens exist for Tab, prefer the highest gen for which we have a handle; refine with durable cf_gen when present (interrupted migration may leave an incomplete higher gen — recovery should prefer the recorded live gen once that path is fully wired).

Cutover does not recreate a “canonical” CF name. The versioned target CF is the new live CF; the previous generation is dropped. No second full copy.

                    ┌─────────────────────────────────────┐
                    │  Old CF gen G (reads; encoding E_old)│
                    │  name: {d,Tab} or {d,Tab,G}         │
                    └──────────────┬──────────────────────┘
                                   │ dual-write
                                   │ (live updates)
                                   ▼
                    ┌─────────────────────────────────────┐
                    │  New CF gen G+1 (encoding E_new)    │
                    │  name: {d,Tab,G+1}                  │
                    │  + live dual-writes                 │
                    │  + migrator copy (skip if exists)   │
                    └─────────────────────────────────────┘
                                   │
                    ┌──────────────┴──────────────────────┐
                    │  Admin info                         │
                    │  encoding_migration meta, cf_gen    │
                    └─────────────────────────────────────┘

4.1 Phases

Phase	Reads	Writes	Notes
idle	current CF	current CF	Normal operation
dual_write / copying	old CF only	old and new (re-encode)	Migrator walks old → new
copy_done	old	dual-write still on	Optional verify
cutover	new	new only	PT flip; then drop old + scratch

4.2 Ref shape during migration

Live ref in persistent_term (conceptual):

OldRef#{
  migration => NewRef,          %% target CF ref (encoding E_new)
  migration_epoch => pos_integer(),
  migration_phase => copying | copy_done
}

• Reads / select / iterators: use the outer (old) ref only until cutover.
• Inserts / deletes / merges (set): apply to old, then if migration is set, re-encode and apply to NewRef.
• Indexes: update_index runs once, as today, using the live main ref (old CF) for read(R, Key). Index CF layout is independent of main encoding (index keys are {IxVal, Key} under the index ref’s encoding).

4.3 Migrator install rule

For each key K observed on the migration walk:

if not present in live old(K)  → skip   %% deleted after snapshot / walk
else if present in new(K)      → skip   %% dual-write already installed latest
else put new(live_value(K))             %% re-encode with NewRef

Rationale:

• Skip if new has key — dual-write wins; migrator must not clobber newer data.
• Live re-check of old — avoids delete resurrection when iterating a snapshot that still contains K after a dual-delete removed it from both CFs.
• Prefer live value over snapshot-only value when installing.

4.4 Snapshot (best-effort)

While the migrator process lives:

1. Snapshot = rocksdb:snapshot(DbRef) (same DB as old/new CFs).
2. Iterate old CF with {snapshot, Snapshot} read options from cursor.
3. Apply the install rule above (always re-check live old + new).

On migrator crash/restart: snapshot is gone. Resume from scratch cursor on live old CF with the same install rule. Correctness does not depend on the snapshot; the snapshot only reduces iterator churn.

4.5 Scratch metadata

Either a dedicated scratch CF or admin-info keys (admin CF is enough for MWP).

Minimum fields:

#{ table  => tabname()
 , phase  => dual_write | copying | copy_done | cutover
 , target => #{encoding := encoding()}   %% validated
 , cursor => '$first' | Key :: term()    %% last successfully processed logical key
 , epoch  => pos_integer()
 , started_at => millisecond()
 }

• Cursor is the logical key (Erlang term), not the encoded rocksdb key.
• Resume seek on old CF uses old encoding: encode_key(Cursor, OldRef).

4.6 Cutover

1. Ensure phase = copy_done (and optional verification).
2. Barrier: no in-flight activities that still hold a pre-migration ref map (see §5.2), or epoch re-resolution (see below).
3. Schema/admin transaction:
   • put_pt(Tab, NewRef) without migration (NewRef is gen G+1 CF)
   • persist live cf_gen = G+1 in admin info
   • drop old generation CF only
   • clear migration meta
4. Subsequent get_ref(Tab) returns new encoding and cf_gen = G+1.

No rename and no second bulk copy: the migration target CF is permanent.

4.7 Mnesia type deferred

For mrdb callers, ordered seeks and range logic depend on key encoding (e.g. sext vs term), not on mnesia:table_info(Tab, type).

Changing Mnesia type later may use a custom schema transaction so metadata matches reality; that must not be conflated with CF encoding migrate and needs extra care around table_info, load hooks, and any code that branches on type.

---

5. Concurrency and correctness

5.1 Dual-write before copy

Order of start:

1. Create empty new CF (target encoding).
2. Persist migration meta + install migration => NewRef on live ref (dual-write ON).
3. Only then start the copy walk.

If copy ran before dual-write, concurrent writes could land only on old and be missed or overwritten incorrectly.

5.2 Stale in-process db_ref maps

ensure_ref/1 short-circuits when given a map (activities/batches hold refs). After cutover, a process still holding the old map would write a CF that is about to be dropped.

MWP options (pick at least one):

1. Quiesce: refuse cutover while table activity is non-zero / use a short exclusive period.
2. Epoch: bump migration_epoch; ensure_ref(Map) re-fetches from PT if map epoch ≠ current.
3. Delay drop: keep old CF read-only until epoch drained (refcounting).

Recommended MWP: epoch on ref + re-resolve when mismatch, plus delay drop of old CF until a quiet period or explicit admin “finalize”.

5.3 Transactions and batches

Prefer one RocksDB WriteBatch (same db_ref, two cf_handles) so dual-write insert/delete is atomic across old and new under as_batch.

Index updates stay in the same batch as today (batch_if_index).

5.4 Indexes and encoding independence

Index CFs store {IxVal, MainKey} under the index encoding. Main encoding change does not require rewriting index keys. Live writes continue to maintain indexes against the logical object; after cutover, index_read still yields MainKey terms and reads the new main CF with the new encoding.

Do not dual-write index CFs in MWP encoding migration.

5.5 Crash recovery

On admin/backend restart:

1. Reload CF handles and PT from durable admin state.
2. If migration meta says copying / dual_write / copy_done:
   • Re-bind live ref with migration => NewRef.
   • Resume dual-write.
   • If not copy_done, restart migrator from cursor (live iterate).
3. Never drop old CF unless cutover fully committed.

---

6. API sketch (MWP)

%% Start (admin / schema-level). Validates encoding; rejects bag; rejects
%% unsupported type changes.
-spec migrate_encoding(alias(), tabname(), encoding()) -> ok | {error, term()}.
migrate_encoding(Alias, Tab, NewEncoding) -> ...

%% Optional: progress / status
-spec migration_status(tabname()) ->
    idle | #{phase := atom(), cursor := term(), target := map()}.

%% Complete cutover (if not automatic after copy_done + barrier)
-spec finalize_migration(alias(), tabname()) -> ok | {error, term()}.

Reporting can mirror migrate_standalone/3 (Rpt pid + progress messages).

---

7. Risks and open points

Risk	Mitigation
Delete resurrection under snapshot	Live re-check before install (§4.3)
Stale activity refs after cutover	Epoch re-resolve + delayed CF drop (§5.2)
Partial dual-write in batch	Multi-CF WriteBatch (§5.3)
PT lost while admin meta says migrating	Recovery rebuilds PT from admin (§5.5)
Huge tables	Chunked walk + cursor; progress reports
bag / type / index encoding	Explicitly out of MWP

Open (later):

• Automatic vs manual finalize after copy_done.
• Optional full verification (count / sample) before cutover.
• Mnesia type alignment schema transaction.
• Bag and index-encoding migration.

---

8. Relation to application workarounds

Application dual tables (*2) remain valid for record-shape changes. Backend encoding migration is aimed at cases like “same record, wrong key encoding” without app-visible table rename.

Example target: migrate gmmp_generations from {term,{object,term}} to a sext (or other order-preserving) key encoding so height seeks stay valid past 2³² − 1 without a generations2 table.

---

Implementation checklist

Phased so each phase is testable. Bags and Mnesia type change stay excluded.

Phase 0 — Spec freeze and hooks

• Freeze MWP scope: set + encoding only; document rejections for bag / type.
• Define durable migration meta schema (admin info keys vs scratch CF).
• Define public API: migrate_encoding/3, migration_status/1, finalize_migration/2 (names flexible).
• Define progress report messages (align with migrate_standalone rpt).
• List all write paths that must dual-write: insert, delete, delete_object, update_counter/merge (if encoding-compatible), clear_table policy (reject migrate while clearing / dual-clear both).

Phase 1 — Ref + dual-write plumbing

• Extend live db_ref with optional migration, migration_epoch, migration_phase (or keep phase only in admin meta).
• ensure_ref/1: when map has stale epoch, re-fetch from PT (or document quiesce-only MWP if epoch deferred).
• Implement dual_put / dual_delete helpers: old encoding + new encoding; same WriteBatch when as_batch active.
• Wire dual-write into insert_ / delete_ / related set paths only.
• Confirm update_index still runs once and reads from old main ref during migration; no index dual-write.
• Unit tests: concurrent insert/delete while migration set updates both CFs; index remains consistent with logical data.

Phase 2 — Create target CF + start migration

• Validate NewEncoding via mnesia_rocksdb_lib:check_encoding/2.
• Reject bag semantics; reject no-op (same encoding); reject if migration already active.
• Create empty new CF with target encoding (and consistent vsn/access_type).
• Persist migration meta; install dual-write on live ref (schema/admin txn).
• Test: after start, reads still see old data; writes appear on both CFs (inspect via raw/ref).

Phase 3 — Copy walk + scratch cursor

• Migrator process (supervised or admin-linked) with optional RocksDB snapshot.
• Iterate old CF from cursor; for each object apply install rule (§4.3):
  • live old missing → skip
  • new present → skip
  • else put re-encoded live value to new
• Update scratch cursor after each chunk (batch cursor updates).
• Progress reporting every N keys.
• On completion set phase = copy_done.
• Tests:
  • empty table
  • populated table, no concurrent writes
  • concurrent overwrites (new has key → migrator skips; final value correct)
  • concurrent deletes (no resurrection)
  • restart migrator mid-way (cursor resume, no snapshot)

Phase 4 — Recovery

• On backend/admin start: detect incomplete migration from durable meta.
• Re-install dual-write ref; restart migrator from cursor if needed.
• Test kill -9 / admin restart mid-copy; complete and verify data.

Phase 5 — Cutover + drop

• Barrier policy implemented (epoch and/or quiesce and/or delayed drop).
• finalize_migration: PT → NewRef only; clear migration fields.
• Drop old CF; clear scratch/meta.
• Test: post-cutover read/write/select/index_read; encoding is E_new.
• Test: in-flight activity behaviour under chosen barrier policy.

Phase 6 — Optional verify + polish

• Optional key-count or sample verification under dual-write before cutover.
• Metrics: keys copied, skipped (exists), skipped (deleted), duration.
• Docs: user guide section + this design note linked from README.
• Dialyzer / property tests for encode round-trip old→new.

Phase 7 — Explicitly later (not MWP)

• Mnesia type change via custom schema transaction.
• Bag table migration.
• Index CF encoding rebuild.
• Multi-node / multi-replica orchestration.

---

9. Suggested first spike (days, not weeks)

1. Dual-write only (no migrator): force migration => NewRef in test; prove insert/delete hit both CFs; indexes still correct.
2. Migrator on quiescent table: full copy + cutover + drop.
3. Add concurrent writes/deletes + install rule.
4. Add cursor + restart.
5. Snapshot as optimization only.

---

10. Document history

Date	Note
2026-08-03	Initial design from gm_mining_pool / generations encoding discussion