# Using Mnesia Plugins # Copyright (c) 2017-21 Aeternity Anstalt. All Rights Reserved. __Authors:__ Ulf Wiger ([`ulf@wiger.net`](mailto:ulf@wiger.net)). The Mnesia DBMS, part of Erlang/OTP, supports 'backend plugins', making it possible to utilize more capable key-value stores than the `dets` module (limited to 2 GB per table). Unfortunately, this support is undocumented. Below, some informal documentation for the plugin system is provided. This user guide illustrates these concepts using `mnesia_rocksdb` as an example. We will deal with two types of plugin: 1. backend plugins 2. index plugins A backend plugin is a module that implements a `mnesia_backend_type` behavior. Each plugin can support any number of `aliases`, which combined with the plugin module make up a `backend_type`. When using `mnesia_rocksdb`, the default alias is `rocksdb_copies`, and it is registered as a `{rocksdb_copies, mnesia_rocksdb}` pair. Once registered, the alias can be used just like the built-in backend types `ram_copies`, `disc_copies`, `disc_only_copies`. Mnesia asks the plugin module which one of the built-in types' semantics the new type is supposed to mimick: ram-only, ram+disk or disk-only. This is mainly relevant for how Mnesia checkpoints and backs up data. ### Table of Contents ### 1. [Usage](#Usage) 1. [Prerequisites](#Prerequisites) 2. [Getting started](#Getting_started) 3. [New indexing functionality](#New_indexing_functionality) ## Usage ### Prerequisites - rocksdb (included as dependency) - sext (included as dependency) - Erlang/OTP 22.0 or newer (https://github.com/erlang/otp) ### Getting started For the purposes of this user guide, we assume an unnamed, single node mnesia installation. The only place where plugins are affected by distributed Mnesia, is in the table sync callbacks. The simplest way to get all paths in order for experimentation is to check out `mnesia_rocksdb`, building it, and then calling `rebar3 shell`. Unless we note otherwise, this is how a node has been started for each example. > Erlang shell interactions have been slightly beautified by eliding > some text and breaking and indenting some lines #### Adding a backend type to mnesia There are three different ways, all undocumented, to register a backend plugin in mnesia: 1. Add a `backend_types` option when creating the schema, using `mnesia:create_schema/2` ```erlang Erlang/OTP 22 [erts-10.7] ... Eshell V10.7 (abort with ^G) 1> mnesia:create_schema([node()], [{backend_types,[{rocksdb_copies,mnesia_rocksdb}]}]). ok 2> mnesia:start(). ok 3> mnesia_schema:backend_types(). [ram_copies,disc_copies,disc_only_copies,rocksdb_copies] ``` (In `mnesia_rocksdb`, a shortcut for this exists in `mnesia_rocksdb:create_schema(Nodes)`.) 2. Add it when starting mnesia, using `mnesia:start/1` (undocumented) ```erlang Eshell V10.7 (abort with ^G) 1> mnesia:create_schema([node()]). ok 2> mnesia:start([{schema,[{backend_types, [{rocksdb_copies,mnesia_rocksdb}]}]}]). ok 3> mnesia_schema:backend_types(). [ram_copies,disc_copies,disc_only_copies] ``` 3. Call `mnesia_schema:add_backend_type/2` when mnesia is running. ```erlang Eshell V10.7 (abort with ^G) 1> mnesia:create_schema([node()]). ok 2> mnesia:start(). ok 3> mnesia_schema:add_backend_type(rocksdb_copies,mnesia_rocksdb). {atomic,ok} 4> mnesia_schema:backend_types(). [ram_copies,disc_copies,disc_only_copies,rocksdb_copies] ``` In all cases the schema is updated, and other nodes, and subsequently added nodes, will automatically receive the information. The function `mnesia_schema:backend_types()` shows which backend plugin aliases are registered. The information is also displayed when calling `mnesia:info()`: ```erlang 5> mnesia:info(). ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- schema : with 1 records occupying 443 words of mem ===> System info in version "4.16.3", debug level = none <=== opt_disc. Directory "/.../Mnesia.nonode@nohost" is used. use fallback at restart = false running db nodes = [nonode@nohost] stopped db nodes = [] master node tables = [] backend types = rocksdb_copies - mnesia_rocksdb remote = [] ram_copies = [] disc_copies = [schema] disc_only_copies = [] [{nonode@nohost,disc_copies}] = [schema] 2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] ok ``` To illustrate how mnesia persists the information in the schema: ```erlang 6> mnesia:table_info(schema,user_properties). [{mnesia_backend_types,[{rocksdb_copies,mnesia_rocksdb}]}] ``` #### Rocksdb registration shortcut Call `mnesia_rocksdb:register()` after starting mnesia. #### Creating a table Put `{rocksdb_copies, [node()]}` into the table definitions of tables you want to be in RocksDB. ```erlang 4> mnesia:create_table(t, [{rocksdb_copies,[node()]}]). {atomic,ok} 5> rr(mnesia). [commit,cstruct,cyclic,decision,log_header,mnesia_select, tid,tidstore] 6> mnesia:table_info(t,cstruct). #cstruct{name = t,type = set,ram_copies = [], disc_copies = [],disc_only_copies = [], external_copies = [{{rocksdb_copies,mnesia_rocksdb}, [nonode@nohost]}], load_order = 0,access_mode = read_write,majority = false, index = [],snmp = [],local_content = false,record_name = t, attributes = [key,val], user_properties = [],frag_properties = [], storage_properties = [], cookie = {{1621758137965715000,-576460752303423420,1}, nonode@nohost}, version = {{2,0},[]}} ``` In the example above, we take a peek at the `cstruct`, which is the internal metadata structure for mnesia tables. The attribute showing that the table has been created with a `rocksdb_copies` instance, is the `external_copies` attribute. It lists the alias, the callback module and the nodes, where the instances reside. The table works essentially like one of the built-in table types. If we want to find out which type, we can query the callback module: ```erlang 8> mnesia_rocksdb:semantics(rocksdb_copies, storage). disc_only_copies ``` Consult the `mnesia_rocksdb` man page for more info on the `Mod:semantics/2` function. ### New indexing functionality With the introduction of backend plugins, a few improvements were made to mnesia's indexing support. #### Persistent indexes In the past, and still with the built-in types, indexes were always rebuilt on startup. Since backend plugins were introduced mainly in order to support very large tables, a couple of callback functions were added in order to detect whether a full rebuild is needed. > The callback functions are `Mod:is_index_consistent/2` and > `Mod:index_is_consistent/3`. > The first function (figuratively) always returns `false` for indexes > on built-in table types. Backend plugin modules should always return > `false` if they have no information. After building the index, mnesia > calls `Mod:index_is_consistent(Alias, IxTab, true)`, and the callback > is expected to persist this information. `IxTab`, in this case, is > a logical name for the index 'table': `{Tab, index, PosInfo}` #### Ordered indexes A problem in the past with mnesia indexing has been that indexes with very large fan-out were inefficient. Indexes were represented as `bag` tables, and the cost of inserting a secondary key was proportional to the number of identical secondary keys already in the index. When adding the backend plugin support - also not least because the first candidate LevelDb didn't do bags well - support for ordered indexes was added. They turn out to be have much more stable performance for indexes with large fan-out. They also work on all built-in table types. When creating an index, you can specify the type of index as `bag` or `ordered`. If you omit the type, it will default to `bag` for built-in table types, and for external types, whatever is the first type in the list of supported index types returned by `Mod:semantics(Alias, index_types)`. > For `mnesia_rocksdb`, only `ordered` is supported, but a bug in mnesia > makes it ignore this, and try to create a bag index anyway. The > `mnesia_rocksdb` plugin rejects this. > Note that while e.g. mnesia_rocksdb supports regular bag tables, they are not > efficiently implemented. Mnesia currently doesn't allow specifying an index type in `mnesia:add_table_index/2`, so simply indicate the index position, and let the backend choose the default. Having ordered indexes opens up for some new possibilities, but there are currently no functions in mnesia such as index_first, index_next etc., or performing a select in index order. #### Index plugins Index plugins are a great new feature, also almost entirely undocumented. An index plugin is a registered indexing function, which can operate on the entire object, and shall return a list of secondary keys. When registering an index plugin, it is given an alias, a callback module, and an function name, not unlike backend plugins. The index plugin alias must be an atom wrapped inside a 1-tuple, i.e. `{atom()}`. To illustrate, we use a sample indexing function implemented in mnesia_rocksdb, which checks all non-key attributes of an object, and for each value that is a list, makes each list element a secondary key value. ```erlang 9> mnesia_schema:add_index_plugin({lv}, mnesia_rocksdb, ix_listvals). {atomic,ok} 10> mnesia:add_table_index(t,{lv}). {atomic,ok} 11> mnesia:dirty_write({t,1,[a,b]}). ok 12> mnesia:dirty_write({t,2,[b,c]}). ok 13> mnesia:dirty_index_read(t,a,{lv}). [{t,1,[a,b]}] 14> mnesia:dirty_index_read(t,b,{lv}). [{t,1,[a,b]},{t,2,[b,c]}] 15> mnesia:dirty_index_read(t,c,{lv}). [{t,2,[b,c]}] ``` For clarity, this is the implementation of the index callback: ```erlang ix_listvals(_Tab, _Pos, Obj) -> lists:foldl( fun(V, Acc) when is_list(V) -> V ++ Acc; (_, Acc) -> Acc end, [], tl(tuple_to_list(Obj))). ``` Note that the index callback must be a pure function, as it is also relied upon when deleting objects. That is, it must always return the same values when called with a specific set of input arguments.