update serializations

Peter Harpending 2025-03-23 18:11:34 -07:00
parent 5094db8a21
commit c7e5881a6d
4 changed files with 181 additions and 5 deletions

@ -1,5 +1,13 @@
# API Encoding (`xy_ABCD` strings) # API Encoding (`xy_ABCD` strings)
## Links
- [GMSerialization API Encoder](https://git.qpq.swiss/QPQ-AG/gmserialization/src/commit/ac64e01b0f675c1a34c70a827062f381920742db/src/gmser_api_encoder.erl)
- [[BaseN]]
- [[RLP]]
## Introduction
When you are interacting with Gajumaru you often encounter garbage strings like When you are interacting with Gajumaru you often encounter garbage strings like
``` ```

@ -2,7 +2,7 @@
![BaseN diagram](./uploads/baseN-thumb-original-1024x576.png) ![BaseN diagram](./uploads/baseN-thumb-original-1024x576.png)
# Quick Reference ## Quick Reference
1. Original article: <https://zxq9.com/archives/2688> 1. Original article: <https://zxq9.com/archives/2688>
2. Base64 Erlang: 2. Base64 Erlang:
@ -14,7 +14,7 @@
5. Base58 TypeScript: 5. Base58 TypeScript:
<https://github.com/aeternity/Vanillae/blob/829dd2930ff20ea0473cf2ad562e0a1c2aba0411/bindings/typescript/src/b58.ts> <https://github.com/aeternity/Vanillae/blob/829dd2930ff20ea0473cf2ad562e0a1c2aba0411/bindings/typescript/src/b58.ts>
# tl;dr ## tl;dr
Base64 and Base58 are two schema for taking binary data and encoding it Base64 and Base58 are two schema for taking binary data and encoding it
to and from plain text. to and from plain text.
@ -64,7 +64,7 @@ to and from plain text.
2. likely to be entered manually (e.g. wallet/contract 2. likely to be entered manually (e.g. wallet/contract
addresses) addresses)
# Base64 ## Base64
The term "binary" is misleading, because it leads people to think that The term "binary" is misleading, because it leads people to think that
the basic unit in computing is a bit (a 1 or a 0). the basic unit in computing is a bit (a 1 or a 0).
@ -222,7 +222,7 @@ This is marginally trickier because
1. it needs an accumulator 1. it needs an accumulator
2. the order of the cases matters 2. the order of the cases matters
# Long Division ## Long Division
As mentioned above, Base58 is the "general" BaseN algorithm applied to As mentioned above, Base58 is the "general" BaseN algorithm applied to
the case `N=58`. This general algorithm I am calling the the case `N=58`. This general algorithm I am calling the
@ -370,7 +370,7 @@ agrees:
Our Base58 code is going to contain exactly this idea modulo an Our Base58 code is going to contain exactly this idea modulo an
unpleasant volume of edge case stupidity. unpleasant volume of edge case stupidity.
# Base58 algorithm ## Base58 algorithm
Conceptually, we are doing the following: Conceptually, we are doing the following:

25
Home.md

@ -7,3 +7,28 @@
3. [[RLP]]: If the binary `ABCD` is some data with fields (e.g. a transaction), 3. [[RLP]]: If the binary `ABCD` is some data with fields (e.g. a transaction),
how to destructure the binary data to fields how to destructure the binary data to fields
4. [[Serializations]]: Conventions for field order in Gajumaru data structures 4. [[Serializations]]: Conventions for field order in Gajumaru data structures
## Principles
- ~Every article should have a "YouTube Thumbnail" diagram at the top that
visually illustrates whatever is explained in the page
- Things should be illustrated with code examples that work when possible
- Move links to aeternity things to QPQ repos
- Every article should have a "I just need to get a thing to work and don't
have time to read a whole article" section that links to which things you
should just use in practice and maybe a short example of how to use them.
## TODOs
- API-Encoding:
- needs practical reference (which functions to call)
- maybe could use a diagram
- BaseN:
- needs practical reference
- don't link to things in aeternity repos
- RLP:
- practical reference
- don't link to aeternity repos
- Serializations:
- write
- need exhaustive reference

@ -0,0 +1,143 @@
# Serializations Reference
## Links
- [[RLP]]
- [[API Encoding]]
- [GMSerialization lib](https://git.qpq.swiss/QPQ-AG/gmserialization)
- [`gmser_id` module](https://git.qpq.swiss/QPQ-AG/gmserialization/src/commit/ac64e01b0f675c1a34c70a827062f381920742db/src/gmser_id.erl)
- [`gmserialization` module](https://git.qpq.swiss/QPQ-AG/gmserialization/src/commit/ac64e01b0f675c1a34c70a827062f381920742db/src/gmserialization.erl)
- [Gajumaru repo](https://git.qpq.swiss/QPQ-AG/gajumaru/)
## Introduction/Summary
- Chain objects are encoded into binary, using [[RLP]], according to the RLP templates here
- Those binary representations may further be encoded into plain text using the [[API Encoding]]
- As of 2025-03-23, this document is **NOT** exhaustive
- Things with multiple fields are encoded as `[Tag, Version | Fields]`
- This page is meant to succeed [the serializations page in the protocol](https://git.qpq.swiss/QPQ-AG/protocol/src/commit/d6ad171aba76aa34f4763ecd5f3a096475384271/serializations.md)
- That page is out of date
- This page is meant to be a more up-to-date version written in a way that it's easier to
1. keep up-to-date
2. detect when it's out of date
- Every part of the documentation here should have a permalink reference to
the source file. This way a reader can click on the reference and check if
it is out of date
- Each primary object type generally has an "app" dedicated to it in [the
Gajumaru repo](https://git.qpq.swiss/QPQ-AG/gajumaru) which also defines a
sort of namespace like the `aect_*` prefix.
- The serializers and things are defined in terms of those modules, which
serve an an interface to the type as an ADT so that everything can be
called in a regular way (roughly `Type:serialize(This)`).
- Example:
- [`aect_contracts` contract record](https://git.qpq.swiss/QPQ-AG/gajumaru/src/commit/7b3cc1db3bb36053023167b86f7d6f2d5dcbd01d/apps/aecontract/src/aect_contracts.erl#L72-L85)
- [`aect_contracts` serialization template](https://git.qpq.swiss/QPQ-AG/gajumaru/src/commit/7b3cc1db3bb36053023167b86f7d6f2d5dcbd01d/apps/aecontract/src/aect_contracts.erl#L294-L303)
## The `id()` type
Any account or contract etc that is somehow party to a transaction is encoded
using this convention:
```erlang
<<IdTag:8, PubKey:32/bytes>>
```
The Tags are ([source](https://git.qpq.swiss/QPQ-AG/gmserialization/src/commit/ac64e01b0f675c1a34c70a827062f381920742db/src/gmser_id.erl#L111-L118))
```erlang
decode_tag(1) -> account;
decode_tag(2) -> name;
decode_tag(3) -> commitment;
decode_tag(5) -> contract;
decode_tag(6) -> channel;
decode_tag(7) -> associate_chain;
decode_tag(8) -> native_token;
decode_tag(9) -> entry;
```
## Table of object tags
[Source](https://git.qpq.swiss/QPQ-AG/gmserialization/src/commit/ac64e01b0f675c1a34c70a827062f381920742db/src/gmser_chain_objects.erl#L120-L197)
```
rev_tag(10) -> account;
rev_tag(11) -> signed_tx;
rev_tag(12) -> spend_tx;
rev_tag(13) -> data_extend_tx;
rev_tag(30) -> name;
rev_tag(31) -> name_commitment;
rev_tag(32) -> name_claim_tx;
rev_tag(33) -> name_preclaim_tx;
rev_tag(34) -> name_update_tx;
rev_tag(35) -> name_revoke_tx;
rev_tag(36) -> name_transfer_tx;
rev_tag(37) -> name_auction;
rev_tag(40) -> contract;
rev_tag(41) -> contract_call;
rev_tag(42) -> contract_create_tx;
rev_tag(43) -> contract_call_tx;
rev_tag(44) -> contract_source;
rev_tag(50) -> channel_create_tx;
rev_tag(501) -> channel_set_delegates_tx;
rev_tag(502) -> channel_delegates;
rev_tag(51) -> channel_deposit_tx;
rev_tag(52) -> channel_withdraw_tx;
rev_tag(521) -> channel_force_progress_tx;
rev_tag(53) -> channel_close_mutual_tx;
rev_tag(54) -> channel_close_solo_tx;
rev_tag(55) -> channel_slash_tx;
rev_tag(56) -> channel_settle_tx;
rev_tag(57) -> channel_offchain_tx;
rev_tag(570) -> channel_offchain_update_transfer;
rev_tag(571) -> channel_offchain_update_deposit;
rev_tag(572) -> channel_offchain_update_withdraw;
rev_tag(573) -> channel_offchain_update_create_contract;
rev_tag(574) -> channel_offchain_update_call_contract;
rev_tag(576) -> channel_offchain_update_meta;
rev_tag(575) -> channel_client_reconnect_tx;
rev_tag(58) -> channel;
rev_tag(59) -> channel_snapshot_solo_tx;
rev_tag(60) -> trees_poi;
rev_tag(61) -> trees_db;
rev_tag(62) -> state_trees;
rev_tag(63) -> mtree;
rev_tag(64) -> mtree_value;
rev_tag(621) -> contracts_mtree;
rev_tag(622) -> calls_mtree;
rev_tag(623) -> channels_mtree;
rev_tag(624) -> nameservice_mtree;
rev_tag(626) -> accounts_mtree;
rev_tag(627) -> acs_mtree;
rev_tag(628) -> entries_mtree;
rev_tag(70) -> compiler_sophia;
rev_tag(80) -> ga_attach_tx;
rev_tag(81) -> ga_meta_tx;
rev_tag(82) -> paying_for_tx;
rev_tag(810) -> ga_meta_tx_auth_data;
rev_tag(90) -> associate_chain;
rev_tag(91) -> ac_state;
rev_tag(92) -> ac_burn;
rev_tag(93) -> ac_create_tx;
rev_tag(94) -> ac_deposit_tx;
rev_tag(95) -> ac_update_cops_tx;
rev_tag(96) -> ac_rollup_tx;
rev_tag(97) -> ac_proposal_tx;
rev_tag(100) -> key_block;
rev_tag(101) -> micro_block;
rev_tag(102) -> light_micro_block;
rev_tag(110) -> testimony;
rev_tag(111) -> testimony_tx;
rev_tag(120) -> nt_native_token;
rev_tag(121) -> nt_create_tx;
rev_tag(122) -> nt_mint_tx;
rev_tag(123) -> nt_finalize_tx;
rev_tag(124) -> nt_trade_tx;
rev_tag(125) -> nt_burn_tx;
rev_tag(140) -> entry;
rev_tag(141) -> entry_create_tx;
rev_tag(142) -> entry_transfer_tx;
rev_tag(143) -> entry_destroy_tx;
rev_tag(200) -> pof.
```