Compare commits

..

32 Commits

Author SHA1 Message Date
Ulf Wiger 4036133476 Updated asn1 experiment, now exploring PER and OER
Gajumaru Serialization Tests / tests (push) Successful in 12s
2026-07-08 10:53:31 +02:00
Ulf Wiger 9817af8a46 Add thin ASN.1-driven RLP production layer + equivalence tests
Gajumaru Serialization Tests / tests (push) Successful in 10s
Implements gmser_asn1_rlp:encode/1 that accepts terms shaped according
to the GajumaruSerialization.asn model (e.g. GajumaruData with
templateFields or concrete CHOICEs like signedTx) and emits exactly
the same RLP bytes as the legacy gmserialization + gmser_rlp stack.

Purpose:
- Use ASN.1 as a formal, multi-language friendly schema and type model.
- Provide a thin, portable "production layer" that other languages can
  implement (ASN.1-generated types + equivalent RLP emitter).
- Preserve the existing compact RLP wire format and on-the-wire
  equivalence (no DER bloat for the legacy path).

The implementation walks the ASN.1-shaped value and applies the same
encoding rules as gmserialization:encode_field/2 (minimal unsigned
integers, positional values for static maps, etc.) followed by
gmser_rlp:encode/1.

Includes EUnit equivalence tests covering:
- Simple fields, zero/empty values
- Lists and tuples
- List-of-tuples (e.g. type_info)
- Concrete signedTx
- ContractV3 (including bool in structured data)

All tests assert byte-for-byte identity with legacy serialization and
that the resulting bytes are still accepted by legacy decoders.

This continues the proof-of-concept on the uw-asn1 branch.
2026-07-07 10:48:30 +02:00
Ulf Wiger 6d7ab1e4ad PoC: ASN.1 / DER serialization experiment
Gajumaru Serialization Tests / tests (push) Successful in 10s
This branch contains a proof-of-concept exploration of representing
the RLP (gmser_rlp) and gmserialization layers using ASN.1 with
DER encoding.

Goals of the experiment:
- Model the static serialization templates and dynamic types in ASN.1.
- Demonstrate reliable detection of legacy RLP data vs. new DER data
  (first byte >= 0xC0 for RLP lists vs. 0x30 for DER SEQUENCE).
- Evaluate wire-size overhead of standard DER TLV encoding compared
  to the extremely compact RLP prefix encoding.
- Verify that the DER path produces deterministic output suitable
  for hashing (idempotent serialization).

Key files:
- GajumaruSerialization.asn   -- the ASN.1 schema
- GajumaruSerialization.{erl,hrl} -- generated encoder/decoder
- detect_demo.erl, size_comparison.erl -- supporting test code

Findings (high level):
- Detection via first byte works cleanly.
- DER is fully deterministic for hashing when the same value is given.
- Small objects pay a significant size overhead (often 3-6x) due to
  TLV tags/lengths and field names in the generic path.
- Concrete SEQUENCE definitions are much more compact than the
  generic templateFields representation.
- Large payloads have acceptable relative overhead.

This is NOT production code. It was created to answer feasibility
questions around ASN.1 modeling, format detection, determinism,
and size characteristics.

Branch created from uw-new-acs for isolated experimentation.
2026-07-07 10:05:30 +02:00
Ulf Wiger c80505c810 Fix ac_acct_state tag and restore key_block mapping
Gajumaru Serialization Tests / tests (push) Successful in 11s
The previous commit accidentally removed key_block (tag 100) instead of
adding ac_acct_state (tag 99).
2026-07-07 08:13:37 +02:00
Ulf Wiger 0e157f824a Add ac_acct_state serialization tag
Gajumaru Serialization Tests / tests (push) Successful in 10s
2026-07-07 08:07:58 +02:00
Ulf Wiger faa20bd00b Add ac_receipt and AC side tx tags from ac793dcaf6
Gajumaru Serialization Tests / tests (push) Successful in 12s
Cherry-pick the chain-object tags from ac793dcaf6 while keeping the
extended-auth account types (account_sig_store, auth_tx, etc.).
2026-07-06 22:27:28 +02:00
Ulf Wiger 79b43b0665 Add account_auth_update_tx tag
Gajumaru Serialization Tests / tests (push) Successful in 11s
2026-07-06 15:30:19 +02:00
Ulf Wiger a84dcc880d Add is_account/1 and account_pubkey/1 helpers
Gajumaru Serialization Tests / tests (push) Successful in 11s
Extended-auth account ids use an {account, Subtype} tag. These helpers
let callers test account membership and extract the 32-byte pubkey
without pattern-matching on specialize/1 results.
2026-07-06 12:26:38 +02:00
Ulf Wiger a368c64f7e Add missing tag for proposal_gossip_tx
Gajumaru Serialization Tests / tests (push) Successful in 11s
2026-07-06 10:55:20 +02:00
Ulf Wiger dbfc013c8a Add auth tx and account sig store
Gajumaru Serialization Tests / tests (push) Successful in 11s
2026-07-06 10:19:38 +02:00
Ulf Wiger 7a8e840793 Add support for extended-auth account types
Gajumaru Serialization Tests / tests (push) Successful in 11s
2026-07-05 08:38:42 +02:00
uwiger eecff86500 Merge pull request 'Add zompify support, verup to 0.2.0' (#59) from uw-zomp-vsn-0.2.0 into master
Gajumaru Serialization Tests / tests (push) Successful in -2m7s
Reviewed-on: #59
2026-04-17 14:25:34 +09:00
Ulf Wiger 2ac9363d30 Add zompify support, verup to 0.2.0
Gajumaru Serialization Tests / tests (push) Successful in -2m7s
2026-04-16 23:31:12 +02:00
uwiger 4698b54832 Merge pull request 'Add keypair encoding, fix seckey size checks' (#57) from uw-account_seckey into master
Gajumaru Serialization Tests / tests (push) Successful in -2m43s
Reviewed-on: #57
2026-03-30 05:26:09 +09:00
Ulf Wiger 4cc6adee2e Add unsafe_encode/2 to forego size check
Gajumaru Serialization Tests / tests (push) Successful in -2m49s
2026-03-25 22:10:06 +01:00
Ulf Wiger 4f97dd1bd1 Add keypair encoding, fix seckey size checks
Gajumaru Serialization Tests / tests (push) Successful in -2m51s
2026-03-24 23:06:29 +01:00
uwiger 05bbf058be Map 'hash' as known api type, add some more tests (#56)
Gajumaru Serialization Tests / tests (push) Successful in -2m53s
I noticed that the 'hash' serialization type wasn't in the `known_type()` set, although it was supported otherwise.
This PR adds it to the `known_type()` and also exports the type `gmser_api_encoder:known_type()`.

Also, some more tests are added to try to detect this sort of thing in the future.

Co-authored-by: Ulf Wiger <ulf@wiger.net>
Reviewed-on: #56
2026-03-24 21:23:20 +09:00
uwiger 2b24b17af3 Merge pull request 'Fix OTP 28 dialyzer warnings, rewrite gmser_dyn decoder' (#55) from uw-gmser_dyn-rewrite into master
Gajumaru Serialization Tests / tests (push) Successful in -2m54s
Reviewed-on: #55
2026-03-24 21:20:30 +09:00
Ulf Wiger 00699b08b7 Fix OTP 28 dialyzer warnings, rewrite gmser_dyn decoder
Gajumaru Serialization Tests / tests (push) Successful in -3m56s
gmser_dyn no longer tries to compress output by omitting type tags.
Decoding streams using custom template codes can either use 'strict'
decoding, in which case matching templates must be registered on the
decoding end; in `strict => false` mode, the stream can still be decoded
without valudation if the custom template is missing.
2026-02-18 21:10:43 +01:00
Ulf Wiger 8734e67c66 WIP refactor gmser_dyn 2026-02-15 12:12:04 +01:00
uwiger dda5cac7a9 Merge pull request 'Serialization got broken by previous PR' (#54) from uw-fix-serialization into master
Gajumaru Serialization Tests / tests (push) Successful in 49m58s
Reviewed-on: #54
2025-04-29 17:22:22 +09:00
Ulf Wiger 07d61722b4 Serialization got broken by previous PR
Gajumaru Serialization Tests / tests (push) Successful in 49m58s
2025-04-29 10:18:59 +02:00
uwiger 4ac7531351 Merge pull request 'uw-switch-semantics' (#53) from uw-switch-semantics into master
Gajumaru Serialization Tests / tests (push) Successful in 49m56s
Reviewed-on: #53
2025-04-29 03:57:16 +09:00
Ulf Wiger f996253e6b Add forgotten exports, expand(Types) function
Gajumaru Serialization Tests / tests (push) Successful in 49m55s
2025-04-28 12:12:43 +02:00
Ulf Wiger b9a51acf55 Add gmser_dyn_types.erl
Gajumaru Serialization Tests / tests (push) Successful in 49m54s
2025-04-28 11:59:27 +02:00
Ulf Wiger 5df23c05c1 test case for 'switch'
Gajumaru Serialization Tests / tests (push) Failing after 49m55s
2025-04-28 11:51:23 +02:00
Ulf Wiger b358dfe914 Add switch semantics
Gajumaru Serialization Tests / tests (push) Failing after 49m54s
2025-04-28 11:36:02 +02:00
uwiger 0288719ae1 Merge pull request 'Save options, test cases for missing labels' (#52) from uw-save-options into master
Gajumaru Serialization Tests / tests (push) Successful in 49m42s
Reviewed-on: #52
2025-04-24 06:46:26 +09:00
Ulf Wiger 795c7f7860 Save options, test cases for missing labels
Gajumaru Serialization Tests / tests (push) Successful in 49m42s
2025-04-23 23:36:03 +02:00
uwiger 0d77ca0388 Fix function_clause bug (#51)
Gajumaru Serialization Tests / tests (push) Successful in 49m16s
Co-authored-by: Ulf Wiger <ulf@wiger.net>
Reviewed-on: #51
2025-04-14 19:10:58 +09:00
uwiger ed204f8526 Merge pull request 'uw-dyn-options' (#50) from uw-dyn-options into master
Gajumaru Serialization Tests / tests (push) Failing after 49m16s
Reviewed-on: #50
2025-04-14 18:59:38 +09:00
Ulf Wiger a949d166f6 Add options for deserialization of missing labels
Gajumaru Serialization Tests / tests (push) Failing after 49m16s
2025-04-14 11:54:48 +02:00
48 changed files with 14104 additions and 341 deletions
+139 -16
View File
@@ -49,8 +49,25 @@ range from 10 to 200, and also to stay within 1 byte.)
When encoding `map` types, the map elements are first sorted.
When specifying a map type for template-driven encoding, use
the `#{items => [{Key, Value}]}` construct.
the `#{items => [{Key, ValueType} | {opt, Key, ValueType}]}` construct.
The key names are included in the encoding, and are match against the item
specs during decoding. If the key names don't match, the decoding fails, unless
for an `{opt, K, V}` item, in which case that item spec is skipped.
```erlang
T = #{items => [{a,int},{opt,b,int},{c,int}]}
E1 = gmser_dyn:encode_typed(T, #{a => 1, b => 2, c => 3}) ->
[<<0>>,<<1>>,[<<252>>,
[[[<<255>>,<<97>>],[<<248>>,<<1>>]],
[[<<255>>,<<98>>],[<<248>>,<<2>>]],
[[<<255>>,<<99>>],[<<248>>,<<3>>]]]]]
E2 = gmser_dyn:encode_typed(T, #{a => 1, c => 3}) ->
[<<0>>,<<1>>,[<<252>>,
[[[<<255>>,<<97>>],[<<248>>,<<1>>]],
[[<<255>>,<<99>>],[<<248>>,<<3>>]]]]]
gmser_dyn:decode_typed(T,E2) ->
#{c => 3,a => 1}
```
## Labels
@@ -58,6 +75,24 @@ Labels correspond to (existing) atoms in Erlang.
Decoding of a label results in a call to `binary_to_existing_atom/2`, so will
fail if the corresponding atom does not already exist.
This behavior can be modified using the option `#{missing_labels => fail | create | convert}`,
where `fail` is the default, as described above, `convert` means that missing atoms are
converted to binaries, and `create` means that the atom is created dynamically.
The option can be passed e.g.:
```erlang
gmser_dyn:deserialize(Binary, gmser_dyn:set_opts(#{missing_labels => convert}))
```
or
```erlang
gmser_dyn:deserialize(Binary, gmser_dyn:set_opts(#{missing_labels => convert}, Types))
```
By calling `gmser_dyn:register_types/1`, after having added options to the type map,
the options can be made to take effect automatically.
It's possible to cache labels for more compact encoding.
Note that when caching labels, the same cache mapping needs to be used on the
decoder side.
@@ -96,31 +131,70 @@ Templates can be provided to the encoder by either naming an already registered
type, or by passing a template directly. In both cases, the encoder will enforce
the type information in the template.
If the template has been registered, the encoder omits inner type tags (still
inserting the top-level tag), leading to some compression of the output.
This also means that the serialized term cannot be decoded without the same
schema information on the decoder side.
In the case of a directly provided template, all type information is inserted,
such that the serialized term can be decoded without any added type information.
The template types are still enforced during encoding.
If the template has been registered, the encoder uses the registered type specification
to drive the encoding. The code of the registered template is embedded in the encoded
output:
```erlang
ET = fun(Type,Term) -> io:fwrite("~w~n", [gmser_dyn:encode_typed(Type,Term)]) end.
gmser_dyn:encode_typed({int,int,int}, {1,2,3}) ->
[<<0>>,<<1>>,[<<253>>,
[[<<248>>,<<1>>],[<<248>>,<<2>>],[<<248>>,<<3>>]]]]
ET([{int,int}], [{1,2}]) -> [<<0>>,<<1>>,[<<251>>,[[[<<248>>,<<1>>],[<<248>>,<<2>>]]]]]
gmser_dyn:register_type(1000,lt2i,[{int,int}]).
ET(lt2i, [{1,2}]) -> [<<0>>,<<1>>,[<<3,232>>,[[<<1>>,<<2>>]]]]
Types = gmser_dyn_types:add_type(t3,1013,{int,int,int}).
gmser_dyn:encode_typed(t3, {1,2,3}, Types) ->
[<<0>>,<<1>>,[[<<3,245>>,<<253>>],
[[<<248>>,<<1>>],[<<248>>,<<2>>],[<<248>>,<<3>>]]]]
```
Note that the original `<<253>>` type code is wrapped as `[<<3,245>>,<<253>>]`,
where `<<3,245>>` corresponds to the custom code `1013`.
Using the default option `#{strict => true}`, the decoder will extract the custom
type spec, and validate the encoded data against it. If the custom code is missing,
the decoder aborts. Using `#{strict => false}`, the custom code is used if it exists,
but otherwise, it's ignored, and the encoded data is decoded using the dynamic type
info.
### Alternative types
The dynamic encoder supports two additions to the `gmserialization` template
language: `any` and `#{alt => [AltTypes]}`.
The dynamic encoder supports a few additions to the `gmserialization` template
language: `any`, `#{list => Type}`, `#{alt => [AltTypes]}` and `#{switch => [AltTypes]}`.
#### `any`
The `any` type doesn't have an associated code, but enforces dynamic encoding.
#### `list`
The original list type notation expects a key-value list, e.g.
`[{name, binary}, {age, int}]`
```erlang
EL = gmser_dyn:encode_typed([{name,binary},{age,int}], [{name,<<"Ulf">>},{age,29}]) ->
[<<0>>,<<1>>,[<<251>>,
[[<<253>>,[[<<255>>,<<110,97,109,101>>],[<<249>>,<<85,108,102>>]]],
[<<253>>,[[<<255>>,<<97,103,101>>],[<<248>>,<<29>>]]]]]]
```
Note that the encoding explicitly lays out a `[{Key, Value}]` structure, all
dynamically typed. This means it can be dynamically decoded without templates.
```erlang
gmser_dyn:decode(EL).
[{name,<<"Ulf">>},{age,29}]
```
In order to specify something like Erlang's `[integer()]` type, we can use
the following:
```erlang
gmser_dyn:encode_typed(#{list => int}, [1,2,3,4]) ->
[<<0>>,<<1>>,[<<251>>,
[[<<248>>,<<1>>],[<<248>>,<<2>>],[<<248>>,<<3>>],[<<248>>,<<4>>]]]]
```
#### `alt`
The `#{alt => [Type]}` construct also enforces dynamic encoding, and will try
to encode as each type in the list, in the specified order, until one matches.
@@ -132,6 +206,55 @@ gmser_dyn:encode_typed(anyint,-5) -> [<<0>>,<<1>>,[<<246>>,[<<247>>,<<5>>]]]
gmser_dyn:encode_typed(anyint,5) -> [<<0>>,<<1>>,[<<246>>,[<<248>>,<<5>>]]]
```
#### `switch`
The `switch` type allows for encoding a 'tagged' object, where the tag determines
the type.
```erlang
E1 = gmser_dyn:encode_typed(#{switch => #{name => binary, age => int}}, #{age => 29}) ->
[<<0>>,<<1>>,[<<252>>,[[[<<255>>,<<97,103,101>>],[<<248>>,<<29>>]]]]]
gmser_dyn:decode_typed(#{switch => #{name => binary, age => int}}, E1) ->
#{age => 29}
E2 = gmser_dyn:encode_typed(#{switch => #{name => binary, age => int}}, #{name => <<"Ulf">>}) ->
[<<0>>,<<1>>,[<<252>>,[[[<<255>>,<<110,97,109,101>>],[<<249>>,<<85,108,102>>]]]]]
gmser_dyn:decode_typed(#{switch => #{name => binary, age => int}}, E1) ->
#{name => <<"Ulf">>}
```
A practical use of `switch` would be in a protocol schema:
```erlang
t_msg(_) ->
#{switch => #{ call => t_call
, reply => t_reply
, notification => t_notification }}.
t_call(_) ->
#{items => [ {id, anyint}
, {req, t_req} ]}.
t_reply(_) ->
#{alt => [#{items => [ {id, anyint}
, {result, t_result} ]},
#{items => [ {id, anyint}
, {code, anyint}
, {message, binary} ]}
]}.
```
In this scenario, messages are 'taggged' as 1-element maps, e.g.:
```erlang
async_request(Msg) ->
Id = erlang:unique_integer(),
gmmp_cp:to_server(
whereis(gmmp_core_connector),
#{call => #{ id => Id
, req => Msg }}),
Id.
```
### Notes
Note that `anyint` is a standard type. The static serializer supports only
+212
View File
@@ -0,0 +1,212 @@
# ASN.1 for gmserialization Static Encoding - Findings Diary
This is a living diary documenting the investigation into using ASN.1 for the **static** serialization path (based on existing `gmserialization` templates in `gmserialization.erl` and `gmser_chain_objects.erl`). Dynamic encoding is out of scope.
Focus areas:
- Modeling static templates with ASN.1 (portability goal).
- Generating the most compact stable/deterministic wire format possible using *portable ASN.1 techniques* (UPER etc.).
- Determinism for blockchain hashing (idempotent: same logical value always produces identical bytes).
- (Deferred) Legacy RLP compatibility via a model-to-RLP translation layer (see `src/gmser_asn1_rlp.erl`).
The single source of truth is the ASN.1 schema in `GajumaruSerialization.asn`.
---
## 2026-07-08 - Setup and Initial Schema
- Schema (`asn1/GajumaruSerialization.asn`) models:
- `GajumaruData` as top-level (tag + vsn + content).
- `Content` CHOICE with `templateFields` (generic) and concrete types (e.g. `SignedTx`, `ContractV*`).
- `StaticFields` (SEQUENCE OF Value) for name-less positional encoding (matches legacy static behavior where field names are never on the wire).
- `Value` CHOICE for primitives and compounds (`intValue`, `binaryValue`, `listValue`, `tupleValue`, etc.).
- Supports all static template types: `int`, `bool`, `binary`, `id`, `[T]`, tuples, `#{items => [...]}`.
- Initial schema comments were DER-oriented (migration path). Updated to emphasize compact UPER + portability.
- Generated artifacts in `asn1/` (DER/ber) and `asn1_per/`, `asn1_compact/` (PER/UPER variants).
- Key files:
- `asn1/GajumaruSerialization.asn` (source)
- `src/gmser_asn1_rlp.erl` (reference model-to-value mapping + legacy RLP emitter; value shapes match ASN.1)
- Tests in `test/gmser_chain_objects_tests.erl` and inside `gmser_asn1_rlp` for equivalence.
## 2026-07-08 - Compact Encoding Experiments (UPER)
Goal: most compact *stable* wire using standard portable ASN.1 (not custom non-portable rules, not RLP).
- Tried standard DER → too verbose (tiny case: ~36 bytes vs legacy RLP 5 bytes).
- Switched to **UPER (Unaligned PER)** — the most compact *standard* ASN.1 encoding rule with good canonical/deterministic properties.
- Compiled via `asn1ct:compile(..., [uper])`.
- Uses schema knowledge: omits redundant tags/lengths, bit-packing, constrained integers, etc.
- Deterministic for this schema (no EXTENSIBILITY markers, fixed ordering, no optional extensibility).
- Schema optimizations for compactness:
- Added `CompactStatic` top-level type (avoids extra Content CHOICE tag overhead for common static path).
- `staticFields` (pure `SEQUENCE OF Value`) — no IA5String names on wire.
- Constrained `tag`/`vsn` (INTEGER (0..65535), (0..255)) for better packing.
- Prefer concrete SEQUENCES (e.g. `SignedTx`) or `staticFields` over generic `templateFields` (names add cost).
- Kept `TemplateFields` for debug/transition only.
- Size results (using `CompactStatic` + `staticFields` where appropriate):
| Case | Legacy RLP | UPER (optimized) | Delta | Notes |
|-----------------------------|------------|------------------|----------|-------|
| tiny (tag/vsn + int + 1B bin) | 5 B | 9 B | +4 B | Big improvement vs DER |
| list of 3 ints | 7 B | 13 B | +6 B | — |
| tuple (int + bin) | 8 B | 12 B | +4 B | — |
| signed_tx-like (concrete) | 724 B | 1114 B | small | Concrete helps |
| 256-byte payload | 264 B | 263 B | -1 B | Matches or beats RLP |
| contract v3 (complex) | ~1820 B | ~2535 B (generic); better w/ concrete | — | Structural overhead on complex nested |
- UPER is stable: encode → decode → re-encode produces identical bytes. Roundtrips work.
- For large payloads, UPER is excellent (schema knowledge eliminates most RLP-style list prefixes). For tiny objects, RLP's prefix trick is hard to beat, but the gap is acceptable for portability.
- OER (Octet Encoding Rules) also compiled but was larger (18 B on tiny case).
## 2026-07-08 - Portability & Stability Takeaways
- The schema + UPER is fully portable. Other languages can:
1. Compile the `.asn` with their ASN.1 tool.
2. Build a value matching `CompactStatic` / `staticFields` / concrete types.
3. Call their UPER encoder → identical compact bytes.
- No Erlang-specific runtime required for the new wire format.
- Determinism comes from:
- UPER canonical packing rules.
- Constrained types in schema.
- Explicit staticFields (no map iteration, names omitted).
- Same rules as legacy for ints (minimal), ordering, etc.
- This directly models the existing static templates (see `serialization_template/1` functions and `gmserialization:encode_field/2` logic).
- Concrete types in schema give best compactness for known objects.
- Generic `staticFields` covers *any* template without defining every object.
## Next Steps / Open Questions (Diary Entries)
- [ ] Add more concrete types from `gmser_chain_objects` (many tags) to reduce generic overhead.
- [ ] Add more ASN.1 constraints (SIZE, value ranges) to help PER pack tighter.
- [ ] Measure on real on-chain objects (key_block, etc.).
- [ ] Decide on top-level header for the new format (keep tag/vsn?).
- [ ] (Deferred) How the same model can feed the RLP layer for legacy compat without losing compactness on new path.
- [ ] Consider if a custom "ASN.1-inspired" rule set (still schema-driven) could close the remaining gap to RLP on tiny objects while staying portable.
## 2026-07-08 - Schema Updated for Bignums
- Updated `GajumaruSerialization.asn`:
- Introduced `BigInt ::= INTEGER (0..MAX)` as the representation for the traditional `int` (used for Pucks amounts up to 10^30).
- `Value` CHOICE now uses `bigIntValue` for the bignum case.
- Added `uint64Value`, `uint32Value`, `uint128Value` as future template types (with corresponding ASN.1 subtypes).
- Updated header comments to document the bignum nature of `int`.
- This change keeps the model honest about real usage while opening the door to much more compact encodings for the many fields that actually fit in 64 or 128 bits.
- Next: We should extend the Erlang-side `type()` in `gmserialization.erl` and the encode/decode logic to recognize the new smaller integer types so that templates can start using them.
---
*This file should be kept as a living diary. Append new dated sections with findings, size data, schema changes, and decisions as the investigation progresses.*
## 2026-07-08 - Handling of `int` as Bignums (Pucks, amounts, etc.)
Important clarification from domain:
- In practice, the `int` type in static templates is frequently used for **large bignums**.
- Example: Amount fields (balances, transaction amounts, etc.) are denominated in "Pucks".
- Maximum value mentioned: 1 × 10^30.
- This is ~ 2^99.66, i.e., requires up to ~13 bytes in minimal unsigned encoding.
Current legacy handling (in `gmserialization.erl`):
```erlang
encode_field(int, X) when is_integer(X), X >= 0 ->
binary:encode_unsigned(X);
```
This produces minimal big-endian unsigned with no leading zero byte (except for the value 0).
Implications for ASN.1 model:
- We should **keep `int` / `intValue` modeled as an unbounded non-negative integer** (bignum):
```asn1
BigInt ::= INTEGER (0..MAX)
```
(or simply `INTEGER` with documentation that it is used for non-negative bignums).
- Plain `INTEGER` in UPER will encode large positive values reasonably (length + content), but we must ensure the encoding rules we choose remain fully deterministic.
- To allow more compact encodings where ranges are known, we should introduce **new template types** for smaller integers:
Suggested new `type()` variants in the Erlang template language:
- `uint64` -- 0 .. 2^64-1
- `uint32` -- 0 .. 2^32-1
- `uint16`, `uint8`, `uint128` etc. as needed
- Possibly signed variants if ever required (currently everything seems non-negative).
Corresponding in ASN.1 (inside Value CHOICE or as reusable types):
```asn1
Uint64 ::= INTEGER (0..18446744073709551615)
Uint32 ::= INTEGER (0..4294967295)
Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
```
- In the schema's Value CHOICE we can evolve to:
```asn1
Value ::= CHOICE {
bigIntValue [0] BigInt, -- the classic "int" for Pucks etc.
uint64Value [7] Uint64,
uint32Value [8] Uint32,
...
-- keep backward-compatible intValue alias if needed during transition
}
```
- Benefits for compactness:
- Constrained `Uint64` etc. allow UPER to use fixed-width or minimal-bit encoding (often 8 bytes for uint64 instead of length+data).
- Still fully portable and deterministic.
- Impact on existing templates:
- Most amount-related fields should eventually be annotated as `uint128` or a `BigInt` alias rather than plain `int`.
- Small counters, versions, indices can use `uint32` / `uint64`.
- This may require extending the `type()` in `gmserialization.erl` and the encoders.
## 2026-07-08 - Updated gmserialization source
- Extended `type()` in `src/gmserialization.erl` with `uint128`, `uint64`, `uint32`, `uint16`, `uint8` (keeping `int` for bignums).
- Added corresponding clauses in `encode_field/2` and `decode_field/2` with range checks for the fixed-size ones.
- Updated `src/gmser_asn1_rlp.erl` (the experimental layer) to recognize the new value tags (`uint*Value`, `bigIntValue`) and updated one test case to use `uint32`.
- Updated `doc/static.md` example types.
- `int` remains fully backward compatible for bignum use.
- All existing tests + new type usage pass.
- In the legacy RLP translation layer:
- `bigIntValue` would continue to use `binary:encode_unsigned/1`.
- Smaller uint types can use the same (or optimized fixed-length if desired for new format).
Next action items:
- Update `GajumaruSerialization.asn` to introduce `BigInt`, `Uint*` types and adjust Value.
- Decide on naming in the Erlang template DSL (`int` remains bignum alias? or rename?).
- Add example in the schema for a balance field.
- Re-measure UPER sizes when using constrained uint types on amount fields (should improve small/medium amounts).
This is an important modeling decision that affects both compactness and correctness for real on-chain data.
## 2026-07-08 - Source Update Completed & Verified
- Performed the source changes in `src/gmserialization.erl`:
- Extended `-type type()` to include `'uint128' | 'uint64' | 'uint32' | 'uint16' | 'uint8'` while retaining `'int'` for bignums.
- Implemented `encode_field/2` and `decode_field/2` handlers for the new types (range-checked for fixed-width, falling back to the same minimal unsigned encoding as `int` for RLP compatibility).
- Synced `src/gmser_asn1_rlp.erl`:
- Added support for new ASN.1 value variants (`{uint*Value, ...}`, `{bigIntValue, ...}`) in `encode_asn1_value/1`.
- Updated test data and comments to use the new types.
- Updated `doc/static.md` to reflect the extended type language.
- Verified:
- Legacy templates using `int` continue to work unchanged.
- New types (e.g. `{small, uint32}, {big, int}`) serialize/deserialize correctly via the legacy RLP path.
- All 7 equivalence tests in `gmser_asn1_rlp` still pass.
- `gmser_chain_objects_tests` (12 tests) still pass.
- The ASN.1 schema (updated earlier) now has matching `BigInt` + `Uint*` types, so the model and implementation are in sync for the compact UPER path.
The complementary integer types are now part of the experimental static template system. This enables the ASN.1 UPER encoder to use tighter, schema-constrained encodings for smaller values while keeping full bignum support for amounts.
Next diary items (still open):
- Extend more concrete types in the schema.
- Add actual UPER-based encode path in the library (beyond the RLP layer).
- Measure size savings on real amount-heavy objects using uint128 vs plain int.
+267
View File
@@ -0,0 +1,267 @@
-- GajumaruSerialization.asn
--
-- ASN.1 model for the structures serialized by gmserialization.erl
-- (the layer on top of RLP).
--
-- Purpose:
-- * Provide a formal, toolable description of the data for portability
-- (other languages use ASN.1 compilers to get types/parsers).
-- * Define compact canonical wire format using standard ASN.1 techniques
-- (primarily Unaligned PER / UPER with constraints for packing).
-- * Static encoding only (templates from gmserialization).
-- * Legacy RLP compatibility is achieved via separate translation layer
-- (deferred in current focus).
--
-- Detection of legacy vs new:
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
-- This means the first byte is in the range 0xC0 .. 0xFF.
--
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
-- constructed, universal tag) for the outermost GajumaruData (or any
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
-- check reliably distinguishes the two formats.
--
-- Usage in Erlang (after compiling with asn1ct for compact wire):
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [uper]).
-- {ok, Bytes} = 'GajumaruSerialization':encode('GajumaruData', Asn1Value).
-- Bytes is the compact UPER wire format (deterministic with this schema).
-- For legacy RLP, use the model-to-RLP layer instead (see gmser_asn1_rlp).
--
-- Notes on type mapping from gmserialization.erl (static templates):
-- - int (bignum) -> BigInt ::= INTEGER (0..MAX)
-- In practice used for amounts/balances in Pucks (up to 1*10^30).
-- Encoded as non-negative bignum.
-- - New smaller integer types will be added to templates (uint64, uint32, ...)
-- for cases where range is known → enables tighter UPER packing.
-- - binary -> OCTET STRING
-- - bool -> BOOLEAN
-- - id -> Id (SEQUENCE)
-- - [T] -> SEQUENCE OF T
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
-- (static maps carry *values only*, no keys on the wire)
-- - tag + vsn are preserved as the first two fields for compatibility
-- with existing dispatch logic.
--
-- WARNING:
-- Any cryptographic hash or signature computed over a serialized object
-- will change when switching from RLP to DER for that object. Plan a
-- coordinated upgrade.
GajumaruSerialization DEFINITIONS
AUTOMATIC TAGS ::=
BEGIN
EXPORTS ALL;
-- ============================================================
-- Top-level wrapper used for new DER data.
-- This is what a decoder will see first.
-- ============================================================
GajumaruData ::= SEQUENCE {
-- Constrained for better PER packing
tag INTEGER (0..65535),
vsn INTEGER (0..255),
content Content
}
-- Preferred top-level for the compact static wire format.
-- Avoids the extra Content CHOICE tag when the structure is known to be static.
CompactStatic ::= SEQUENCE {
tag INTEGER (0..65535),
vsn INTEGER (0..255),
fields StaticFields
}
-- Content can be a specific structured type (preferred) or a generic
-- representation of a template-driven object.
Content ::= CHOICE {
-- Generic fallback that can represent any [{Field, Type}] template
-- without having a pre-defined SEQUENCE for every object.
templateFields [0] TemplateFields,
-- Static-optimized: no field names on wire (matches legacy static behavior exactly)
-- Preferred for compact wire format of known templates.
staticFields [10] StaticFields,
-- Examples of concrete versioned types (extend as needed)
account [1] Account,
signedTx [2] SignedTx,
contract [3] ContractCode
-- Add more alternatives for other tags from gmser_chain_objects
}
-- ============================================================
-- Generic template-driven representation
-- (useful during transition or for unregistered types)
-- ============================================================
TemplateFields ::= SEQUENCE OF TemplateField
TemplateField ::= SEQUENCE {
-- Field name is included for debuggability / generic processing.
-- In the original static encoding the name is NOT on the wire;
-- only position and type matter. We include it here for convenience.
name IA5String OPTIONAL,
value Value
}
-- Optimized for static wire format: just the values in order, no names.
-- This matches the legacy static encoding where maps/records are positional only.
StaticFields ::= SEQUENCE OF Value
Value ::= CHOICE {
-- "int" in static templates is used for bignums in practice
-- (e.g. balances and amounts in Pucks, up to 1*10^30).
-- We keep it as unbounded non-negative integer.
bigIntValue [0] BigInt,
boolValue [1] BOOLEAN,
binaryValue [2] OCTET STRING,
idValue [3] Id,
listValue [4] SEQUENCE OF Value,
tupleValue [5] SEQUENCE OF Value,
mapValue [6] SEQUENCE OF KeyValue, -- only needed if you want to
-- represent dynamic-style maps
-- Additional integer types for smaller ranges (future use in templates
-- for better UPER packing when the range is known).
uint64Value [7] Uint64,
uint32Value [8] Uint32,
uint128Value [9] Uint128
}
-- Bignum integer (non-negative). Used for the traditional "int" in static
-- templates. Max practical value mentioned: 1*10^30 (≈ 2^100 bits).
BigInt ::= INTEGER (0..MAX)
-- Convenience sized unsigned integer types.
Uint64 ::= INTEGER (0..18446744073709551615)
Uint32 ::= INTEGER (0..4294967295)
Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
KeyValue ::= SEQUENCE {
key Value,
val Value
}
-- ============================================================
-- Common types
-- ============================================================
Id ::= SEQUENCE {
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
-- and the extended account subtype (high bit in legacy).
type INTEGER (0..255),
value OCTET STRING (SIZE (32))
}
-- ============================================================
-- Concrete object examples (derived from usage in the codebase)
-- Add / evolve per version as you introduce new vsns.
-- ============================================================
-- Example: a very simple account-like object used in tests
Account ::= SEQUENCE {
foo INTEGER,
bar OCTET STRING
}
-- Simplified signed transaction
SignedTx ::= SEQUENCE {
signatures SEQUENCE OF OCTET STRING,
transaction OCTET STRING
}
-- Contract code objects (see gmser_contract_code.erl)
-- We model the three versions that exist today.
ContractCode ::= CHOICE {
v1 [0] ContractV1,
v2 [1] ContractV2,
v3 [2] ContractV3
}
ContractV1 ::= SEQUENCE {
sourceHash OCTET STRING,
-- typeInfo is a list of 4-tuples in legacy:
-- {typeHash, name, argType, outType}
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING
}
ContractV2 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING,
compilerVersion OCTET STRING
}
ContractV3 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV3,
byteCode OCTET STRING,
compilerVersion OCTET STRING,
payable BOOLEAN
}
TypeInfoV1 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
argType OCTET STRING,
outType OCTET STRING
}
TypeInfoV3 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
payable BOOLEAN,
argType OCTET STRING,
outType OCTET STRING
}
-- ============================================================
-- Notes for implementers
-- ============================================================
-- 1. Detection (recommended decoder entry point):
--
-- decode(Binary) ->
-- case Binary of
-- <<B, _/binary>> when B >= 16#C0 ->
-- decode_legacy_rlp(Binary); % existing gmser_* path
-- _ ->
-- {ok, Term} =
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
-- Term
-- end.
--
-- This works because:
-- - All current top-level output from serialize() is an RLP list
-- (first byte 0xC0-0xFF).
-- - GajumaruData and the concrete choices above are SEQUENCEs
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
--
-- 2. When you add a new object type or version, prefer adding a
-- concrete SEQUENCE alternative in the Content CHOICE rather than
-- always falling back to TemplateFields. This gives you better
-- validation and generated types.
--
-- 3. For the compact wire format we use UPER (unaligned PER).
-- It is stable/deterministic for a given schema (no extensibility
-- markers on these types, fixed order).
--
-- 4. INTEGER uses ASN.1 PER encoding (canonical for the constraints).
-- This may differ from legacy RLP minimal unsigned; new format
-- will have different hashes (expected when introducing new encoding).
--
-- 5. Dynamic encoding (gmser_dyn) is not in scope here.
--
-- 6. To generate the compact wire:
-- asn1ct:compile(GajumaruSerialization, [uper]).
-- {ok, CompactBytes} = 'GajumaruSerialization':encode('GajumaruData', Value).
--
-- Use staticFields (not templateFields) for best compactness on static data.
END
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
%% Generated by the Erlang ASN.1 compiler. Version: 5.4.3
%% Purpose: Erlang record definitions for each named and unnamed
%% SEQUENCE and SET, and macro definitions for each value
%% definition in module GajumaruSerialization.
-ifndef(_GAJUMARUSERIALIZATION_HRL_).
-define(_GAJUMARUSERIALIZATION_HRL_, true).
-record('GajumaruData', {
tag,
vsn,
content
}).
-record('TemplateField', {
name = asn1_NOVALUE,
value
}).
-record('KeyValue', {
key,
val
}).
-record('Id', {
type,
value
}).
-record('Account', {
foo,
bar
}).
-record('SignedTx', {
signatures,
transaction
}).
-record('ContractV1', {
sourceHash,
typeInfo,
byteCode
}).
-record('ContractV2', {
sourceHash,
typeInfo,
byteCode,
compilerVersion
}).
-record('ContractV3', {
sourceHash,
typeInfo,
byteCode,
compilerVersion,
payable
}).
-record('TypeInfoV1', {
typeHash,
name,
argType,
outType
}).
-record('TypeInfoV3', {
typeHash,
name,
payable,
argType,
outType
}).
-endif. %% _GAJUMARUSERIALIZATION_HRL_
+27
View File
@@ -0,0 +1,27 @@
-module(detect_demo).
-export([run/0]).
run() ->
application:ensure_all_started(gmserialization),
code:add_path("asn1"),
Sample = {'GajumaruData', 10, 1,
{templateFields, [
{'TemplateField', <<"foo">>, {intValue, 42}},
{'TemplateField', <<"bar">>, {binaryValue, <<"hello">>}}
]}},
{ok, Der} = 'GajumaruSerialization':encode('GajumaruData', Sample),
io:format("DER first byte: ~p (0x~2.16.0B)~n", [binary:at(Der,0), binary:at(Der,0)]),
{ok, _Dec} = 'GajumaruSerialization':decode('GajumaruData', Der),
io:format("DER roundtrip OK~n"),
Legacy = gmser_chain_objects:serialize(account, 1,
[{foo,int},{bar,binary}],
[{foo,42},{bar,<<"hello">>}]),
<<L0>> = binary:part(Legacy,0,1),
<<D0>> = binary:part(Der,0,1),
io:format("Legacy 0x~2.16.0B (>= 0xC0 -> legacy: ~p)~n", [L0, L0 >= 16#C0]),
io:format("DER 0x~2.16.0B (< 0xC0 -> new DER: ~p)~n", [D0, D0 < 16#C0]),
ok.
+114
View File
@@ -0,0 +1,114 @@
-module(size_comparison).
-export([run/0]).
run() ->
application:ensure_all_started(gmserialization),
code:add_path("asn1"),
io:format("=== Size Comparison: Legacy RLP vs DER ===~n~n"),
Compare = fun(Desc, LegacyBin, DerBin) ->
L = byte_size(LegacyBin),
D = byte_size(DerBin),
Overhead = D - L,
Pct = case L of 0 -> 0; _ -> round(Overhead * 100 / L) end,
io:format("~s~n", [Desc]),
io:format(" Legacy: ~p bytes ~w~n", [L, LegacyBin]),
Show = binary:part(DerBin, 0, min(18, D)),
io:format(" DER: ~p bytes ~w~n", [D, Show]),
io:format(" Overhead: +~p bytes (~p%)~n~n", [Overhead, Pct])
end,
%% Case 1: tag+vsn + small int + small binary
T1 = [{foo,int},{bar,binary}],
V1 = [{foo,1},{bar,<<2>>}],
Leg1 = gmser_chain_objects:serialize(account, 1, T1, V1),
DerVal1 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"foo">>, {intValue, 1}},
{'TemplateField', <<"bar">>, {binaryValue, <<2>>}}
]}},
{ok, Der1} = 'GajumaruSerialization':encode('GajumaruData', DerVal1),
Compare("Case 1: tag+vsn + small int + tiny binary (2 fields)", Leg1, Der1),
%% Case 2: zero + empty binary
V2 = [{foo,0},{bar,<<>>}],
Leg2 = gmser_chain_objects:serialize(account, 1, T1, V2),
DerVal2 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"foo">>, {intValue, 0}},
{'TemplateField', <<"bar">>, {binaryValue, <<>>}}
]}},
{ok, Der2} = 'GajumaruSerialization':encode('GajumaruData', DerVal2),
Compare("Case 2: zero int + empty binary", Leg2, Der2),
%% Case 3: list of ints
T3 = [{xs,[int]}],
V3 = [{xs,[1,2,3]}],
Leg3 = gmser_chain_objects:serialize(account, 1, T3, V3),
DerVal3 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"xs">>, {listValue, [
{intValue,1},{intValue,2},{intValue,3}
]}}
]}},
{ok, Der3} = 'GajumaruSerialization':encode('GajumaruData', DerVal3),
Compare("Case 3: list of 3 small ints", Leg3, Der3),
%% Case 4: tuple (int, binary)
T4 = [{p,{int,binary}}],
V4 = [{p,{42,<<"hi">>}}],
Leg4 = gmser_chain_objects:serialize(account, 1, T4, V4),
DerVal4 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"p">>, {tupleValue, [
{intValue,42}, {binaryValue,<<"hi">>}
]}}
]}},
{ok, Der4} = 'GajumaruSerialization':encode('GajumaruData', DerVal4),
Compare("Case 4: fixed tuple (int + 2-byte binary)", Leg4, Der4),
%% Case 5: 256-byte payload
Bin5 = crypto:strong_rand_bytes(256),
T5 = [{data,binary}],
V5 = [{data,Bin5}],
Leg5 = gmser_chain_objects:serialize(account, 1, T5, V5),
DerVal5 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"data">>, {binaryValue, Bin5}}
]}},
{ok, Der5} = 'GajumaruSerialization':encode('GajumaruData', DerVal5),
Compare("Case 5: 256-byte binary payload only", Leg5, Der5),
%% Case 6: Concrete SignedTx style (no field names in DER)
T6 = [{signatures,[binary]},{tx,binary}],
V6 = [{signatures,[<<"sig1">>,<<"sig2">>]},{tx,<<"txbody123">>}],
Leg6 = gmser_chain_objects:serialize(signed_tx, 1, T6, V6),
DerVal6 = {'GajumaruData', 11, 1, {signedTx, {'SignedTx',
[<<"sig1">>, <<"sig2">>], <<"txbody123">>}}},
{ok, Der6} = 'GajumaruSerialization':encode('GajumaruData', DerVal6),
Compare("Case 6: SignedTx-like (concrete DER, no names)", Leg6, Der6),
%% Case 7: 33-byte id-like value
IdBin = <<1, 0:256>>,
T7 = [{owner,binary}],
V7 = [{owner,IdBin}],
Leg7 = gmser_chain_objects:serialize(account, 1, T7, V7),
DerVal7 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"owner">>, {binaryValue, IdBin}}
]}},
{ok, Der7} = 'GajumaruSerialization':encode('GajumaruData', DerVal7),
Compare("Case 7: 33-byte value (id-like)", Leg7, Der7),
%% Case 8: Deeper nesting / more fields
T8 = [{a,int},{b,binary},{c,[int]},{d,{int,int}}],
V8 = [{a,123456},{b,<<"abcdef">>},{c,[10,20,30]},{d,{7,8}}],
Leg8 = gmser_chain_objects:serialize(account, 1, T8, V8),
DerVal8 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"a">>, {intValue, 123456}},
{'TemplateField', <<"b">>, {binaryValue, <<"abcdef">>}},
{'TemplateField', <<"c">>, {listValue, [{intValue,10},{intValue,20},{intValue,30}]}},
{'TemplateField', <<"d">>, {tupleValue, [{intValue,7},{intValue,8}]}}
]}},
{ok, Der8} = 'GajumaruSerialization':encode('GajumaruData', DerVal8),
Compare("Case 8: 4 fields mixed (int, bin, list, tuple)", Leg8, Der8),
io:format("=== Analysis ===~n"),
io:format("Note: Generic templateFields path includes IA5String field names.~n"),
io:format("Concrete types (e.g. SignedTx) avoid name overhead.~n"),
ok.
+267
View File
@@ -0,0 +1,267 @@
-- GajumaruSerialization.asn
--
-- ASN.1 model for the structures serialized by gmserialization.erl
-- (the layer on top of RLP).
--
-- Purpose:
-- * Provide a formal, toolable description of the data for portability
-- (other languages use ASN.1 compilers to get types/parsers).
-- * Define compact canonical wire format using standard ASN.1 techniques
-- (primarily Unaligned PER / UPER with constraints for packing).
-- * Static encoding only (templates from gmserialization).
-- * Legacy RLP compatibility is achieved via separate translation layer
-- (deferred in current focus).
--
-- Detection of legacy vs new:
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
-- This means the first byte is in the range 0xC0 .. 0xFF.
--
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
-- constructed, universal tag) for the outermost GajumaruData (or any
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
-- check reliably distinguishes the two formats.
--
-- Usage in Erlang (after compiling with asn1ct for compact wire):
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [uper]).
-- {ok, Bytes} = 'GajumaruSerialization':encode('GajumaruData', Asn1Value).
-- Bytes is the compact UPER wire format (deterministic with this schema).
-- For legacy RLP, use the model-to-RLP layer instead (see gmser_asn1_rlp).
--
-- Notes on type mapping from gmserialization.erl (static templates):
-- - int (bignum) -> BigInt ::= INTEGER (0..MAX)
-- In practice used for amounts/balances in Pucks (up to 1*10^30).
-- Encoded as non-negative bignum.
-- - New smaller integer types will be added to templates (uint64, uint32, ...)
-- for cases where range is known → enables tighter UPER packing.
-- - binary -> OCTET STRING
-- - bool -> BOOLEAN
-- - id -> Id (SEQUENCE)
-- - [T] -> SEQUENCE OF T
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
-- (static maps carry *values only*, no keys on the wire)
-- - tag + vsn are preserved as the first two fields for compatibility
-- with existing dispatch logic.
--
-- WARNING:
-- Any cryptographic hash or signature computed over a serialized object
-- will change when switching from RLP to DER for that object. Plan a
-- coordinated upgrade.
GajumaruSerialization DEFINITIONS
AUTOMATIC TAGS ::=
BEGIN
EXPORTS ALL;
-- ============================================================
-- Top-level wrapper used for new DER data.
-- This is what a decoder will see first.
-- ============================================================
GajumaruData ::= SEQUENCE {
-- Constrained for better PER packing
tag INTEGER (0..65535),
vsn INTEGER (0..255),
content Content
}
-- Preferred top-level for the compact static wire format.
-- Avoids the extra Content CHOICE tag when the structure is known to be static.
CompactStatic ::= SEQUENCE {
tag INTEGER (0..65535),
vsn INTEGER (0..255),
fields StaticFields
}
-- Content can be a specific structured type (preferred) or a generic
-- representation of a template-driven object.
Content ::= CHOICE {
-- Generic fallback that can represent any [{Field, Type}] template
-- without having a pre-defined SEQUENCE for every object.
templateFields [0] TemplateFields,
-- Static-optimized: no field names on wire (matches legacy static behavior exactly)
-- Preferred for compact wire format of known templates.
staticFields [10] StaticFields,
-- Examples of concrete versioned types (extend as needed)
account [1] Account,
signedTx [2] SignedTx,
contract [3] ContractCode
-- Add more alternatives for other tags from gmser_chain_objects
}
-- ============================================================
-- Generic template-driven representation
-- (useful during transition or for unregistered types)
-- ============================================================
TemplateFields ::= SEQUENCE OF TemplateField
TemplateField ::= SEQUENCE {
-- Field name is included for debuggability / generic processing.
-- In the original static encoding the name is NOT on the wire;
-- only position and type matter. We include it here for convenience.
name IA5String OPTIONAL,
value Value
}
-- Optimized for static wire format: just the values in order, no names.
-- This matches the legacy static encoding where maps/records are positional only.
StaticFields ::= SEQUENCE OF Value
Value ::= CHOICE {
-- "int" in static templates is used for bignums in practice
-- (e.g. balances and amounts in Pucks, up to 1*10^30).
-- We keep it as unbounded non-negative integer.
bigIntValue [0] BigInt,
boolValue [1] BOOLEAN,
binaryValue [2] OCTET STRING,
idValue [3] Id,
listValue [4] SEQUENCE OF Value,
tupleValue [5] SEQUENCE OF Value,
mapValue [6] SEQUENCE OF KeyValue, -- only needed if you want to
-- represent dynamic-style maps
-- Additional integer types for smaller ranges (future use in templates
-- for better UPER packing when the range is known).
uint64Value [7] Uint64,
uint32Value [8] Uint32,
uint128Value [9] Uint128
}
-- Bignum integer (non-negative). Used for the traditional "int" in static
-- templates. Max practical value mentioned: 1*10^30 (≈ 2^100 bits).
BigInt ::= INTEGER (0..MAX)
-- Convenience sized unsigned integer types.
Uint64 ::= INTEGER (0..18446744073709551615)
Uint32 ::= INTEGER (0..4294967295)
Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
KeyValue ::= SEQUENCE {
key Value,
val Value
}
-- ============================================================
-- Common types
-- ============================================================
Id ::= SEQUENCE {
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
-- and the extended account subtype (high bit in legacy).
type INTEGER (0..255),
value OCTET STRING (SIZE (32))
}
-- ============================================================
-- Concrete object examples (derived from usage in the codebase)
-- Add / evolve per version as you introduce new vsns.
-- ============================================================
-- Example: a very simple account-like object used in tests
Account ::= SEQUENCE {
foo INTEGER,
bar OCTET STRING
}
-- Simplified signed transaction
SignedTx ::= SEQUENCE {
signatures SEQUENCE OF OCTET STRING,
transaction OCTET STRING
}
-- Contract code objects (see gmser_contract_code.erl)
-- We model the three versions that exist today.
ContractCode ::= CHOICE {
v1 [0] ContractV1,
v2 [1] ContractV2,
v3 [2] ContractV3
}
ContractV1 ::= SEQUENCE {
sourceHash OCTET STRING,
-- typeInfo is a list of 4-tuples in legacy:
-- {typeHash, name, argType, outType}
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING
}
ContractV2 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING,
compilerVersion OCTET STRING
}
ContractV3 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV3,
byteCode OCTET STRING,
compilerVersion OCTET STRING,
payable BOOLEAN
}
TypeInfoV1 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
argType OCTET STRING,
outType OCTET STRING
}
TypeInfoV3 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
payable BOOLEAN,
argType OCTET STRING,
outType OCTET STRING
}
-- ============================================================
-- Notes for implementers
-- ============================================================
-- 1. Detection (recommended decoder entry point):
--
-- decode(Binary) ->
-- case Binary of
-- <<B, _/binary>> when B >= 16#C0 ->
-- decode_legacy_rlp(Binary); % existing gmser_* path
-- _ ->
-- {ok, Term} =
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
-- Term
-- end.
--
-- This works because:
-- - All current top-level output from serialize() is an RLP list
-- (first byte 0xC0-0xFF).
-- - GajumaruData and the concrete choices above are SEQUENCEs
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
--
-- 2. When you add a new object type or version, prefer adding a
-- concrete SEQUENCE alternative in the Content CHOICE rather than
-- always falling back to TemplateFields. This gives you better
-- validation and generated types.
--
-- 3. For the compact wire format we use UPER (unaligned PER).
-- It is stable/deterministic for a given schema (no extensibility
-- markers on these types, fixed order).
--
-- 4. INTEGER uses ASN.1 PER encoding (canonical for the constraints).
-- This may differ from legacy RLP minimal unsigned; new format
-- will have different hashes (expected when introducing new encoding).
--
-- 5. Dynamic encoding (gmser_dyn) is not in scope here.
--
-- 6. To generate the compact wire:
-- asn1ct:compile(GajumaruSerialization, [uper]).
-- {ok, CompactBytes} = 'GajumaruSerialization':encode('GajumaruData', Value).
--
-- Use staticFields (not templateFields) for best compactness on static data.
END
Binary file not shown.
File diff suppressed because it is too large Load Diff
+82
View File
@@ -0,0 +1,82 @@
%% Generated by the Erlang ASN.1 compiler. Version: 5.4.3
%% Purpose: Erlang record definitions for each named and unnamed
%% SEQUENCE and SET, and macro definitions for each value
%% definition in module GajumaruSerialization.
-ifndef(_GAJUMARUSERIALIZATION_HRL_).
-define(_GAJUMARUSERIALIZATION_HRL_, true).
-record('GajumaruData', {
tag,
vsn,
content
}).
-record('CompactStatic', {
tag,
vsn,
fields
}).
-record('TemplateField', {
name = asn1_NOVALUE,
value
}).
-record('KeyValue', {
key,
val
}).
-record('Id', {
type,
value
}).
-record('Account', {
foo,
bar
}).
-record('SignedTx', {
signatures,
transaction
}).
-record('ContractV1', {
sourceHash,
typeInfo,
byteCode
}).
-record('ContractV2', {
sourceHash,
typeInfo,
byteCode,
compilerVersion
}).
-record('ContractV3', {
sourceHash,
typeInfo,
byteCode,
compilerVersion,
payable
}).
-record('TypeInfoV1', {
typeHash,
name,
argType,
outType
}).
-record('TypeInfoV3', {
typeHash,
name,
payable,
argType,
outType
}).
-endif. %% _GAJUMARUSERIALIZATION_HRL_
+247
View File
@@ -0,0 +1,247 @@
-- GajumaruSerialization.asn
--
-- ASN.1 model for the structures serialized by gmserialization.erl
-- (the layer on top of RLP).
--
-- Purpose:
-- * Provide a formal, toolable description of the data for portability
-- (other languages use ASN.1 compilers to get types/parsers).
-- * Define compact canonical wire format using standard ASN.1 techniques
-- (primarily Unaligned PER / UPER with constraints for packing).
-- * Static encoding only (templates from gmserialization).
-- * Legacy RLP compatibility is achieved via separate translation layer
-- (deferred in current focus).
--
-- Detection of legacy vs new:
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
-- This means the first byte is in the range 0xC0 .. 0xFF.
--
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
-- constructed, universal tag) for the outermost GajumaruData (or any
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
-- check reliably distinguishes the two formats.
--
-- Usage in Erlang (after compiling with asn1ct for compact wire):
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [uper]).
-- {ok, Bytes} = 'GajumaruSerialization':encode('GajumaruData', Asn1Value).
-- Bytes is the compact UPER wire format (deterministic with this schema).
-- For legacy RLP, use the model-to-RLP layer instead (see gmser_asn1_rlp).
--
-- Notes on type mapping from gmserialization.erl:
-- - int -> INTEGER (new data uses canonical DER INTEGER)
-- - binary -> OCTET STRING
-- - bool -> BOOLEAN
-- - id -> Id (SEQUENCE) (cleaner than the legacy packed 33-byte form)
-- - [T] -> SEQUENCE OF T
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
-- (static maps carry *values only*, no keys on the wire)
-- - tag + vsn are preserved as the first two fields for compatibility
-- with existing dispatch logic.
--
-- Legacy integers had strict unsigned-minimal no-leading-zero (except for 0)
-- and RLP-level rules. New DER data does not need to follow those rules.
--
-- WARNING:
-- Any cryptographic hash or signature computed over a serialized object
-- will change when switching from RLP to DER for that object. Plan a
-- coordinated upgrade.
GajumaruSerialization DEFINITIONS
AUTOMATIC TAGS ::=
BEGIN
EXPORTS ALL;
-- ============================================================
-- Top-level wrapper used for new DER data.
-- This is what a decoder will see first.
-- ============================================================
GajumaruData ::= SEQUENCE {
-- Constrained for better PER packing
tag INTEGER (0..65535),
vsn INTEGER (0..255),
content Content
}
-- Preferred top-level for the compact static wire format.
-- Avoids the extra Content CHOICE tag when the structure is known to be static.
CompactStatic ::= SEQUENCE {
tag INTEGER (0..65535),
vsn INTEGER (0..255),
fields StaticFields
}
-- Content can be a specific structured type (preferred) or a generic
-- representation of a template-driven object.
Content ::= CHOICE {
-- Generic fallback that can represent any [{Field, Type}] template
-- without having a pre-defined SEQUENCE for every object.
templateFields [0] TemplateFields,
-- Static-optimized: no field names on wire (matches legacy static behavior exactly)
-- Preferred for compact wire format of known templates.
staticFields [10] StaticFields,
-- Examples of concrete versioned types (extend as needed)
account [1] Account,
signedTx [2] SignedTx,
contract [3] ContractCode
-- Add more alternatives for other tags from gmser_chain_objects
}
-- ============================================================
-- Generic template-driven representation
-- (useful during transition or for unregistered types)
-- ============================================================
TemplateFields ::= SEQUENCE OF TemplateField
TemplateField ::= SEQUENCE {
-- Field name is included for debuggability / generic processing.
-- In the original static encoding the name is NOT on the wire;
-- only position and type matter. We include it here for convenience.
name IA5String OPTIONAL,
value Value
}
-- Optimized for static wire format: just the values in order, no names.
-- This matches the legacy static encoding where maps/records are positional only.
StaticFields ::= SEQUENCE OF Value
Value ::= CHOICE {
intValue [0] INTEGER,
boolValue [1] BOOLEAN,
binaryValue [2] OCTET STRING,
idValue [3] Id,
listValue [4] SEQUENCE OF Value,
tupleValue [5] SEQUENCE OF Value,
mapValue [6] SEQUENCE OF KeyValue -- only needed if you want to
-- represent dynamic-style maps
}
KeyValue ::= SEQUENCE {
key Value,
val Value
}
-- ============================================================
-- Common types
-- ============================================================
Id ::= SEQUENCE {
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
-- and the extended account subtype (high bit in legacy).
type INTEGER (0..255),
value OCTET STRING (SIZE (32))
}
-- ============================================================
-- Concrete object examples (derived from usage in the codebase)
-- Add / evolve per version as you introduce new vsns.
-- ============================================================
-- Example: a very simple account-like object used in tests
Account ::= SEQUENCE {
foo INTEGER,
bar OCTET STRING
}
-- Simplified signed transaction
SignedTx ::= SEQUENCE {
signatures SEQUENCE OF OCTET STRING,
transaction OCTET STRING
}
-- Contract code objects (see gmser_contract_code.erl)
-- We model the three versions that exist today.
ContractCode ::= CHOICE {
v1 [0] ContractV1,
v2 [1] ContractV2,
v3 [2] ContractV3
}
ContractV1 ::= SEQUENCE {
sourceHash OCTET STRING,
-- typeInfo is a list of 4-tuples in legacy:
-- {typeHash, name, argType, outType}
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING
}
ContractV2 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING,
compilerVersion OCTET STRING
}
ContractV3 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV3,
byteCode OCTET STRING,
compilerVersion OCTET STRING,
payable BOOLEAN
}
TypeInfoV1 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
argType OCTET STRING,
outType OCTET STRING
}
TypeInfoV3 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
payable BOOLEAN,
argType OCTET STRING,
outType OCTET STRING
}
-- ============================================================
-- Notes for implementers
-- ============================================================
-- 1. Detection (recommended decoder entry point):
--
-- decode(Binary) ->
-- case Binary of
-- <<B, _/binary>> when B >= 16#C0 ->
-- decode_legacy_rlp(Binary); % existing gmser_* path
-- _ ->
-- {ok, Term} =
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
-- Term
-- end.
--
-- This works because:
-- - All current top-level output from serialize() is an RLP list
-- (first byte 0xC0-0xFF).
-- - GajumaruData and the concrete choices above are SEQUENCEs
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
--
-- 2. When you add a new object type or version, prefer adding a
-- concrete SEQUENCE alternative in the Content CHOICE rather than
-- always falling back to TemplateFields. This gives you better
-- validation and generated types.
--
-- 3. For the compact wire format we use UPER (unaligned PER).
-- It is stable/deterministic for a given schema (no extensibility
-- markers on these types, fixed order).
--
-- 4. INTEGER uses ASN.1 PER encoding (canonical for the constraints).
-- This may differ from legacy RLP minimal unsigned; new format
-- will have different hashes (expected when introducing new encoding).
--
-- 5. Dynamic encoding (gmser_dyn) is not in scope here.
--
-- 6. To generate the compact wire:
-- asn1ct:compile(GajumaruSerialization, [uper]).
-- {ok, CompactBytes} = 'GajumaruSerialization':encode('GajumaruData', Value).
--
-- Use staticFields (not templateFields) for best compactness on static data.
END
Binary file not shown.
File diff suppressed because it is too large Load Diff
+82
View File
@@ -0,0 +1,82 @@
%% Generated by the Erlang ASN.1 compiler. Version: 5.4.3
%% Purpose: Erlang record definitions for each named and unnamed
%% SEQUENCE and SET, and macro definitions for each value
%% definition in module GajumaruSerialization.
-ifndef(_GAJUMARUSERIALIZATION_HRL_).
-define(_GAJUMARUSERIALIZATION_HRL_, true).
-record('GajumaruData', {
tag,
vsn,
content
}).
-record('CompactStatic', {
tag,
vsn,
fields
}).
-record('TemplateField', {
name = asn1_NOVALUE,
value
}).
-record('KeyValue', {
key,
val
}).
-record('Id', {
type,
value
}).
-record('Account', {
foo,
bar
}).
-record('SignedTx', {
signatures,
transaction
}).
-record('ContractV1', {
sourceHash,
typeInfo,
byteCode
}).
-record('ContractV2', {
sourceHash,
typeInfo,
byteCode,
compilerVersion
}).
-record('ContractV3', {
sourceHash,
typeInfo,
byteCode,
compilerVersion,
payable
}).
-record('TypeInfoV1', {
typeHash,
name,
argType,
outType
}).
-record('TypeInfoV3', {
typeHash,
name,
payable,
argType,
outType
}).
-endif. %% _GAJUMARUSERIALIZATION_HRL_
+235
View File
@@ -0,0 +1,235 @@
-- GajumaruSerialization.asn
--
-- ASN.1 model for the structures serialized by gmserialization.erl
-- (the layer on top of RLP).
--
-- Purpose:
-- * Provide a formal, toolable description of the data.
-- * Enable migration from the legacy RLP-based format to DER.
--
-- Detection of legacy vs new:
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
-- This means the first byte is in the range 0xC0 .. 0xFF.
--
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
-- constructed, universal tag) for the outermost GajumaruData (or any
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
-- check reliably distinguishes the two formats.
--
-- Usage in Erlang (after compiling with asn1ct):
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [ber, der]).
-- {ok, Term} = 'GajumaruSerialization':decode('GajumaruData', DerBinary).
--
-- Notes on type mapping from gmserialization.erl:
-- - int -> INTEGER (new data uses canonical DER INTEGER)
-- - binary -> OCTET STRING
-- - bool -> BOOLEAN
-- - id -> Id (SEQUENCE) (cleaner than the legacy packed 33-byte form)
-- - [T] -> SEQUENCE OF T
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
-- (static maps carry *values only*, no keys on the wire)
-- - tag + vsn are preserved as the first two fields for compatibility
-- with existing dispatch logic.
--
-- Legacy integers had strict unsigned-minimal no-leading-zero (except for 0)
-- and RLP-level rules. New DER data does not need to follow those rules.
--
-- WARNING:
-- Any cryptographic hash or signature computed over a serialized object
-- will change when switching from RLP to DER for that object. Plan a
-- coordinated upgrade.
GajumaruSerialization DEFINITIONS
AUTOMATIC TAGS ::=
BEGIN
EXPORTS ALL;
-- ============================================================
-- Top-level wrapper used for new DER data.
-- This is what a decoder will see first.
-- ============================================================
GajumaruData ::= SEQUENCE {
-- Constrained for better PER packing
tag INTEGER (0..65535),
vsn INTEGER (0..255),
content Content
}
-- Content can be a specific structured type (preferred) or a generic
-- representation of a template-driven object.
Content ::= CHOICE {
-- Generic fallback that can represent any [{Field, Type}] template
-- without having a pre-defined SEQUENCE for every object.
templateFields [0] TemplateFields,
-- Static-optimized: no field names on wire (matches legacy static behavior exactly)
-- Preferred for compact wire format of known templates.
staticFields [10] StaticFields,
-- Examples of concrete versioned types (extend as needed)
account [1] Account,
signedTx [2] SignedTx,
contract [3] ContractCode
-- Add more alternatives for other tags from gmser_chain_objects
}
-- ============================================================
-- Generic template-driven representation
-- (useful during transition or for unregistered types)
-- ============================================================
TemplateFields ::= SEQUENCE OF TemplateField
TemplateField ::= SEQUENCE {
-- Field name is included for debuggability / generic processing.
-- In the original static encoding the name is NOT on the wire;
-- only position and type matter. We include it here for convenience.
name IA5String OPTIONAL,
value Value
}
-- Optimized for static wire format: just the values in order, no names.
-- This matches the legacy static encoding where maps/records are positional only.
StaticFields ::= SEQUENCE OF Value
Value ::= CHOICE {
intValue [0] INTEGER,
boolValue [1] BOOLEAN,
binaryValue [2] OCTET STRING,
idValue [3] Id,
listValue [4] SEQUENCE OF Value,
tupleValue [5] SEQUENCE OF Value,
mapValue [6] SEQUENCE OF KeyValue -- only needed if you want to
-- represent dynamic-style maps
}
KeyValue ::= SEQUENCE {
key Value,
val Value
}
-- ============================================================
-- Common types
-- ============================================================
Id ::= SEQUENCE {
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
-- and the extended account subtype (high bit in legacy).
type INTEGER (0..255),
value OCTET STRING (SIZE (32))
}
-- ============================================================
-- Concrete object examples (derived from usage in the codebase)
-- Add / evolve per version as you introduce new vsns.
-- ============================================================
-- Example: a very simple account-like object used in tests
Account ::= SEQUENCE {
foo INTEGER,
bar OCTET STRING
}
-- Simplified signed transaction
SignedTx ::= SEQUENCE {
signatures SEQUENCE OF OCTET STRING,
transaction OCTET STRING
}
-- Contract code objects (see gmser_contract_code.erl)
-- We model the three versions that exist today.
ContractCode ::= CHOICE {
v1 [0] ContractV1,
v2 [1] ContractV2,
v3 [2] ContractV3
}
ContractV1 ::= SEQUENCE {
sourceHash OCTET STRING,
-- typeInfo is a list of 4-tuples in legacy:
-- {typeHash, name, argType, outType}
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING
}
ContractV2 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING,
compilerVersion OCTET STRING
}
ContractV3 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV3,
byteCode OCTET STRING,
compilerVersion OCTET STRING,
payable BOOLEAN
}
TypeInfoV1 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
argType OCTET STRING,
outType OCTET STRING
}
TypeInfoV3 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
payable BOOLEAN,
argType OCTET STRING,
outType OCTET STRING
}
-- ============================================================
-- Notes for implementers
-- ============================================================
-- 1. Detection (recommended decoder entry point):
--
-- decode(Binary) ->
-- case Binary of
-- <<B, _/binary>> when B >= 16#C0 ->
-- decode_legacy_rlp(Binary); % existing gmser_* path
-- _ ->
-- {ok, Term} =
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
-- Term
-- end.
--
-- This works because:
-- - All current top-level output from serialize() is an RLP list
-- (first byte 0xC0-0xFF).
-- - GajumaruData and the concrete choices above are SEQUENCEs
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
--
-- 2. When you add a new object type or version, prefer adding a
-- concrete SEQUENCE alternative in the Content CHOICE rather than
-- always falling back to TemplateFields. This gives you better
-- validation and generated types.
--
-- 3. INTEGER in DER is signed and uses a different minimal encoding
-- than the legacy unsigned big-endian no leading zero form. This is
-- fine for new data.
--
-- 4. If you need to preserve exact legacy integer wire bytes inside
-- the new format (e.g. for some hash preimage reason), you can
-- carry selected integers as OCTET STRING (legacy unsigned minimal bytes).
--
-- 5. Dynamic encoding (gmser_dyn) is intentionally not modeled here
-- in full, because it is runtime-schema driven (type codes 246-255,
-- labels, alt/switch, etc.). You can still use the generic
-- TemplateFields + Value for some dynamic cases, or model specific
-- message schemas as additional CHOICE arms.
--
-- 6. Compile with DER for canonical output:
-- asn1ct:compile(GajumaruSerialization, [der]).
--
-- The ber option is also accepted; der implies the stricter rules.
END
Binary file not shown.
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
%% Generated by the Erlang ASN.1 compiler. Version: 5.4.3
%% Purpose: Erlang record definitions for each named and unnamed
%% SEQUENCE and SET, and macro definitions for each value
%% definition in module GajumaruSerialization.
-ifndef(_GAJUMARUSERIALIZATION_HRL_).
-define(_GAJUMARUSERIALIZATION_HRL_, true).
-record('GajumaruData', {
tag,
vsn,
content
}).
-record('TemplateField', {
name = asn1_NOVALUE,
value
}).
-record('KeyValue', {
key,
val
}).
-record('Id', {
type,
value
}).
-record('Account', {
foo,
bar
}).
-record('SignedTx', {
signatures,
transaction
}).
-record('ContractV1', {
sourceHash,
typeInfo,
byteCode
}).
-record('ContractV2', {
sourceHash,
typeInfo,
byteCode,
compilerVersion
}).
-record('ContractV3', {
sourceHash,
typeInfo,
byteCode,
compilerVersion,
payable
}).
-record('TypeInfoV1', {
typeHash,
name,
argType,
outType
}).
-record('TypeInfoV3', {
typeHash,
name,
payable,
argType,
outType
}).
-endif. %% _GAJUMARUSERIALIZATION_HRL_
+225
View File
@@ -0,0 +1,225 @@
-- GajumaruSerialization.asn
--
-- ASN.1 model for the structures serialized by gmserialization.erl
-- (the layer on top of RLP).
--
-- Purpose:
-- * Provide a formal, toolable description of the data.
-- * Enable migration from the legacy RLP-based format to DER.
--
-- Detection of legacy vs new:
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
-- This means the first byte is in the range 0xC0 .. 0xFF.
--
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
-- constructed, universal tag) for the outermost GajumaruData (or any
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
-- check reliably distinguishes the two formats.
--
-- Usage in Erlang (after compiling with asn1ct):
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [ber, der]).
-- {ok, Term} = 'GajumaruSerialization':decode('GajumaruData', DerBinary).
--
-- Notes on type mapping from gmserialization.erl:
-- - int -> INTEGER (new data uses canonical DER INTEGER)
-- - binary -> OCTET STRING
-- - bool -> BOOLEAN
-- - id -> Id (SEQUENCE) (cleaner than the legacy packed 33-byte form)
-- - [T] -> SEQUENCE OF T
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
-- (static maps carry *values only*, no keys on the wire)
-- - tag + vsn are preserved as the first two fields for compatibility
-- with existing dispatch logic.
--
-- Legacy integers had strict unsigned-minimal no-leading-zero (except for 0)
-- and RLP-level rules. New DER data does not need to follow those rules.
--
-- WARNING:
-- Any cryptographic hash or signature computed over a serialized object
-- will change when switching from RLP to DER for that object. Plan a
-- coordinated upgrade.
GajumaruSerialization DEFINITIONS
AUTOMATIC TAGS ::=
BEGIN
EXPORTS ALL;
-- ============================================================
-- Top-level wrapper used for new DER data.
-- This is what a decoder will see first.
-- ============================================================
GajumaruData ::= SEQUENCE {
tag INTEGER,
vsn INTEGER,
content Content
}
-- Content can be a specific structured type (preferred) or a generic
-- representation of a template-driven object.
Content ::= CHOICE {
-- Generic fallback that can represent any [{Field, Type}] template
-- without having a pre-defined SEQUENCE for every object.
templateFields [0] TemplateFields,
-- Examples of concrete versioned types (extend as needed)
account [1] Account,
signedTx [2] SignedTx,
contract [3] ContractCode
-- Add more alternatives for other tags from gmser_chain_objects
}
-- ============================================================
-- Generic template-driven representation
-- (useful during transition or for unregistered types)
-- ============================================================
TemplateFields ::= SEQUENCE OF TemplateField
TemplateField ::= SEQUENCE {
-- Field name is included for debuggability / generic processing.
-- In the original static encoding the name is NOT on the wire;
-- only position and type matter. We include it here for convenience.
name IA5String OPTIONAL,
value Value
}
Value ::= CHOICE {
intValue [0] INTEGER,
boolValue [1] BOOLEAN,
binaryValue [2] OCTET STRING,
idValue [3] Id,
listValue [4] SEQUENCE OF Value,
tupleValue [5] SEQUENCE OF Value,
mapValue [6] SEQUENCE OF KeyValue -- only needed if you want to
-- represent dynamic-style maps
}
KeyValue ::= SEQUENCE {
key Value,
val Value
}
-- ============================================================
-- Common types
-- ============================================================
Id ::= SEQUENCE {
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
-- and the extended account subtype (high bit in legacy).
type INTEGER (0..255),
value OCTET STRING (SIZE (32))
}
-- ============================================================
-- Concrete object examples (derived from usage in the codebase)
-- Add / evolve per version as you introduce new vsns.
-- ============================================================
-- Example: a very simple account-like object used in tests
Account ::= SEQUENCE {
foo INTEGER,
bar OCTET STRING
}
-- Simplified signed transaction
SignedTx ::= SEQUENCE {
signatures SEQUENCE OF OCTET STRING,
transaction OCTET STRING
}
-- Contract code objects (see gmser_contract_code.erl)
-- We model the three versions that exist today.
ContractCode ::= CHOICE {
v1 [0] ContractV1,
v2 [1] ContractV2,
v3 [2] ContractV3
}
ContractV1 ::= SEQUENCE {
sourceHash OCTET STRING,
-- typeInfo is a list of 4-tuples in legacy:
-- {typeHash, name, argType, outType}
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING
}
ContractV2 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING,
compilerVersion OCTET STRING
}
ContractV3 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV3,
byteCode OCTET STRING,
compilerVersion OCTET STRING,
payable BOOLEAN
}
TypeInfoV1 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
argType OCTET STRING,
outType OCTET STRING
}
TypeInfoV3 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
payable BOOLEAN,
argType OCTET STRING,
outType OCTET STRING
}
-- ============================================================
-- Notes for implementers
-- ============================================================
-- 1. Detection (recommended decoder entry point):
--
-- decode(Binary) ->
-- case Binary of
-- <<B, _/binary>> when B >= 16#C0 ->
-- decode_legacy_rlp(Binary); % existing gmser_* path
-- _ ->
-- {ok, Term} =
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
-- Term
-- end.
--
-- This works because:
-- - All current top-level output from serialize() is an RLP list
-- (first byte 0xC0-0xFF).
-- - GajumaruData and the concrete choices above are SEQUENCEs
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
--
-- 2. When you add a new object type or version, prefer adding a
-- concrete SEQUENCE alternative in the Content CHOICE rather than
-- always falling back to TemplateFields. This gives you better
-- validation and generated types.
--
-- 3. INTEGER in DER is signed and uses a different minimal encoding
-- than the legacy unsigned big-endian no leading zero form. This is
-- fine for new data.
--
-- 4. If you need to preserve exact legacy integer wire bytes inside
-- the new format (e.g. for some hash preimage reason), you can
-- carry selected integers as OCTET STRING (legacy unsigned minimal bytes).
--
-- 5. Dynamic encoding (gmser_dyn) is intentionally not modeled here
-- in full, because it is runtime-schema driven (type codes 246-255,
-- labels, alt/switch, etc.). You can still use the generic
-- TemplateFields + Value for some dynamic cases, or model specific
-- message schemas as additional CHOICE arms.
--
-- 6. Compile with DER for canonical output:
-- asn1ct:compile(GajumaruSerialization, [der]).
--
-- The ber option is also accepted; der implies the stricter rules.
END
Binary file not shown.
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
%% Generated by the Erlang ASN.1 compiler. Version: 5.4.3
%% Purpose: Erlang record definitions for each named and unnamed
%% SEQUENCE and SET, and macro definitions for each value
%% definition in module GajumaruSerialization.
-ifndef(_GAJUMARUSERIALIZATION_HRL_).
-define(_GAJUMARUSERIALIZATION_HRL_, true).
-record('GajumaruData', {
tag,
vsn,
content
}).
-record('TemplateField', {
name = asn1_NOVALUE,
value
}).
-record('KeyValue', {
key,
val
}).
-record('Id', {
type,
value
}).
-record('Account', {
foo,
bar
}).
-record('SignedTx', {
signatures,
transaction
}).
-record('ContractV1', {
sourceHash,
typeInfo,
byteCode
}).
-record('ContractV2', {
sourceHash,
typeInfo,
byteCode,
compilerVersion
}).
-record('ContractV3', {
sourceHash,
typeInfo,
byteCode,
compilerVersion,
payable
}).
-record('TypeInfoV1', {
typeHash,
name,
argType,
outType
}).
-record('TypeInfoV3', {
typeHash,
name,
payable,
argType,
outType
}).
-endif. %% _GAJUMARUSERIALIZATION_HRL_
+225
View File
@@ -0,0 +1,225 @@
-- GajumaruSerialization.asn
--
-- ASN.1 model for the structures serialized by gmserialization.erl
-- (the layer on top of RLP).
--
-- Purpose:
-- * Provide a formal, toolable description of the data.
-- * Enable migration from the legacy RLP-based format to DER.
--
-- Detection of legacy vs new:
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
-- This means the first byte is in the range 0xC0 .. 0xFF.
--
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
-- constructed, universal tag) for the outermost GajumaruData (or any
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
-- check reliably distinguishes the two formats.
--
-- Usage in Erlang (after compiling with asn1ct):
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [ber, der]).
-- {ok, Term} = 'GajumaruSerialization':decode('GajumaruData', DerBinary).
--
-- Notes on type mapping from gmserialization.erl:
-- - int -> INTEGER (new data uses canonical DER INTEGER)
-- - binary -> OCTET STRING
-- - bool -> BOOLEAN
-- - id -> Id (SEQUENCE) (cleaner than the legacy packed 33-byte form)
-- - [T] -> SEQUENCE OF T
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
-- (static maps carry *values only*, no keys on the wire)
-- - tag + vsn are preserved as the first two fields for compatibility
-- with existing dispatch logic.
--
-- Legacy integers had strict unsigned-minimal no-leading-zero (except for 0)
-- and RLP-level rules. New DER data does not need to follow those rules.
--
-- WARNING:
-- Any cryptographic hash or signature computed over a serialized object
-- will change when switching from RLP to DER for that object. Plan a
-- coordinated upgrade.
GajumaruSerialization DEFINITIONS
AUTOMATIC TAGS ::=
BEGIN
EXPORTS ALL;
-- ============================================================
-- Top-level wrapper used for new DER data.
-- This is what a decoder will see first.
-- ============================================================
GajumaruData ::= SEQUENCE {
tag INTEGER,
vsn INTEGER,
content Content
}
-- Content can be a specific structured type (preferred) or a generic
-- representation of a template-driven object.
Content ::= CHOICE {
-- Generic fallback that can represent any [{Field, Type}] template
-- without having a pre-defined SEQUENCE for every object.
templateFields [0] TemplateFields,
-- Examples of concrete versioned types (extend as needed)
account [1] Account,
signedTx [2] SignedTx,
contract [3] ContractCode
-- Add more alternatives for other tags from gmser_chain_objects
}
-- ============================================================
-- Generic template-driven representation
-- (useful during transition or for unregistered types)
-- ============================================================
TemplateFields ::= SEQUENCE OF TemplateField
TemplateField ::= SEQUENCE {
-- Field name is included for debuggability / generic processing.
-- In the original static encoding the name is NOT on the wire;
-- only position and type matter. We include it here for convenience.
name IA5String OPTIONAL,
value Value
}
Value ::= CHOICE {
intValue [0] INTEGER,
boolValue [1] BOOLEAN,
binaryValue [2] OCTET STRING,
idValue [3] Id,
listValue [4] SEQUENCE OF Value,
tupleValue [5] SEQUENCE OF Value,
mapValue [6] SEQUENCE OF KeyValue -- only needed if you want to
-- represent dynamic-style maps
}
KeyValue ::= SEQUENCE {
key Value,
val Value
}
-- ============================================================
-- Common types
-- ============================================================
Id ::= SEQUENCE {
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
-- and the extended account subtype (high bit in legacy).
type INTEGER (0..255),
value OCTET STRING (SIZE (32))
}
-- ============================================================
-- Concrete object examples (derived from usage in the codebase)
-- Add / evolve per version as you introduce new vsns.
-- ============================================================
-- Example: a very simple account-like object used in tests
Account ::= SEQUENCE {
foo INTEGER,
bar OCTET STRING
}
-- Simplified signed transaction
SignedTx ::= SEQUENCE {
signatures SEQUENCE OF OCTET STRING,
transaction OCTET STRING
}
-- Contract code objects (see gmser_contract_code.erl)
-- We model the three versions that exist today.
ContractCode ::= CHOICE {
v1 [0] ContractV1,
v2 [1] ContractV2,
v3 [2] ContractV3
}
ContractV1 ::= SEQUENCE {
sourceHash OCTET STRING,
-- typeInfo is a list of 4-tuples in legacy:
-- {typeHash, name, argType, outType}
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING
}
ContractV2 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV1,
byteCode OCTET STRING,
compilerVersion OCTET STRING
}
ContractV3 ::= SEQUENCE {
sourceHash OCTET STRING,
typeInfo SEQUENCE OF TypeInfoV3,
byteCode OCTET STRING,
compilerVersion OCTET STRING,
payable BOOLEAN
}
TypeInfoV1 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
argType OCTET STRING,
outType OCTET STRING
}
TypeInfoV3 ::= SEQUENCE {
typeHash OCTET STRING,
name OCTET STRING,
payable BOOLEAN,
argType OCTET STRING,
outType OCTET STRING
}
-- ============================================================
-- Notes for implementers
-- ============================================================
-- 1. Detection (recommended decoder entry point):
--
-- decode(Binary) ->
-- case Binary of
-- <<B, _/binary>> when B >= 16#C0 ->
-- decode_legacy_rlp(Binary); % existing gmser_* path
-- _ ->
-- {ok, Term} =
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
-- Term
-- end.
--
-- This works because:
-- - All current top-level output from serialize() is an RLP list
-- (first byte 0xC0-0xFF).
-- - GajumaruData and the concrete choices above are SEQUENCEs
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
--
-- 2. When you add a new object type or version, prefer adding a
-- concrete SEQUENCE alternative in the Content CHOICE rather than
-- always falling back to TemplateFields. This gives you better
-- validation and generated types.
--
-- 3. INTEGER in DER is signed and uses a different minimal encoding
-- than the legacy unsigned big-endian no leading zero form. This is
-- fine for new data.
--
-- 4. If you need to preserve exact legacy integer wire bytes inside
-- the new format (e.g. for some hash preimage reason), you can
-- carry selected integers as OCTET STRING (legacy unsigned minimal bytes).
--
-- 5. Dynamic encoding (gmser_dyn) is intentionally not modeled here
-- in full, because it is runtime-schema driven (type codes 246-255,
-- labels, alt/switch, etc.). You can still use the generic
-- TemplateFields + Value for some dynamic cases, or model specific
-- message schemas as additional CHOICE arms.
--
-- 6. Compile with DER for canonical output:
-- asn1ct:compile(GajumaruSerialization, [der]).
--
-- The ber option is also accepted; der implies the stricter rules.
END
Binary file not shown.
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
%% Generated by the Erlang ASN.1 compiler. Version: 5.4.3
%% Purpose: Erlang record definitions for each named and unnamed
%% SEQUENCE and SET, and macro definitions for each value
%% definition in module GajumaruSerialization.
-ifndef(_GAJUMARUSERIALIZATION_HRL_).
-define(_GAJUMARUSERIALIZATION_HRL_, true).
-record('GajumaruData', {
tag,
vsn,
content
}).
-record('TemplateField', {
name = asn1_NOVALUE,
value
}).
-record('KeyValue', {
key,
val
}).
-record('Id', {
type,
value
}).
-record('Account', {
foo,
bar
}).
-record('SignedTx', {
signatures,
transaction
}).
-record('ContractV1', {
sourceHash,
typeInfo,
byteCode
}).
-record('ContractV2', {
sourceHash,
typeInfo,
byteCode,
compilerVersion
}).
-record('ContractV3', {
sourceHash,
typeInfo,
byteCode,
compilerVersion,
payable
}).
-record('TypeInfoV1', {
typeHash,
name,
argType,
outType
}).
-record('TypeInfoV3', {
typeHash,
name,
payable,
argType,
outType
}).
-endif. %% _GAJUMARUSERIALIZATION_HRL_
+70
View File
@@ -0,0 +1,70 @@
# ASN.1 for Static Serialization - Compact Wire Format
## Goal
Use portable ASN.1 techniques to produce the most compact *deterministic* (stable/idempotent for hashing) wire format for gmserialization **static** encoding, based on existing templates.
Focus is on the wire format itself (not RLP translation for legacy, which is deferred).
## Approach
- The `asn1/GajumaruSerialization.asn` is the single source of truth (abstract syntax).
- Use **Unaligned PER (UPER)** as the standard compact canonical encoding rule provided by the ASN.1 framework.
- Portable across languages/tools that support ASN.1 UPER.
- Deterministic for a fixed schema (no extensibility, consistent packing).
- Optimize the schema for packing:
- Constrain INTEGER ranges.
- Provide `staticFields` (SEQUENCE OF Value) -- no field names (names are never on the wire for static case).
- Provide `CompactStatic` top-level type to avoid unnecessary CHOICE overhead for the common static path.
- Concrete SEQUENCEs for well-known objects (SignedTx, ContractV* etc.) when possible.
- Encode with the generated ASN.1 module: ` 'GajumaruSerialization':encode('CompactStatic', Value) `.
## Results (example sizes)
Using current optimized UPER:
- Tiny object (tag/vsn + int + 1-byte bin): 9 bytes (legacy RLP = 5)
- List of 3 ints: 13 bytes (legacy = 7)
- Signed tx example: 11 bytes (legacy = 7)
- 256-byte payload: ~263 bytes (legacy ~264) -- matches or slightly better
PER/UPER overhead is mainly the structural tags for the generic case. Concrete types and `staticFields` + `CompactStatic` minimize it.
Compared to DER (previous orientation): dramatically better (e.g. tiny case was ~36B in DER).
## Usage in Erlang (for the compact format)
```erlang
% Build value according to schema (using staticFields for best compactness)
Value = {'CompactStatic', Tag, Vsn, [
{'intValue', 42},
{'binaryValue', <<"data">>}
% ...
]},
{ok, CompactBytes} = 'GajumaruSerialization':encode('CompactStatic', Value).
```
Compile the schema with:
```
asn1ct:compile("GajumaruSerialization.asn", [uper]).
```
## Schema Notes
- `staticFields` should be used for generic static templates (mirrors legacy positional encoding).
- Concrete types (e.g. `signedTx`) are preferred when the structure is fixed.
- `TemplateFields` (with names) is kept for debug/transition but not optimal for wire size.
- The model directly reflects the static `template()` types from `gmserialization.erl`.
## Portability
Any language with an ASN.1 UPER codec can produce and consume the exact same bytes by using the schema and the same value construction rules.
## Stability
- UPER encoding of this schema is stable (tested roundtrip + re-encode identical).
- No random/padding choices.
- Same input value always produces identical bytes.
## Limitations / Future
- For very small objects, hand-crafted RLP is still smaller because it has almost no structural overhead.
- If an even more compact custom encoding is desired while keeping the model, a custom "encoding rule" can be implemented driven by the schema (similar to how the RLP layer works, but targeting a new bit-packed format).
- Dynamic encoder (gmser_dyn) is out of scope.
See also: `asn1/GajumaruSerialization.asn`, `asn1_compact/`, tests in `src/gmser_asn1_rlp.erl` (for value shapes), `doc/static.md`.
+2 -1
View File
@@ -59,7 +59,8 @@ The template 'language' is defined by these types:
```erlang
-type template() :: [{field_name(), type()}].
-type field_name() :: atom().
-type type() :: 'int'
-type type() :: 'int' % bignum (non-negative, for amounts etc. up to 10^30 Pucks)
| 'uint128' | 'uint64' | 'uint32' | 'uint16' | 'uint8'
| 'bool'
| 'binary'
| 'id' %% As defined in aec_id.erl
+1 -1
View File
@@ -1,6 +1,6 @@
{application,gmserialization,
[{description,"Serialization of data for the Gajumaru"},
{vsn,"0.1.2"},
{vsn,"0.2.0"},
{registered,[]},
{applications,[kernel,stdlib,crypto,base58]},
{env,[]},
+8 -1
View File
@@ -7,4 +7,11 @@
{enacl,
{git,
"https://git.qpq.swiss/QPQ-AG/enacl.git",
{ref, "4eb7ec70084ba7c87b1af8797c4c4e90c84f95a2"}}}]}.
{ref, "4eb7ec70084ba7c87b1af8797c4c4e90c84f95a2"}}},
{eblake2, "1.0.0"}
]}.
{dialyzer,
[ {plt_apps, all_deps},
{base_plt_apps, [erts, kernel, stdlib, enacl, base58, eblake2]}
]}.
+9 -1
View File
@@ -1,8 +1,16 @@
{"1.2.0",
[{<<"base58">>,
{git,"https://git.qpq.swiss/QPQ-AG/erl-base58.git",
{ref,"e6aa62eeae3d4388311401f06e4b939bf4e94b9c"}},
0},
{<<"eblake2">>,{pkg,<<"eblake2">>,<<"1.0.0">>},0},
{<<"enacl">>,
{git,"https://git.qpq.swiss/QPQ-AG/enacl.git",
{ref,"4eb7ec70084ba7c87b1af8797c4c4e90c84f95a2"}},
0}].
0}]}.
[
{pkg_hash,[
{<<"eblake2">>, <<"EC8AD20E438AAB3F2E8D5D118C366A0754219195F8A0F536587440F8F9BCF2EF">>}]},
{pkg_hash_ext,[
{<<"eblake2">>, <<"3C4D300A91845B25D501929A26AC2E6F7157480846FAB2347A4C11AE52E08A99">>}]}
].
+93 -6
View File
@@ -6,14 +6,20 @@
%%% @end
%%%-------------------------------------------------------------------
-module(gmser_api_encoder).
-vsn("0.1.2").
-vsn("0.2.0").
-export([encode/2,
decode/1,
safe_decode/2,
byte_size_for_type/1]).
-export_type([encoded/0]).
-export([encode_keypair/1,
safe_decode_keypair/1]).
-export([unsafe_encode/2]). %% Encode without size checks
-export_type([encoded/0,
known_type/0]).
-type known_type() :: key_block_hash
| micro_block_hash
@@ -30,6 +36,7 @@
| transaction
| tx_hash
| account_pubkey
| {account_pubkey, 0..6}
| account_seckey
| associate_chain
| entry
@@ -38,6 +45,7 @@
| native_token
| commitment
| peer_pubkey
| hash
| state
| poi
| state_trees
@@ -51,14 +59,70 @@
-type payload() :: binary().
-type encoded() :: binary().
-type keypair() :: #{public := <<_:(32*8)>>, secret := <<_:(64*8)>>}.
-type encoded_keypair() :: #{binary() => binary()}.
-export_type([ keypair/0
, encoded_keypair/0 ]).
-define(BASE58, 1).
-define(BASE64, 2).
-spec encode_keypair(keypair()) -> encoded_keypair().
encode_keypair(#{public := Pub, secret := Sec}) ->
case Sec of
<<Seed:32/binary, Pub1:32/binary>> when Pub1 =:= Pub ->
#{ <<"pub">> => encode(account_pubkey, Pub)
, <<"priv">> => encode(account_seckey, Seed) };
_ ->
erlang:error(invalid_keypair)
end.
-spec safe_decode_keypair(encoded_keypair()) -> {'ok', keypair()} | {'error', atom()}.
safe_decode_keypair(#{<<"pub">> := EncPub, <<"priv">> := EncPriv}) ->
case safe_decode(account_pubkey, EncPub) of
{ok, Pub} ->
case safe_decode(account_seckey, EncPriv) of
{ok, Seed} when byte_size(Seed) =:= 32 ->
case enacl:sign_seed_keypair(Seed) of
#{public := Pub, secret := _} = KP ->
{ok, KP};
_ ->
{error, illegal_encoding}
end;
{ok, <<Seed:32/binary, Pub:32/binary>>} ->
case enacl:sign_seed_keypair(Seed) of
#{public := Pub} = KP ->
{ok, KP};
_ ->
{error, illegal_encoding}
end;
{ok, _} ->
{error, illegal_encoding};
{error, _} = Error1 ->
Error1
end;
Error ->
Error
end.
-spec encode(known_type(), payload() | gmser_id:id()) -> encoded().
encode(id_hash, Payload) ->
{IdType, Val} = gmser_id:specialize(Payload),
encode(id2type(IdType), Val);
case gmser_id:to_map(Payload) of
#{type := account, subtype := SubT, value := Val} ->
encode({account_pubkey, SubT}, Val);
#{type := IdType, value := Val} ->
encode(id2type(IdType), Val)
end;
encode(Type, Payload) ->
case type_size_check(Type, Payload) of
ok ->
unsafe_encode(Type, Payload);
{error, Reason} ->
erlang:error(Reason)
end.
unsafe_encode(Type, Payload) ->
Pfx = type2pfx(Type),
Enc = case type2enc(Type) of
?BASE58 -> base58_check(Payload);
@@ -66,6 +130,7 @@ encode(Type, Payload) ->
end,
<<Pfx/binary, "_", Enc/binary>>.
-spec decode(binary()) -> {known_type(), payload()}.
decode(Bin0) ->
case split(Bin0) of
@@ -81,6 +146,13 @@ decode(Bin0) ->
erlang:error(missing_prefix)
end.
type_size_check(account_seckey, Bin) ->
case byte_size(Bin) of
Sz when Sz =:= 32; Sz =:= 64 ->
ok;
_ ->
{error, incorrect_size}
end;
type_size_check(Type, Bin) ->
case byte_size_for_type(Type) of
not_applicable -> ok;
@@ -170,17 +242,16 @@ id2type(associate_chain) -> associate_chain;
id2type(channel) -> channel;
id2type(commitment) -> commitment;
id2type(contract) -> contract_pubkey;
id2type(contract_source) -> contract_source;
id2type(name) -> name;
id2type(native_token) -> native_token;
id2type(entry) -> entry.
type2id({account_pubkey, SubT}) -> {account, SubT};
type2id(account_pubkey) -> account;
type2id(associate_chain) -> associate_chain;
type2id(channel) -> channel;
type2id(commitment) -> commitment;
type2id(contract_pubkey) -> contract;
type2id(contract_source) -> contract_source;
type2id(name) -> name;
type2id(native_token) -> native_token;
type2id(entry) -> entry.
@@ -199,6 +270,7 @@ type2enc(contract_store_value) -> ?BASE64;
type2enc(contract_source) -> ?BASE64;
type2enc(transaction) -> ?BASE64;
type2enc(tx_hash) -> ?BASE58;
type2enc({account_pubkey, _}) -> ?BASE58;
type2enc(account_pubkey) -> ?BASE58;
type2enc(account_seckey) -> ?BASE58;
type2enc(associate_chain) -> ?BASE58;
@@ -231,6 +303,13 @@ type2pfx(contract_store_value) -> <<"cv">>;
type2pfx(contract_source) -> <<"cx">>;
type2pfx(transaction) -> <<"tx">>;
type2pfx(tx_hash) -> <<"th">>;
type2pfx({account_pubkey,0}) -> <<"a0">>;
type2pfx({account_pubkey,1}) -> <<"a1">>;
type2pfx({account_pubkey,2}) -> <<"a2">>;
type2pfx({account_pubkey,3}) -> <<"a3">>;
type2pfx({account_pubkey,4}) -> <<"a4">>;
type2pfx({account_pubkey,5}) -> <<"a5">>;
type2pfx({account_pubkey,6}) -> <<"a6">>;
type2pfx(account_pubkey) -> <<"ak">>;
type2pfx(account_seckey) -> <<"sk">>;
type2pfx(associate_chain) -> <<"ac">>;
@@ -262,6 +341,13 @@ pfx2type(<<"ct">>) -> contract_pubkey;
pfx2type(<<"cx">>) -> contract_source;
pfx2type(<<"tx">>) -> transaction;
pfx2type(<<"th">>) -> tx_hash;
pfx2type(<<"a0">>) -> {account_pubkey, 0};
pfx2type(<<"a1">>) -> {account_pubkey, 1};
pfx2type(<<"a2">>) -> {account_pubkey, 2};
pfx2type(<<"a3">>) -> {account_pubkey, 3};
pfx2type(<<"a4">>) -> {account_pubkey, 4};
pfx2type(<<"a5">>) -> {account_pubkey, 5};
pfx2type(<<"a6">>) -> {account_pubkey, 6};
pfx2type(<<"ak">>) -> account_pubkey;
pfx2type(<<"sk">>) -> account_seckey;
pfx2type(<<"ac">>) -> associate_chain;
@@ -296,6 +382,7 @@ byte_size_for_type(contract_source) -> not_applicable;
byte_size_for_type(transaction) -> not_applicable;
byte_size_for_type(tx_hash) -> 32;
byte_size_for_type(account_pubkey) -> 32;
byte_size_for_type({account_pubkey, _}) -> 32;
byte_size_for_type(account_seckey) -> 32;
byte_size_for_type(associate_chain) -> 32;
byte_size_for_type(signature) -> 64;
+315
View File
@@ -0,0 +1,315 @@
%%%-------------------------------------------------------------------
%%% @copyright (C) 2026, QPQ AG (experiment)
%%% @doc
%%% Thin RLP production layer driven by ASN.1-shaped values.
%%%
%%% This module implements a "thin translation" from structures that
%%% mirror the ASN.1 definitions in asn1/GajumaruSerialization.asn
%%% to the exact same RLP wire format produced by the legacy
%%% gmserialization + gmser_rlp + gmser_chain_objects stack.
%%%
%%% Goal:
%%% - Use ASN.1 as a formal, multi-language-friendly schema.
%%% - Keep the compact legacy RLP on the wire (no DER bloat).
%%% - Provide a reference implementation that other languages can
%%% port (types from ASN.1 compiler + this thin RLP emitter).
%%%
%%% The layer does NOT use the ASN.1 BER/DER codec at runtime for
%%% the wire format. It walks ASN.1-like Erlang terms and emits
%%% RLP using the same rules as gmserialization:encode_field/2
%%% and gmser_rlp:encode/1.
%%%
%%% Supported shapes (matching the ASN.1 value notation):
%%% {'GajumaruData', Tag, Vsn, Content}
%%% Content is a CHOICE:
%%% {templateFields, [ {'TemplateField', Name, Value}, ... ]}
%%% {signedTx, {'SignedTx', Sigs, Tx}}
%%% {account, {'Account', Foo, Bar}}
%%% ...
%%% Value is one of:
%%% {bigIntValue, integer()} % bignum (the original "int" for Pucks etc.)
%%% {uint128Value, integer()}
%%% {uint64Value, integer()}
%%% {uint32Value, integer()}
%%% {uint16Value, integer()}
%%% {uint8Value, integer()}
%%% {binaryValue, binary()}
%%% {boolValue, boolean()}
%%% {listValue, [Value]}
%%% {tupleValue, [Value]} % or tuple, both accepted
%%% {idValue, ...} % basic support
%%%
%%% For templateFields (used for generic/static equivalence), field
%%% *names* are ignored on the wire (matching legacy static behavior
%%% where only values are sent in template order).
%%%
%%% Equivalence with legacy is the primary contract of this module.
%%% @end
%%%-------------------------------------------------------------------
-module(gmser_asn1_rlp).
-vsn("0.1.0-experiment").
-export([encode/1]).
%% For tests and other-language ports, these helpers are useful
-export([encode_basic/2,
encode_asn1_value/1]).
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
-endif.
%%%===================================================================
%%% API
%%%===================================================================
-spec encode(term()) -> binary().
encode({'GajumaruData', Tag, Vsn, Content}) ->
TagB = encode_basic(int, Tag),
VsnB = encode_basic(int, Vsn),
Payload = encode_content(Content),
gmser_rlp:encode([TagB, VsnB | Payload]);
encode(Other) ->
error({unsupported_asn1_top_level, Other}).
%%%===================================================================
%%% Internal: content (the CHOICE after tag/vsn)
%%%===================================================================
encode_content({templateFields, FieldList}) when is_list(FieldList) ->
%% For wire compatibility with legacy static serialization we
%% emit only the values (in order). Names are not sent on the wire.
[encode_asn1_value(Val) || {_, _Name, Val} <- FieldList];
encode_content({signedTx, {'SignedTx', Sigs, Tx}}) ->
SigsEnc = [encode_basic(binary, S) || S <- Sigs],
TxEnc = encode_basic(binary, Tx),
[SigsEnc, TxEnc];
encode_content({account, {'Account', Foo, Bar}}) ->
[encode_basic(int, Foo), encode_basic(binary, Bar)];
encode_content({contract, Contract}) ->
encode_contract(Contract);
encode_content(Other) ->
error({unsupported_content_choice, Other}).
encode_contract({v1, {'ContractV1', Hash, TypeInfo, ByteCode}}) ->
[encode_basic(binary, Hash),
[encode_type_info_v1(TI) || TI <- TypeInfo],
encode_basic(binary, ByteCode)];
encode_contract({v2, {'ContractV2', Hash, TypeInfo, ByteCode, CompilerVsn}}) ->
[encode_basic(binary, Hash),
[encode_type_info_v1(TI) || TI <- TypeInfo],
encode_basic(binary, ByteCode),
encode_basic(binary, CompilerVsn)];
encode_contract({v3, {'ContractV3', Hash, TypeInfo, ByteCode, CompilerVsn, Payable}}) ->
[encode_basic(binary, Hash),
[encode_type_info_v3(TI) || TI <- TypeInfo],
encode_basic(binary, ByteCode),
encode_basic(binary, CompilerVsn),
encode_basic(bool, Payable)].
encode_type_info_v1({_TypeInfoV1, H, N, A, O}) ->
[encode_basic(binary, H),
encode_basic(binary, N),
encode_basic(binary, A),
encode_basic(binary, O)];
encode_type_info_v1(T) when is_tuple(T), tuple_size(T) =:= 4 ->
%% Accept plain 4-tuple as well for convenience
[encode_basic(binary, element(I, T)) || I <- lists:seq(1,4)].
encode_type_info_v3({_TypeInfoV3, H, N, P, A, O}) ->
[encode_basic(binary, H),
encode_basic(binary, N),
encode_basic(bool, P),
encode_basic(binary, A),
encode_basic(binary, O)];
encode_type_info_v3(T) when is_tuple(T), tuple_size(T) =:= 5 ->
%% TypeInfoV3 layout: binary, binary, bool, binary, binary
[encode_basic(binary, element(1,T)),
encode_basic(binary, element(2,T)),
encode_basic(bool, element(3,T)),
encode_basic(binary, element(4,T)),
encode_basic(binary, element(5,T))].
%%%===================================================================
%%% Value encoding (recursive, mirrors ASN.1 Value CHOICE)
%%%===================================================================
-spec encode_asn1_value(term()) -> binary() | [term()].
encode_asn1_value({bigIntValue, I}) -> encode_basic(int, I);
encode_asn1_value({uint128Value, I}) -> encode_basic(int, I);
encode_asn1_value({uint64Value, I}) -> encode_basic(int, I);
encode_asn1_value({uint32Value, I}) -> encode_basic(int, I);
encode_asn1_value({uint16Value, I}) -> encode_basic(int, I);
encode_asn1_value({uint8Value, I}) -> encode_basic(int, I);
encode_asn1_value({binaryValue, B}) -> encode_basic(binary, B);
encode_asn1_value({boolValue, B}) -> encode_basic(bool, B);
encode_asn1_value({listValue, L}) when is_list(L) ->
[encode_asn1_value(E) || E <- L];
encode_asn1_value({tupleValue, T}) when is_list(T) ->
[encode_asn1_value(E) || E <- T];
encode_asn1_value({tupleValue, T}) when is_tuple(T) ->
[encode_asn1_value(E) || E <- tuple_to_list(T)];
encode_asn1_value({idValue, {'Id', Type, Val}}) ->
%% Basic support: encode as the legacy 33-byte id form if possible,
%% otherwise fall back to treating the value as binary.
try
Id = gmser_id:create(decode_id_tag(Type), Val),
gmser_id:encode(Id)
catch _:_ ->
encode_basic(binary, Val)
end;
encode_asn1_value({idValue, Bin}) when is_binary(Bin) ->
%% Convenience: bare 33-byte id value
encode_basic(binary, Bin);
encode_asn1_value(Other) ->
error({unsupported_asn1_value, Other}).
decode_id_tag(1) -> account;
decode_id_tag(2) -> name;
decode_id_tag(3) -> commitment;
decode_id_tag(5) -> contract;
decode_id_tag(6) -> channel;
decode_id_tag(7) -> associate_chain;
decode_id_tag(8) -> native_token;
decode_id_tag(9) -> entry;
decode_id_tag(T) when is_integer(T) -> error({unknown_id_tag, T}).
%%%===================================================================
%%% Basic encoders matching gmserialization rules
%%%===================================================================
-spec encode_basic(atom(), term()) -> binary().
encode_basic(int, X) when is_integer(X), X >= 0 ->
binary:encode_unsigned(X);
encode_basic(binary, X) when is_binary(X) ->
X;
encode_basic(bool, true) -> <<1:8>>;
encode_basic(bool, false) -> <<0:8>>;
encode_basic(id, Val) ->
try gmser_id:encode(Val)
catch _:_ -> error({illegal, id, Val})
end;
encode_basic(Type, Val) ->
error({unsupported_basic_type, Type, Val}).
%%%===================================================================
%%% EUnit equivalence tests
%%%===================================================================
-ifdef(TEST).
%% These tests assert that encoding an ASN.1-shaped value produces
%% *exactly* the same bytes as the legacy gmserialization stack.
%% This is the key property for a thin RLP production layer.
equivalence_simple_fields_test() ->
T = [{foo, uint32}, {bar, binary}],
V = [{foo, 1}, {bar, <<2>>}],
Legacy = gmser_chain_objects:serialize(account, 1, T, V),
Asn1 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"foo">>, {uint32Value, 1}},
{'TemplateField', <<"bar">>, {binaryValue, <<2>>}}
]}},
New = encode(Asn1),
?assertEqual(Legacy, New),
%% Also check we can roundtrip via legacy decoder
Dec = gmser_chain_objects:deserialize(account, 1, T, New),
?assertEqual(V, Dec).
equivalence_zero_and_empty_test() ->
T = [{foo, int}, {bar, binary}],
V = [{foo, 0}, {bar, <<>>}],
Legacy = gmser_chain_objects:serialize(account, 1, T, V),
Asn1 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"foo">>, {bigIntValue, 0}},
{'TemplateField', <<"bar">>, {binaryValue, <<>>}}
]}},
?assertEqual(Legacy, encode(Asn1)).
equivalence_list_field_test() ->
T = [{xs, [int]}],
V = [{xs, [1,2,3]}],
Legacy = gmser_chain_objects:serialize(account, 1, T, V),
Asn1 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"xs">>, {listValue, [
{bigIntValue, 1}, {bigIntValue, 2}, {bigIntValue, 3}
]}}
]}},
?assertEqual(Legacy, encode(Asn1)).
equivalence_tuple_field_test() ->
T = [{p, {int, binary}}],
V = [{p, {42, <<"hi">>}}],
Legacy = gmser_chain_objects:serialize(account, 1, T, V),
Asn1 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"p">>, {tupleValue, [
{bigIntValue, 42}, {binaryValue, <<"hi">>}
]}}
]}},
?assertEqual(Legacy, encode(Asn1)).
equivalence_signed_tx_concrete_test() ->
T = [{signatures, [binary]}, {tx, binary}],
V = [{signatures, [<<"sig1">>, <<"sig2">>]}, {tx, <<"txbody123">>}],
Legacy = gmser_chain_objects:serialize(signed_tx, 1, T, V),
Asn1 = {'GajumaruData', 11, 1, {signedTx, {'SignedTx',
[<<"sig1">>, <<"sig2">>], <<"txbody123">>}}},
?assertEqual(Legacy, encode(Asn1)).
equivalence_list_of_tuples_test() ->
%% Corresponds to type_info style: list of 4-tuples
T = [{type_info, [{binary, binary, binary, binary}]}],
V = [{type_info, [
{<<"h1">>, <<"n1">>, <<"a1">>, <<"o1">>},
{<<"h2">>, <<"n2">>, <<"a2">>, <<"o2">>}
]}],
Legacy = gmser_chain_objects:serialize(account, 1, T, V),
Asn1 = {'GajumaruData', 10, 1, {templateFields, [
{'TemplateField', <<"type_info">>, {listValue, [
{tupleValue, [{binaryValue,<<"h1">>},{binaryValue,<<"n1">>},
{binaryValue,<<"a1">>},{binaryValue,<<"o1">>}]},
{tupleValue, [{binaryValue,<<"h2">>},{binaryValue,<<"n2">>},
{binaryValue,<<"a2">>},{binaryValue,<<"o2">>}]}
]}}
]}},
?assertEqual(Legacy, encode(Asn1)).
equivalence_contract_v3_test() ->
T = [ {source_hash, binary}
, {type_info, [{binary, binary, bool, binary, binary}]}
, {byte_code, binary}
, {compiler_version, binary}
, {payable, bool}
],
TI = [{<<"h">>, <<"n">>, true, <<"a">>, <<"o">>}],
V = [ {source_hash, <<"hash">>}
, {type_info, TI}
, {byte_code, <<"code">>}
, {compiler_version, <<"vsn">>}
, {payable, true}
],
Legacy = gmser_chain_objects:serialize(contract, 3, T, V),
Asn1 = {'GajumaruData', 40, 3, {contract, {v3, {'ContractV3',
<<"hash">>,
[ {<<"h">>, <<"n">>, true, <<"a">>, <<"o">>} ],
<<"code">>,
<<"vsn">>,
true
}}}},
?assertEqual(Legacy, encode(Asn1)).
-endif.
+25 -3
View File
@@ -8,7 +8,7 @@
%%%-------------------------------------------------------------------
-module(gmser_chain_objects).
-vsn("0.1.2").
-vsn("0.2.0").
-export([ serialize/4
, deserialize/4
@@ -100,6 +100,8 @@ tag(ac_deposit_tx) -> 94;
tag(ac_update_cops_tx) -> 95;
tag(ac_rollup_tx) -> 96;
tag(ac_proposal_tx) -> 97;
tag(ac_receipt) -> 98;
tag(ac_acct_state) -> 99;
tag(key_block) -> 100;
tag(micro_block) -> 101;
tag(light_micro_block) -> 102;
@@ -115,7 +117,16 @@ tag(entry) -> 140;
tag(entry_create_tx) -> 141;
tag(entry_transfer_tx) -> 142;
tag(entry_destroy_tx) -> 143;
tag(pof) -> 200.
tag(account_key_store) -> 144;
tag(account_create_tx) -> 145;
tag(account_sig_store) -> 146;
tag(auth_tx) -> 147;
tag(proposal_gossip_tx) -> 148;
tag(account_auth_update_tx) -> 149;
tag(pof) -> 200;
%% Gajumaru AC side transactions
tag(ac_side_withdraw_tx) -> 300;
tag(ac_side_rollup_tx) -> 301.
rev_tag(10) -> account;
rev_tag(11) -> signed_tx;
@@ -179,6 +190,8 @@ rev_tag(94) -> ac_deposit_tx;
rev_tag(95) -> ac_update_cops_tx;
rev_tag(96) -> ac_rollup_tx;
rev_tag(97) -> ac_proposal_tx;
rev_tag(98) -> ac_receipt;
rev_tag(99) -> ac_acct_state;
rev_tag(100) -> key_block;
rev_tag(101) -> micro_block;
rev_tag(102) -> light_micro_block;
@@ -194,4 +207,13 @@ rev_tag(140) -> entry;
rev_tag(141) -> entry_create_tx;
rev_tag(142) -> entry_transfer_tx;
rev_tag(143) -> entry_destroy_tx;
rev_tag(200) -> pof.
rev_tag(144) -> account_key_store;
rev_tag(145) -> account_create_tx;
rev_tag(146) -> account_sig_store;
rev_tag(147) -> auth_tx;
rev_tag(148) -> proposal_gossip_tx;
rev_tag(149) -> account_auth_update_tx;
rev_tag(200) -> pof;
%% Gajumaru AC side transactions
rev_tag(300) -> ac_side_withdraw_tx;
rev_tag(301) -> ac_side_rollup_tx.
+1 -1
View File
@@ -6,7 +6,7 @@
%%% @end
%%%-------------------------------------------------------------------
-module(gmser_contract_code).
-vsn("0.1.2").
-vsn("0.2.0").
-include("gmser_contract_code.hrl").
+1 -1
View File
@@ -6,7 +6,7 @@
%%% @end
%%%-------------------------------------------------------------------
-module(gmser_delegation).
-vsn("0.1.2").
-vsn("0.2.0").
-export([ aens_preclaim_sig/3
, aens_name_sig/4
+863 -286
View File
File diff suppressed because it is too large Load Diff
+72
View File
@@ -0,0 +1,72 @@
-module(gmser_dyn_types).
-vsn("0.2.0").
-export([ add_type/3 %% (Tag, Code, Template) -> Types1
, add_type/4 %% (Tag, Code, Template, Types) -> Types1
, from_list/2
, expand/1 ]).
-export([ next_code/1 ]).
next_code(#{codes := Codes}) ->
lists:max(maps:keys(Codes)) + 1.
-spec add_type(Tag, Code, Template) -> Types
when Tag :: gmser_dyn:tag()
, Code :: gmser_dyn:code()
, Template :: gmser_dyn:template()
, Types :: gmser_dyn:types().
add_type(Tag, Code, Template) ->
add_type(Tag, Code, Template, gmser_dyn:registered_types()).
add_type(Tag, Code, Template, Types) ->
elem_to_type({Tag, Code, Template}, Types).
from_list(L, Types) ->
lists:foldl(fun elem_to_type/2, Types, L).
expand(#{vsn := V, templates := Templates0} = Types) ->
Templates =
maps:map(
fun(_, F) when is_function(F, 0) ->
F();
(_, F) when is_function(F, 1) ->
F(V);
(_, T) ->
T
end, Templates0),
Types#{templates := Templates}.
elem_to_type({Tag, Code, Template}, Acc) when is_atom(Tag), is_integer(Code) ->
#{codes := Codes, rev := Rev, templates := Temps} = Acc,
case {is_map_key(Tag, Rev), is_map_key(Code, Codes)} of
{false, false} ->
Acc#{ codes := Codes#{Code => Tag}
, rev := Rev#{Tag => Code}
, templates => Temps#{Tag => Template}
};
{true, _} -> error({duplicate_tag, Tag});
{_, true} -> error({duplicate_code, Code})
end;
elem_to_type({modify, {Tag, Template}}, Acc) ->
#{codes := _, rev := Rev, templates := Templates} = Acc,
_ = maps:get(Tag, Rev),
Templates1 = Templates#{Tag := Template},
Acc#{templates := Templates1};
elem_to_type({labels, Lbls}, Acc) ->
lists:foldl(fun add_label/2, Acc, Lbls);
elem_to_type({vsn, V}, Acc) ->
Acc#{vsn => V};
elem_to_type(Elem, _) ->
error({invalid_type, Elem}).
add_label({L, Code}, #{labels := Lbls, rev_labels := RevLbls} = Acc)
when is_atom(L), is_integer(Code), Code > 0 ->
case {is_map_key(L, Lbls), is_map_key(Code, RevLbls)} of
{false, false} ->
Acc#{labels := Lbls#{L => Code},
rev_labels := RevLbls#{Code => L}};
{true, _} -> error({duplicate_label, L});
{_, true} -> error({duplicate_label_code, Code})
end;
add_label(Elem, _) ->
error({invalid_label, Elem}).
+68 -6
View File
@@ -8,15 +8,21 @@
%%%-------------------------------------------------------------------
-module(gmser_id).
-vsn("0.1.2").
-vsn("0.2.0").
-export([ create/2
, specialize/1
, specialize/2
, specialize_type/1
, is_account/1
, account_pubkey/1
, to_map/1
, is_id/1
]).
-export([ t_id/1
]).
%% For aec_serialization
-export([ encode/1
, decode/1
@@ -26,11 +32,23 @@
, val
}).
-type tag() :: 'account' | 'name'
| 'commitment' | 'contract' | 'channel'
| 'associate_chain' | 'entry' .
-type subtype() :: 0..6.
-type id_map() :: #{ type := simple_tag()
, subtype => subtype()
, value := binary() }.
-type tag() :: {'account', subtype()} | simple_tag().
-type simple_tag() :: 'account'
| 'associate_chain'
| 'channel'
| 'commitment'
| 'contract'
| 'name'
| 'native_token'
| 'entry'.
-type val() :: <<_:256>>.
-opaque(id() :: #id{}).
-type id() :: #id{}.
-export_type([ id/0
, tag/0
@@ -47,7 +65,8 @@
___TAG___ =:= contract;
___TAG___ =:= channel;
___TAG___ =:= associate_chain;
___TAG___ =:= entry
___TAG___ =:= entry;
___TAG___ =:= native_token
).
-define(IS_VAL(___VAL___), byte_size(___VAL___) =:= 32).
@@ -59,6 +78,8 @@
create(Tag, Val) when ?IS_TAG(Tag), ?IS_VAL(Val) ->
#id{ tag = Tag
, val = Val};
create({account,I}, Val) when is_binary(Val), I >= 0, I =< 6 ->
#id{ tag = {account, I}, val = Val};
create(Tag, Val) when ?IS_VAL(Val) ->
error({illegal_tag, Tag});
create(Tag, Val) when ?IS_TAG(Tag)->
@@ -68,32 +89,73 @@ create(Tag, Val) ->
-spec specialize(id()) -> {tag(), val()}.
specialize(#id{tag = {Tag,_}, val = Val}) ->
{Tag, Val};
specialize(#id{tag = Tag, val = Val}) ->
{Tag, Val}.
-spec specialize(id(), tag()) -> val().
specialize(#id{tag = {Tag, _}, val = Val}, Tag) when is_binary(Val) ->
Val;
specialize(#id{tag = Tag, val = Val}, Tag) when ?IS_TAG(Tag), ?IS_VAL(Val) ->
Val.
-spec specialize_type(id()) -> tag().
specialize_type(#id{tag = {Tag, _}}) when ?IS_TAG(Tag) ->
Tag;
specialize_type(#id{tag = Tag}) when ?IS_TAG(Tag) ->
Tag.
-spec is_account(id() | term()) -> boolean().
is_account(#id{tag = account}) ->
true;
is_account(#id{tag = {account, _}}) ->
true;
is_account(_) ->
false.
-spec account_pubkey(id()) -> val().
account_pubkey(#id{tag = account, val = Val}) when ?IS_VAL(Val) ->
Val;
account_pubkey(#id{tag = {account, _}, val = Val}) when ?IS_VAL(Val) ->
Val.
-spec to_map(id()) -> id_map().
to_map(#id{tag = {Tag, SubType}, val = Val}) when ?IS_TAG(Tag) ->
#{ type => Tag
, subtype => SubType
, value => Val };
to_map(#id{tag = Tag, val = Val}) when ?IS_TAG(Tag) ->
#{ type => Tag
, value => Val }.
-spec is_id(term()) -> boolean().
is_id(#id{}) -> true;
is_id(_) -> false.
-spec encode(id()) -> binary().
encode(#id{tag = {account, N}, val = Val}) when N =< 2#111_1111 ->
Ext = 2#1000_0000 bor N,
<<Ext:8, Val/binary>>;
encode(#id{tag = Tag, val = Val}) ->
Res = <<(encode_tag(Tag)):?TAG_SIZE/unit:8, Val/binary>>,
true = ?SERIALIZED_SIZE =:= byte_size(Res),
Res.
-spec decode(binary()) -> id().
decode(<<Ext:8, Rest/binary>>) when Ext >= 2#1000_0000 ->
%% Extended account id type
Type = Ext band 2#0111_1111,
#id{ tag = {account, Type}
, val = Rest };
decode(<<Tag:?TAG_SIZE/unit:8, Val:?PUB_SIZE/binary>>) ->
#id{ tag = decode_tag(Tag)
, val = Val}.
-spec t_id(any()) -> id().
t_id(#id{} = Id) -> Id.
%%%===================================================================
%%% Internal functions
%%%===================================================================
+1 -1
View File
@@ -11,7 +11,7 @@
%%%-------------------------------------------------------------------
-module(gmser_rlp).
-vsn("0.1.2").
-vsn("0.2.0").
-export([ decode/1
, decode_one/1
+1 -1
View File
@@ -1,6 +1,6 @@
{application, gmserialization,
[{description, "Serialization of data for the Gajumaru"},
{vsn, "0.1.0"},
{vsn, "zomp"},
{registered, []},
{applications,
[kernel,
+14
View File
@@ -0,0 +1,14 @@
%% -*- erlang-mode; erlang-indent-level: 4; indent-tabs-mode: nil -*-
[{application, Name, Opts}] = CONFIG.
case lists:keyfind(vsn, 1, Opts) of
{vsn, "zomp"} ->
ZompMetaF = filename:join(filename:dirname(filename:dirname(SCRIPT)), "zomp.meta"),
{ok, ZMeta} = file:consult(ZompMetaF),
{_, {_, _, {Vmaj,Vmin,Vpatch}}} = lists:keyfind(package_id, 1, ZMeta),
VsnStr = unicode:characters_to_list(io_lib:fwrite("~w.~w.~w", [Vmaj, Vmin, Vpatch])),
Opts1 = lists:keyreplace(vsn, 1, Opts, {vsn, VsnStr}),
[{application, Name, Opts1}];
_ ->
CONFIG
end.
+42 -2
View File
@@ -7,7 +7,7 @@
%%%-------------------------------------------------------------------
-module(gmserialization).
-vsn("0.1.2").
-vsn("0.2.0").
-export([ decode_fields/2
, decode_field/2
@@ -29,7 +29,12 @@
-type template() :: [{field_name(), type()}].
-type field_name() :: atom().
-type type() :: 'int'
-type type() :: 'int' % bignum (non-negative, arbitrary size; used for Pucks amounts etc. up to 10^30)
| 'uint128'
| 'uint64'
| 'uint32'
| 'uint16'
| 'uint8'
| 'bool'
| 'binary'
| 'id' %% As defined in aec_id.erl
@@ -118,6 +123,16 @@ encode_field(#{items := Items}, Map) ->
encode_field(Type, T) when tuple_size(Type) =:= tuple_size(T) ->
Zipped = lists:zip(tuple_to_list(Type), tuple_to_list(T)),
[encode_field(X, Y) || {X, Y} <- Zipped];
encode_field(uint128, X) when is_integer(X), X >= 0, X < (1 bsl 128) ->
binary:encode_unsigned(X);
encode_field(uint64, X) when is_integer(X), X >= 0, X < (1 bsl 64) ->
binary:encode_unsigned(X);
encode_field(uint32, X) when is_integer(X), X >= 0, X < (1 bsl 32) ->
binary:encode_unsigned(X);
encode_field(uint16, X) when is_integer(X), X >= 0, X < (1 bsl 16) ->
binary:encode_unsigned(X);
encode_field(uint8, X) when is_integer(X), X >= 0, X < (1 bsl 8) ->
binary:encode_unsigned(X);
encode_field(int, X) when is_integer(X), X >= 0 ->
binary:encode_unsigned(X);
encode_field(binary, X) when is_binary(X) -> X;
@@ -141,6 +156,31 @@ decode_field(#{items := Items}, List) when length(List) =:= length(Items) ->
decode_field(Type, List) when length(List) =:= tuple_size(Type) ->
Zipped = lists:zip(tuple_to_list(Type), List),
list_to_tuple([decode_field(X, Y) || {X, Y} <- Zipped]);
decode_field(uint128, X) when is_binary(X) ->
I = binary:decode_unsigned(X),
if I < (1 bsl 128) -> I;
true -> error({illegal, uint128, X})
end;
decode_field(uint64, X) when is_binary(X) ->
I = binary:decode_unsigned(X),
if I < (1 bsl 64) -> I;
true -> error({illegal, uint64, X})
end;
decode_field(uint32, X) when is_binary(X) ->
I = binary:decode_unsigned(X),
if I < (1 bsl 32) -> I;
true -> error({illegal, uint32, X})
end;
decode_field(uint16, X) when is_binary(X) ->
I = binary:decode_unsigned(X),
if I < (1 bsl 16) -> I;
true -> error({illegal, uint16, X})
end;
decode_field(uint8, X) when is_binary(X) ->
I = binary:decode_unsigned(X),
if I < (1 bsl 8) -> I;
true -> error({illegal, uint8, X})
end;
decode_field(int, <<0:8, X/binary>> = B) when X =/= <<>> ->
error({illegal, int, B});
decode_field(int, X) when is_binary(X) -> binary:decode_unsigned(X);
+101 -12
View File
@@ -22,10 +22,39 @@
, {native_token , 32}
, {commitment , 32}
, {peer_pubkey , 32}
, {hash , 32}
, {state , 32}
, {poi , not_applicable}]).
encode_decode_test_() ->
encode_decode_test_(?TYPES).
encode_decode_known_types_test_() ->
KnownTypes = known_types(),
SizedTypes = [{T, ?TEST_MODULE:byte_size_for_type(T)} || T <- KnownTypes],
encode_decode_test_(SizedTypes).
prefixes_are_known_types_test() ->
MappedPfxs = mapped_prefixes(),
KnownTypes = known_types(),
lists:foreach(
fun({Pfx, Type}) ->
case lists:member(Type, KnownTypes) of
true -> ok;
false ->
error({not_a_known_type, Pfx, Type})
end
end, MappedPfxs),
lists:foreach(
fun(Type) ->
case lists:keyfind(Type, 2, MappedPfxs) of
{_, _} -> ok;
false ->
error({has_no_mapped_prefix, Type})
end
end, KnownTypes).
encode_decode_test_(Types) ->
[{"Byte sizes are correct",
fun() ->
lists:foreach(
@@ -33,7 +62,7 @@ encode_decode_test_() ->
{_Type, _, ByteSize} = {Type, ByteSize,
?TEST_MODULE:byte_size_for_type(Type)}
end,
?TYPES)
Types)
end
},
{"Serialize/deserialize known types",
@@ -50,7 +79,7 @@ encode_decode_test_() ->
{Type, Key} = ?TEST_MODULE:decode(EncodedKey),
{ok, Key} = ?TEST_MODULE:safe_decode(Type, EncodedKey)
end,
?TYPES)
Types)
end
},
{"Key size check works",
@@ -58,17 +87,18 @@ encode_decode_test_() ->
lists:foreach(
fun({_Type, not_applicable}) -> ok;
({Type, ByteSize}) ->
CheckIlligalSize =
CheckIllegalSize =
fun(S) ->
Key = <<42:S/unit:8>>,
EncodedKey = ?TEST_MODULE:encode(Type, Key),
?assertError(incorrect_size, ?TEST_MODULE:encode(Type, Key)),
EncodedKey = ?TEST_MODULE:unsafe_encode(Type, Key), %% no size check
{error, invalid_encoding} = ?TEST_MODULE:safe_decode(Type, EncodedKey)
end,
CheckIlligalSize(0),
CheckIlligalSize(ByteSize - 1),
CheckIlligalSize(ByteSize + 1)
CheckIllegalSize(0),
CheckIllegalSize(ByteSize - 1),
CheckIllegalSize(ByteSize + 1)
end,
?TYPES)
Types)
end
},
{"Missing prefix",
@@ -91,7 +121,7 @@ encode_decode_test_() ->
<<_WholePrefix:3/unit:8, RestOfKey2/binary>> = EncodedKey,
{error, invalid_encoding} = ?TEST_MODULE:safe_decode(Type, RestOfKey2)
end,
?TYPES)
Types)
end
},
{"Piece of encoded key",
@@ -110,7 +140,7 @@ encode_decode_test_() ->
{error, invalid_encoding} = ?TEST_MODULE:safe_decode(Type, HalfKey),
{error, invalid_encoding} = ?TEST_MODULE:safe_decode(Type, RestOfKey)
end,
?TYPES)
Types)
end
},
{"Encode/decode binary with only zeros",
@@ -131,8 +161,67 @@ encode_decode_test_() ->
Encoded1 = base58:binary_to_base58(Bin),
Decoded1 = base58:base58_to_binary(Encoded1),
?assertEqual(Bin, Decoded1)
end, ?TYPES)
end, Types)
end,
Bins)
end}
end},
{"Encode/decode keypairs",
fun() ->
KP1 = enacl:sign_keypair(),
Enc1 = ?TEST_MODULE:encode_keypair(KP1),
{ok, KP1} = ?TEST_MODULE:safe_decode_keypair(Enc1),
KP2 = enacl:sign_keypair(),
Enc2 = ?TEST_MODULE:encode_keypair(KP2),
{ok, KP2} = ?TEST_MODULE:safe_decode_keypair(Enc2),
BadEnc = Enc1#{~"priv" => maps:get(~"priv", Enc2)},
{error, illegal_encoding} = ?TEST_MODULE:safe_decode_keypair(BadEnc)
end
},
{"Encode AND decode both 32-byte and 64-byte account_seckey",
fun() ->
%% Originally, we could encode a 64-byte seckey, but decode would fail.
#{public := Pub, secret := Sec} = enacl:sign_keypair(),
<<Seed:32/binary, Pub:32/binary>> = Sec,
EncSeed = ?TEST_MODULE:encode(account_seckey, Seed),
EncSec = ?TEST_MODULE:encode(account_seckey, Sec),
{ok, Seed} = ?TEST_MODULE:safe_decode(account_seckey, EncSeed),
{ok, Sec} = ?TEST_MODULE:safe_decode(account_seckey, EncSec)
end
}
].
known_types() ->
Forms = get_forms(),
[{type, _, union, Types}] =
[Def || {attribute, _, type, {known_type, Def, []}} <- Forms],
lists:flatmap(fun known_type_entry/1, Types).
known_type_entry({atom, _, Name}) ->
[Name];
known_type_entry({type, _, tuple,
[{atom, _, account_pubkey},
{type, _, range, [{integer, _, Lo}, {integer, _, Hi}]}]}) ->
[{account_pubkey, N} || N <- lists:seq(Lo, Hi)];
known_type_entry(Other) ->
error({unsupported_known_type, Other}).
mapped_prefixes() ->
Forms = get_forms(),
[Clauses] = [Cs || {function,_,pfx2type,1,Cs} <- Forms],
Abst = [{B, A} || {clause,_,[B],[],[A]} <- Clauses],
lists:map(
fun({B, A}) ->
{eval_expr(B), eval_expr(A)}
end, Abst).
get_forms() ->
get_forms(code:which(?TEST_MODULE)).
get_forms(Beam) ->
{ok, {_, [{abstract_code, {raw_abstract_v1, Forms}}]}} =
beam_lib:chunks(Beam, [abstract_code]),
Forms.
eval_expr(Expr) ->
{value, Val, []} = erl_eval:expr(Expr, []),
Val.
+44
View File
@@ -0,0 +1,44 @@
-module(gmser_id_tests).
-include_lib("eunit/include/eunit.hrl").
-define(PUBKEY, <<12345:32/unit:8>>).
is_account_test() ->
{"is_account recognizes standard and extended account ids",
fun() ->
?assert(gmser_id:is_account(gmser_id:create(account, ?PUBKEY))),
?assert(gmser_id:is_account(gmser_id:create({account, 0}, ?PUBKEY))),
?assert(gmser_id:is_account(gmser_id:create({account, 5}, ?PUBKEY))),
?assertNot(gmser_id:is_account(gmser_id:create(contract, ?PUBKEY))),
?assertNot(gmser_id:is_account(not_an_id))
end}.
account_pubkey_test() ->
{"account_pubkey returns the 32-byte account hash",
fun() ->
?assertEqual(?PUBKEY,
gmser_id:account_pubkey(gmser_id:create(account, ?PUBKEY))),
?assertEqual(?PUBKEY,
gmser_id:account_pubkey(gmser_id:create({account, 3}, ?PUBKEY))),
?assertEqual(?PUBKEY,
gmser_id:account_pubkey(gmser_id:create({account, 6}, ?PUBKEY)))
end}.
account_pubkey_matches_specialize_test() ->
{"account_pubkey agrees with specialize/2 for account ids",
fun() ->
Id = gmser_id:create({account, 2}, ?PUBKEY),
?assertEqual(gmser_id:specialize(Id, account),
gmser_id:account_pubkey(Id))
end}.
extended_account_roundtrip_test() ->
{"extended account ids round-trip through encode/decode",
fun() ->
Id = gmser_id:create({account, 4}, ?PUBKEY),
?assert(gmser_id:is_account(Id)),
Id1 = gmser_id:decode(gmser_id:encode(Id)),
?assertEqual(?PUBKEY, gmser_id:account_pubkey(Id1)),
?assertEqual(account, gmser_id:specialize_type(Id1))
end}.
+2 -2
View File
@@ -2,9 +2,9 @@
{type,lib}.
{modules,[]}.
{prefix,none}.
{author,"Hans Svensson"}.
{desc,"Serialization helpers for the Gajumaru."}.
{package_id,{"otpr","gmserialization",{0,1,2}}}.
{author,"Hans Svensson"}.
{package_id,{"otpr","gmserialization",{0,2,0}}}.
{deps,[{"otpr","eblake2",{1,0,1}},{"otpr","base58",{0,1,1}}]}.
{key_name,none}.
{a_email,[]}.
Executable
+49
View File
@@ -0,0 +1,49 @@
#!/bin/sh
set -e
APP=$(basename "$PWD")
SRC="_build/default/lib/$APP"
DST="$PWD/_build/zomp/lib/$APP"
IGNORE_FILE="zomp.ignore"
mkdir -p "$DST"
# Remove broken symlinks
find "$SRC" -type l ! -exec test -e {} \; -delete || true
# Build ignore matcher
IGNORE_TEMP=$(mktemp)
trap "rm -f $IGNORE_TEMP" EXIT
# Expand globs in zomp.ignore to patterns suitable for grep
if [ -e "$IGNORE_FILE" ]; then
grep -v '^\s*#' "$IGNORE_FILE" | sed 's#/#\\/#g' | sed 's/\./\\./g' | sed 's/\*/.*/g' > "$IGNORE_TEMP"
fi
# Copy Git-tracked and Zomp-allowed files
git ls-files -z | while IFS= read -r -d '' file; do
# Skip if ignored
echo "$file" | grep -Eq -f "$IGNORE_TEMP" && continue
# Only copy if file exists in the build dir
if [ -e "$SRC/$file" ]; then
mkdir -p "$DST/$(dirname "$file")"
cp -a "$SRC/$file" "$DST/$file"
fi
done
rm "$IGNORE_TEMP"
# Copy metadata
cp "$PWD/zomp.meta" "$DST/"
cp "$PWD/Emakefile" "$DST/"
# copy generated schema
SCHEMA="$SRC/priv/gmhc_schema.json"
if [ -e "$SCHEMA" ]; then
mkdir -p "$DST/priv"
cp -a "$SCHEMA" "$DST/priv/$(basename "$SCHEMA")"
fi
# Clean up beam files just in case
[ -d "$DST/ebin" ] && find "$DST/ebin" -name '*.beam' -exec rm -f {} + || true