gmserialization/asn1/FINDINGS.md

ASN.1 for gmserialization Static Encoding - Findings Diary

This is a living diary documenting the investigation into using ASN.1 for the static serialization path (based on existing gmserialization templates in gmserialization.erl and gmser_chain_objects.erl). Dynamic encoding is out of scope.

Focus areas:

• Modeling static templates with ASN.1 (portability goal).
• Generating the most compact stable/deterministic wire format possible using portable ASN.1 techniques (UPER etc.).
• Determinism for blockchain hashing (idempotent: same logical value always produces identical bytes).
• (Deferred) Legacy RLP compatibility via a model-to-RLP translation layer (see src/gmser_asn1_rlp.erl).

The single source of truth is the ASN.1 schema in GajumaruSerialization.asn.

---

2026-07-08 - Setup and Initial Schema

• Schema (asn1/GajumaruSerialization.asn) models:

  • GajumaruData as top-level (tag + vsn + content).
  • Content CHOICE with templateFields (generic) and concrete types (e.g. SignedTx, ContractV*).
  • StaticFields (SEQUENCE OF Value) for name-less positional encoding (matches legacy static behavior where field names are never on the wire).
  • Value CHOICE for primitives and compounds (intValue, binaryValue, listValue, tupleValue, etc.).
  • Supports all static template types: int, bool, binary, id, [T], tuples, #{items => [...]}.

• Initial schema comments were DER-oriented (migration path). Updated to emphasize compact UPER + portability.

• Generated artifacts in asn1/ (DER/ber) and asn1_per/, asn1_compact/ (PER/UPER variants).

• Key files:

  • asn1/GajumaruSerialization.asn (source)
  • src/gmser_asn1_rlp.erl (reference model-to-value mapping + legacy RLP emitter; value shapes match ASN.1)
  • Tests in test/gmser_chain_objects_tests.erl and inside gmser_asn1_rlp for equivalence.

2026-07-08 - Compact Encoding Experiments (UPER)

Goal: most compact stable wire using standard portable ASN.1 (not custom non-portable rules, not RLP).

• Tried standard DER → too verbose (tiny case: ~36 bytes vs legacy RLP 5 bytes).

• Switched to UPER (Unaligned PER) — the most compact standard ASN.1 encoding rule with good canonical/deterministic properties.

  • Compiled via asn1ct:compile(..., [uper]).
  • Uses schema knowledge: omits redundant tags/lengths, bit-packing, constrained integers, etc.
  • Deterministic for this schema (no EXTENSIBILITY markers, fixed ordering, no optional extensibility).

• Schema optimizations for compactness:

  • Added CompactStatic top-level type (avoids extra Content CHOICE tag overhead for common static path).
  • staticFields (pure SEQUENCE OF Value) — no IA5String names on wire.
  • Constrained tag/vsn (INTEGER (0..65535), (0..255)) for better packing.
  • Prefer concrete SEQUENCES (e.g. SignedTx) or staticFields over generic templateFields (names add cost).
  • Kept TemplateFields for debug/transition only.

• Size results (using CompactStatic + staticFields where appropriate):

  Case	Legacy RLP	UPER (optimized)	Delta	Notes
  tiny (tag/vsn + int + 1B bin)	5 B	9 B	+4 B	Big improvement vs DER
  list of 3 ints	7 B	13 B	+6 B	—
  tuple (int + bin)	8 B	12 B	+4 B	—
  signed_tx-like (concrete)	7–24 B	11–14 B	small	Concrete helps
  256-byte payload	264 B	263 B	-1 B	Matches or beats RLP
  contract v3 (complex)	~18–20 B	~25–35 B (generic); better w/ concrete	—	Structural overhead on complex nested

• UPER is stable: encode → decode → re-encode produces identical bytes. Roundtrips work.

• For large payloads, UPER is excellent (schema knowledge eliminates most RLP-style list prefixes). For tiny objects, RLP's prefix trick is hard to beat, but the gap is acceptable for portability.

• OER (Octet Encoding Rules) also compiled but was larger (18 B on tiny case).

2026-07-08 - Portability & Stability Takeaways

• The schema + UPER is fully portable. Other languages can:

  1. Compile the .asn with their ASN.1 tool.
  2. Build a value matching CompactStatic / staticFields / concrete types.
  3. Call their UPER encoder → identical compact bytes.

• No Erlang-specific runtime required for the new wire format.

• Determinism comes from:

  • UPER canonical packing rules.
  • Constrained types in schema.
  • Explicit staticFields (no map iteration, names omitted).
  • Same rules as legacy for ints (minimal), ordering, etc.

• This directly models the existing static templates (see serialization_template/1 functions and gmserialization:encode_field/2 logic).

• Concrete types in schema give best compactness for known objects.

• Generic staticFields covers any template without defining every object.

Next Steps / Open Questions (Diary Entries)

• Add more concrete types from gmser_chain_objects (many tags) to reduce generic overhead.
• Add more ASN.1 constraints (SIZE, value ranges) to help PER pack tighter.
• Measure on real on-chain objects (key_block, etc.).
• Decide on top-level header for the new format (keep tag/vsn?).
• (Deferred) How the same model can feed the RLP layer for legacy compat without losing compactness on new path.
• Consider if a custom "ASN.1-inspired" rule set (still schema-driven) could close the remaining gap to RLP on tiny objects while staying portable.

2026-07-08 - Schema Updated for Bignums

• Updated GajumaruSerialization.asn:

  • Introduced BigInt ::= INTEGER (0..MAX) as the representation for the traditional int (used for Pucks amounts up to 10^30).
  • Value CHOICE now uses bigIntValue for the bignum case.
  • Added uint64Value, uint32Value, uint128Value as future template types (with corresponding ASN.1 subtypes).
  • Updated header comments to document the bignum nature of int.

• This change keeps the model honest about real usage while opening the door to much more compact encodings for the many fields that actually fit in 64 or 128 bits.

• Next: We should extend the Erlang-side type() in gmserialization.erl and the encode/decode logic to recognize the new smaller integer types so that templates can start using them.

---

This file should be kept as a living diary. Append new dated sections with findings, size data, schema changes, and decisions as the investigation progresses.

2026-07-08 - Handling of int as Bignums (Pucks, amounts, etc.)

Important clarification from domain:

• In practice, the int type in static templates is frequently used for large bignums.
• Example: Amount fields (balances, transaction amounts, etc.) are denominated in "Pucks".
• Maximum value mentioned: 1 × 10^30.
• This is ~ 2^99.66, i.e., requires up to ~13 bytes in minimal unsigned encoding.

Current legacy handling (in gmserialization.erl):

encode_field(int, X) when is_integer(X), X >= 0 ->
    binary:encode_unsigned(X);

This produces minimal big-endian unsigned with no leading zero byte (except for the value 0).

Implications for ASN.1 model:

• We should keep int / intValue modeled as an unbounded non-negative integer (bignum):

  BigInt ::= INTEGER (0..MAX)
  

  (or simply INTEGER with documentation that it is used for non-negative bignums).

• Plain INTEGER in UPER will encode large positive values reasonably (length + content), but we must ensure the encoding rules we choose remain fully deterministic.

• To allow more compact encodings where ranges are known, we should introduce new template types for smaller integers: Suggested new type() variants in the Erlang template language:

  • uint64 -- 0 .. 2^64-1
  • uint32 -- 0 .. 2^32-1
  • uint16, uint8, uint128 etc. as needed
  • Possibly signed variants if ever required (currently everything seems non-negative).

  Corresponding in ASN.1 (inside Value CHOICE or as reusable types):

  Uint64  ::= INTEGER (0..18446744073709551615)
  Uint32  ::= INTEGER (0..4294967295)
  Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
  

• In the schema's Value CHOICE we can evolve to:

  Value ::= CHOICE {
      bigIntValue [0] BigInt,     -- the classic "int" for Pucks etc.
      uint64Value [7] Uint64,
      uint32Value [8] Uint32,
      ...
      -- keep backward-compatible intValue alias if needed during transition
  }
  

• Benefits for compactness:

  • Constrained Uint64 etc. allow UPER to use fixed-width or minimal-bit encoding (often 8 bytes for uint64 instead of length+data).
  • Still fully portable and deterministic.

• Impact on existing templates:

  • Most amount-related fields should eventually be annotated as uint128 or a BigInt alias rather than plain int.
  • Small counters, versions, indices can use uint32 / uint64.
  • This may require extending the type() in gmserialization.erl and the encoders.

2026-07-08 - Updated gmserialization source

• Extended type() in src/gmserialization.erl with uint128, uint64, uint32, uint16, uint8 (keeping int for bignums).

• Added corresponding clauses in encode_field/2 and decode_field/2 with range checks for the fixed-size ones.

• Updated src/gmser_asn1_rlp.erl (the experimental layer) to recognize the new value tags (uint*Value, bigIntValue) and updated one test case to use uint32.

• Updated doc/static.md example types.

• int remains fully backward compatible for bignum use.

• All existing tests + new type usage pass.

• In the legacy RLP translation layer:

  • bigIntValue would continue to use binary:encode_unsigned/1.
  • Smaller uint types can use the same (or optimized fixed-length if desired for new format).

Next action items:

• Update GajumaruSerialization.asn to introduce BigInt, Uint* types and adjust Value.
• Decide on naming in the Erlang template DSL (int remains bignum alias? or rename?).
• Add example in the schema for a balance field.
• Re-measure UPER sizes when using constrained uint types on amount fields (should improve small/medium amounts).

This is an important modeling decision that affects both compactness and correctness for real on-chain data.

2026-07-08 - Source Update Completed & Verified

• Performed the source changes in src/gmserialization.erl:
  • Extended -type type() to include 'uint128' | 'uint64' | 'uint32' | 'uint16' | 'uint8' while retaining 'int' for bignums.
  • Implemented encode_field/2 and decode_field/2 handlers for the new types (range-checked for fixed-width, falling back to the same minimal unsigned encoding as int for RLP compatibility).
• Synced src/gmser_asn1_rlp.erl:
  • Added support for new ASN.1 value variants ({uint*Value, ...}, {bigIntValue, ...}) in encode_asn1_value/1.
  • Updated test data and comments to use the new types.
• Updated doc/static.md to reflect the extended type language.
• Verified:
  • Legacy templates using int continue to work unchanged.
  • New types (e.g. {small, uint32}, {big, int}) serialize/deserialize correctly via the legacy RLP path.
  • All 7 equivalence tests in gmser_asn1_rlp still pass.
  • gmser_chain_objects_tests (12 tests) still pass.
• The ASN.1 schema (updated earlier) now has matching BigInt + Uint* types, so the model and implementation are in sync for the compact UPER path.

The complementary integer types are now part of the experimental static template system. This enables the ASN.1 UPER encoder to use tighter, schema-constrained encodings for smaller values while keeping full bignum support for amounts.

Next diary items (still open):

• Extend more concrete types in the schema.
• Add actual UPER-based encode path in the library (beyond the RLP layer).
• Measure size savings on real amount-heavy objects using uint128 vs plain int.