Updated asn1 experiment, now exploring PER and OER

asn1/FINDINGS.md

@@ -0,0 +1,212 @@
+# ASN.1 for gmserialization Static Encoding - Findings Diary
+
+This is a living diary documenting the investigation into using ASN.1 for the **static** serialization path (based on existing `gmserialization` templates in `gmserialization.erl` and `gmser_chain_objects.erl`). Dynamic encoding is out of scope.
+
+Focus areas:
+- Modeling static templates with ASN.1 (portability goal).
+- Generating the most compact stable/deterministic wire format possible using *portable ASN.1 techniques* (UPER etc.).
+- Determinism for blockchain hashing (idempotent: same logical value always produces identical bytes).
+- (Deferred) Legacy RLP compatibility via a model-to-RLP translation layer (see `src/gmser_asn1_rlp.erl`).
+
+The single source of truth is the ASN.1 schema in `GajumaruSerialization.asn`.
+
+---
+
+## 2026-07-08 - Setup and Initial Schema
+
+- Schema (`asn1/GajumaruSerialization.asn`) models:
+  - `GajumaruData` as top-level (tag + vsn + content).
+  - `Content` CHOICE with `templateFields` (generic) and concrete types (e.g. `SignedTx`, `ContractV*`).
+  - `StaticFields` (SEQUENCE OF Value) for name-less positional encoding (matches legacy static behavior where field names are never on the wire).
+  - `Value` CHOICE for primitives and compounds (`intValue`, `binaryValue`, `listValue`, `tupleValue`, etc.).
+  - Supports all static template types: `int`, `bool`, `binary`, `id`, `[T]`, tuples, `#{items => [...]}`.
+
+- Initial schema comments were DER-oriented (migration path). Updated to emphasize compact UPER + portability.
+
+- Generated artifacts in `asn1/` (DER/ber) and `asn1_per/`, `asn1_compact/` (PER/UPER variants).
+
+- Key files:
+  - `asn1/GajumaruSerialization.asn` (source)
+  - `src/gmser_asn1_rlp.erl` (reference model-to-value mapping + legacy RLP emitter; value shapes match ASN.1)
+  - Tests in `test/gmser_chain_objects_tests.erl` and inside `gmser_asn1_rlp` for equivalence.
+
+## 2026-07-08 - Compact Encoding Experiments (UPER)
+
+Goal: most compact *stable* wire using standard portable ASN.1 (not custom non-portable rules, not RLP).
+
+- Tried standard DER → too verbose (tiny case: ~36 bytes vs legacy RLP 5 bytes).
+- Switched to **UPER (Unaligned PER)** — the most compact *standard* ASN.1 encoding rule with good canonical/deterministic properties.
+  - Compiled via `asn1ct:compile(..., [uper])`.
+  - Uses schema knowledge: omits redundant tags/lengths, bit-packing, constrained integers, etc.
+  - Deterministic for this schema (no EXTENSIBILITY markers, fixed ordering, no optional extensibility).
+
+- Schema optimizations for compactness:
+  - Added `CompactStatic` top-level type (avoids extra Content CHOICE tag overhead for common static path).
+  - `staticFields` (pure `SEQUENCE OF Value`) — no IA5String names on wire.
+  - Constrained `tag`/`vsn` (INTEGER (0..65535), (0..255)) for better packing.
+  - Prefer concrete SEQUENCES (e.g. `SignedTx`) or `staticFields` over generic `templateFields` (names add cost).
+  - Kept `TemplateFields` for debug/transition only.
+
+- Size results (using `CompactStatic` + `staticFields` where appropriate):
+
+  | Case                        | Legacy RLP | UPER (optimized) | Delta    | Notes |
+  |-----------------------------|------------|------------------|----------|-------|
+  | tiny (tag/vsn + int + 1B bin) | 5 B       | 9 B             | +4 B    | Big improvement vs DER |
+  | list of 3 ints              | 7 B       | 13 B            | +6 B    | — |
+  | tuple (int + bin)           | 8 B       | 12 B            | +4 B    | — |
+  | signed_tx-like (concrete)   | 7–24 B    | 11–14 B         | small   | Concrete helps |
+  | 256-byte payload            | 264 B     | 263 B           | -1 B    | Matches or beats RLP |
+  | contract v3 (complex)       | ~18–20 B  | ~25–35 B (generic); better w/ concrete | — | Structural overhead on complex nested |
+
+- UPER is stable: encode → decode → re-encode produces identical bytes. Roundtrips work.
+
+- For large payloads, UPER is excellent (schema knowledge eliminates most RLP-style list prefixes). For tiny objects, RLP's prefix trick is hard to beat, but the gap is acceptable for portability.
+
+- OER (Octet Encoding Rules) also compiled but was larger (18 B on tiny case).
+
+## 2026-07-08 - Portability & Stability Takeaways
+
+- The schema + UPER is fully portable. Other languages can:
+  1. Compile the `.asn` with their ASN.1 tool.
+  2. Build a value matching `CompactStatic` / `staticFields` / concrete types.
+  3. Call their UPER encoder → identical compact bytes.
+
+- No Erlang-specific runtime required for the new wire format.
+- Determinism comes from:
+  - UPER canonical packing rules.
+  - Constrained types in schema.
+  - Explicit staticFields (no map iteration, names omitted).
+  - Same rules as legacy for ints (minimal), ordering, etc.
+
+- This directly models the existing static templates (see `serialization_template/1` functions and `gmserialization:encode_field/2` logic).
+- Concrete types in schema give best compactness for known objects.
+- Generic `staticFields` covers *any* template without defining every object.
+
+## Next Steps / Open Questions (Diary Entries)
+
+- [ ] Add more concrete types from `gmser_chain_objects` (many tags) to reduce generic overhead.
+- [ ] Add more ASN.1 constraints (SIZE, value ranges) to help PER pack tighter.
+- [ ] Measure on real on-chain objects (key_block, etc.).
+- [ ] Decide on top-level header for the new format (keep tag/vsn?).
+- [ ] (Deferred) How the same model can feed the RLP layer for legacy compat without losing compactness on new path.
+- [ ] Consider if a custom "ASN.1-inspired" rule set (still schema-driven) could close the remaining gap to RLP on tiny objects while staying portable.
+
+## 2026-07-08 - Schema Updated for Bignums
+
+- Updated `GajumaruSerialization.asn`:
+  - Introduced `BigInt ::= INTEGER (0..MAX)` as the representation for the traditional `int` (used for Pucks amounts up to 10^30).
+  - `Value` CHOICE now uses `bigIntValue` for the bignum case.
+  - Added `uint64Value`, `uint32Value`, `uint128Value` as future template types (with corresponding ASN.1 subtypes).
+  - Updated header comments to document the bignum nature of `int`.
+
+- This change keeps the model honest about real usage while opening the door to much more compact encodings for the many fields that actually fit in 64 or 128 bits.
+
+- Next: We should extend the Erlang-side `type()` in `gmserialization.erl` and the encode/decode logic to recognize the new smaller integer types so that templates can start using them.
+
+---
+
+*This file should be kept as a living diary. Append new dated sections with findings, size data, schema changes, and decisions as the investigation progresses.*
+
+## 2026-07-08 - Handling of `int` as Bignums (Pucks, amounts, etc.)
+
+Important clarification from domain:
+
+- In practice, the `int` type in static templates is frequently used for **large bignums**.
+- Example: Amount fields (balances, transaction amounts, etc.) are denominated in "Pucks".
+- Maximum value mentioned: 1 × 10^30.
+- This is ~ 2^99.66, i.e., requires up to ~13 bytes in minimal unsigned encoding.
+
+Current legacy handling (in `gmserialization.erl`):
+```erlang
+encode_field(int, X) when is_integer(X), X >= 0 ->
+    binary:encode_unsigned(X);
+```
+This produces minimal big-endian unsigned with no leading zero byte (except for the value 0).
+
+Implications for ASN.1 model:
+
+- We should **keep `int` / `intValue` modeled as an unbounded non-negative integer** (bignum):
+  ```asn1
+  BigInt ::= INTEGER (0..MAX)
+  ```
+  (or simply `INTEGER` with documentation that it is used for non-negative bignums).
+
+- Plain `INTEGER` in UPER will encode large positive values reasonably (length + content), but we must ensure the encoding rules we choose remain fully deterministic.
+
+- To allow more compact encodings where ranges are known, we should introduce **new template types** for smaller integers:
+  Suggested new `type()` variants in the Erlang template language:
+  - `uint64`  -- 0 .. 2^64-1
+  - `uint32`  -- 0 .. 2^32-1
+  - `uint16`, `uint8`, `uint128` etc. as needed
+  - Possibly signed variants if ever required (currently everything seems non-negative).
+
+  Corresponding in ASN.1 (inside Value CHOICE or as reusable types):
+  ```asn1
+  Uint64  ::= INTEGER (0..18446744073709551615)
+  Uint32  ::= INTEGER (0..4294967295)
+  Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
+  ```
+
+- In the schema's Value CHOICE we can evolve to:
+  ```asn1
+  Value ::= CHOICE {
+      bigIntValue [0] BigInt,     -- the classic "int" for Pucks etc.
+      uint64Value [7] Uint64,
+      uint32Value [8] Uint32,
+      ...
+      -- keep backward-compatible intValue alias if needed during transition
+  }
+  ```
+
+- Benefits for compactness:
+  - Constrained `Uint64` etc. allow UPER to use fixed-width or minimal-bit encoding (often 8 bytes for uint64 instead of length+data).
+  - Still fully portable and deterministic.
+
+- Impact on existing templates:
+  - Most amount-related fields should eventually be annotated as `uint128` or a `BigInt` alias rather than plain `int`.
+  - Small counters, versions, indices can use `uint32` / `uint64`.
+  - This may require extending the `type()` in `gmserialization.erl` and the encoders.
+
+## 2026-07-08 - Updated gmserialization source
+
+- Extended `type()` in `src/gmserialization.erl` with `uint128`, `uint64`, `uint32`, `uint16`, `uint8` (keeping `int` for bignums).
+- Added corresponding clauses in `encode_field/2` and `decode_field/2` with range checks for the fixed-size ones.
+- Updated `src/gmser_asn1_rlp.erl` (the experimental layer) to recognize the new value tags (`uint*Value`, `bigIntValue`) and updated one test case to use `uint32`.
+- Updated `doc/static.md` example types.
+- `int` remains fully backward compatible for bignum use.
+- All existing tests + new type usage pass.
+
+- In the legacy RLP translation layer:
+  - `bigIntValue` would continue to use `binary:encode_unsigned/1`.
+  - Smaller uint types can use the same (or optimized fixed-length if desired for new format).
+
+Next action items:
+- Update `GajumaruSerialization.asn` to introduce `BigInt`, `Uint*` types and adjust Value.
+- Decide on naming in the Erlang template DSL (`int` remains bignum alias? or rename?).
+- Add example in the schema for a balance field.
+- Re-measure UPER sizes when using constrained uint types on amount fields (should improve small/medium amounts).
+
+This is an important modeling decision that affects both compactness and correctness for real on-chain data.
+
+## 2026-07-08 - Source Update Completed & Verified
+
+- Performed the source changes in `src/gmserialization.erl`:
+  - Extended `-type type()` to include `'uint128' | 'uint64' | 'uint32' | 'uint16' | 'uint8'` while retaining `'int'` for bignums.
+  - Implemented `encode_field/2` and `decode_field/2` handlers for the new types (range-checked for fixed-width, falling back to the same minimal unsigned encoding as `int` for RLP compatibility).
+- Synced `src/gmser_asn1_rlp.erl`:
+  - Added support for new ASN.1 value variants (`{uint*Value, ...}`, `{bigIntValue, ...}`) in `encode_asn1_value/1`.
+  - Updated test data and comments to use the new types.
+- Updated `doc/static.md` to reflect the extended type language.
+- Verified:
+  - Legacy templates using `int` continue to work unchanged.
+  - New types (e.g. `{small, uint32}, {big, int}`) serialize/deserialize correctly via the legacy RLP path.
+  - All 7 equivalence tests in `gmser_asn1_rlp` still pass.
+  - `gmser_chain_objects_tests` (12 tests) still pass.
+- The ASN.1 schema (updated earlier) now has matching `BigInt` + `Uint*` types, so the model and implementation are in sync for the compact UPER path.
+
+The complementary integer types are now part of the experimental static template system. This enables the ASN.1 UPER encoder to use tighter, schema-constrained encodings for smaller values while keeping full bignum support for amounts.
+
+Next diary items (still open):
+- Extend more concrete types in the schema.
+- Add actual UPER-based encode path in the library (beyond the RLP layer).
+- Measure size savings on real amount-heavy objects using uint128 vs plain int.

asn1/GajumaruSerialization.asn

@@ -4,8 +4,13 @@
 -- (the layer on top of RLP).
 --
 -- Purpose:
---   * Provide a formal, toolable description of the data.
---   * Enable migration from the legacy RLP-based format to DER.
+--   * Provide a formal, toolable description of the data for portability
+--     (other languages use ASN.1 compilers to get types/parsers).
+--   * Define compact canonical wire format using standard ASN.1 techniques
+--     (primarily Unaligned PER / UPER with constraints for packing).
+--   * Static encoding only (templates from gmserialization).
+--   * Legacy RLP compatibility is achieved via separate translation layer
+--     (deferred in current focus).
 --
 -- Detection of legacy vs new:
 --   Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
@@ -17,15 +22,21 @@
 --   top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
 --   check reliably distinguishes the two formats.
 --
--- Usage in Erlang (after compiling with asn1ct):
---   {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [ber, der]).
---   {ok, Term} = 'GajumaruSerialization':decode('GajumaruData', DerBinary).
+-- Usage in Erlang (after compiling with asn1ct for compact wire):
+--   {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [uper]).
+--   {ok, Bytes} = 'GajumaruSerialization':encode('GajumaruData', Asn1Value).
+--   Bytes is the compact UPER wire format (deterministic with this schema).
+--   For legacy RLP, use the model-to-RLP layer instead (see gmser_asn1_rlp).
 --
--- Notes on type mapping from gmserialization.erl:
---   - int     -> INTEGER          (new data uses canonical DER INTEGER)
+-- Notes on type mapping from gmserialization.erl (static templates):
+--   - int (bignum) -> BigInt ::= INTEGER (0..MAX)
+--     In practice used for amounts/balances in Pucks (up to 1*10^30).
+--     Encoded as non-negative bignum.
+--   - New smaller integer types will be added to templates (uint64, uint32, ...)
+--     for cases where range is known → enables tighter UPER packing.
 --   - binary  -> OCTET STRING
 --   - bool    -> BOOLEAN
---   - id      -> Id (SEQUENCE)    (cleaner than the legacy packed 33-byte form)
+--   - id      -> Id (SEQUENCE)
 --   - [T]     -> SEQUENCE OF T
 --   - {T1,...} -> SEQUENCE        (fixed-size, order matters)
 --   - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
@@ -33,9 +44,6 @@
 --   - tag + vsn are preserved as the first two fields for compatibility
 --     with existing dispatch logic.
 --
---   Legacy integers had strict unsigned-minimal no-leading-zero (except for 0)
---   and RLP-level rules. New DER data does not need to follow those rules.
---
 -- WARNING:
 --   Any cryptographic hash or signature computed over a serialized object
 --   will change when switching from RLP to DER for that object. Plan a
@@ -53,11 +61,20 @@ EXPORTS ALL;
 -- ============================================================
 
 GajumaruData ::= SEQUENCE {
-    tag     INTEGER,
-    vsn     INTEGER,
+    -- Constrained for better PER packing
+    tag     INTEGER (0..65535),
+    vsn     INTEGER (0..255),
     content Content
 }
 
+-- Preferred top-level for the compact static wire format.
+-- Avoids the extra Content CHOICE tag when the structure is known to be static.
+CompactStatic ::= SEQUENCE {
+    tag    INTEGER (0..65535),
+    vsn    INTEGER (0..255),
+    fields StaticFields
+}
+
 -- Content can be a specific structured type (preferred) or a generic
 -- representation of a template-driven object.
 Content ::= CHOICE {
@@ -65,6 +82,10 @@ Content ::= CHOICE {
     -- without having a pre-defined SEQUENCE for every object.
     templateFields  [0] TemplateFields,
 
+    -- Static-optimized: no field names on wire (matches legacy static behavior exactly)
+    -- Preferred for compact wire format of known templates.
+    staticFields    [10] StaticFields,
+
     -- Examples of concrete versioned types (extend as needed)
     account         [1] Account,
     signedTx        [2] SignedTx,
@@ -87,17 +108,41 @@ TemplateField ::= SEQUENCE {
     value   Value
 }
 
+-- Optimized for static wire format: just the values in order, no names.
+-- This matches the legacy static encoding where maps/records are positional only.
+StaticFields ::= SEQUENCE OF Value
+
+
 Value ::= CHOICE {
-    intValue    [0] INTEGER,
+    -- "int" in static templates is used for bignums in practice
+    -- (e.g. balances and amounts in Pucks, up to 1*10^30).
+    -- We keep it as unbounded non-negative integer.
+    bigIntValue [0] BigInt,
+
     boolValue   [1] BOOLEAN,
     binaryValue [2] OCTET STRING,
     idValue     [3] Id,
     listValue   [4] SEQUENCE OF Value,
     tupleValue  [5] SEQUENCE OF Value,
-    mapValue    [6] SEQUENCE OF KeyValue   -- only needed if you want to
-                                           -- represent dynamic-style maps
+    mapValue    [6] SEQUENCE OF KeyValue,   -- only needed if you want to
+                                            -- represent dynamic-style maps
+
+    -- Additional integer types for smaller ranges (future use in templates
+    -- for better UPER packing when the range is known).
+    uint64Value  [7] Uint64,
+    uint32Value  [8] Uint32,
+    uint128Value [9] Uint128
 }
 
+-- Bignum integer (non-negative). Used for the traditional "int" in static
+-- templates. Max practical value mentioned: 1*10^30 (≈ 2^100 bits).
+BigInt  ::= INTEGER (0..MAX)
+
+-- Convenience sized unsigned integer types.
+Uint64  ::= INTEGER (0..18446744073709551615)
+Uint32  ::= INTEGER (0..4294967295)
+Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
+
 KeyValue ::= SEQUENCE {
     key     Value,
     val     Value
@@ -203,23 +248,20 @@ TypeInfoV3 ::= SEQUENCE {
 --    always falling back to TemplateFields. This gives you better
 --    validation and generated types.
 --
--- 3. INTEGER in DER is signed and uses a different minimal encoding
---    than the legacy unsigned big-endian no leading zero form. This is
---    fine for new data.
+-- 3. For the compact wire format we use UPER (unaligned PER).
+--    It is stable/deterministic for a given schema (no extensibility
+--    markers on these types, fixed order).
 --
--- 4. If you need to preserve exact legacy integer wire bytes inside
---    the new format (e.g. for some hash preimage reason), you can
---    carry selected integers as OCTET STRING (legacy unsigned minimal bytes).
+-- 4. INTEGER uses ASN.1 PER encoding (canonical for the constraints).
+--    This may differ from legacy RLP minimal unsigned; new format
+--    will have different hashes (expected when introducing new encoding).
 --
--- 5. Dynamic encoding (gmser_dyn) is intentionally not modeled here
---    in full, because it is runtime-schema driven (type codes 246-255,
---    labels, alt/switch, etc.). You can still use the generic
---    TemplateFields + Value for some dynamic cases, or model specific
---    message schemas as additional CHOICE arms.
+-- 5. Dynamic encoding (gmser_dyn) is not in scope here.
 --
--- 6. Compile with DER for canonical output:
---       asn1ct:compile(GajumaruSerialization, [der]).
+-- 6. To generate the compact wire:
+--       asn1ct:compile(GajumaruSerialization, [uper]).
+--       {ok, CompactBytes} = 'GajumaruSerialization':encode('GajumaruData', Value).
 --
---    The ber option is also accepted; der implies the stricter rules.
+--    Use staticFields (not templateFields) for best compactness on static data.
 
 END