Ulf Wiger
212 lines
12 KiB
Markdown
212 lines
12 KiB
Markdown
# ASN.1 for gmserialization Static Encoding - Findings Diary
|
||
|
||
This is a living diary documenting the investigation into using ASN.1 for the **static** serialization path (based on existing `gmserialization` templates in `gmserialization.erl` and `gmser_chain_objects.erl`). Dynamic encoding is out of scope.
|
||
|
||
Focus areas:
|
||
- Modeling static templates with ASN.1 (portability goal).
|
||
- Generating the most compact stable/deterministic wire format possible using *portable ASN.1 techniques* (UPER etc.).
|
||
- Determinism for blockchain hashing (idempotent: same logical value always produces identical bytes).
|
||
- (Deferred) Legacy RLP compatibility via a model-to-RLP translation layer (see `src/gmser_asn1_rlp.erl`).
|
||
|
||
The single source of truth is the ASN.1 schema in `GajumaruSerialization.asn`.
|
||
|
||
---
|
||
|
||
## 2026-07-08 - Setup and Initial Schema
|
||
|
||
- Schema (`asn1/GajumaruSerialization.asn`) models:
|
||
- `GajumaruData` as top-level (tag + vsn + content).
|
||
- `Content` CHOICE with `templateFields` (generic) and concrete types (e.g. `SignedTx`, `ContractV*`).
|
||
- `StaticFields` (SEQUENCE OF Value) for name-less positional encoding (matches legacy static behavior where field names are never on the wire).
|
||
- `Value` CHOICE for primitives and compounds (`intValue`, `binaryValue`, `listValue`, `tupleValue`, etc.).
|
||
- Supports all static template types: `int`, `bool`, `binary`, `id`, `[T]`, tuples, `#{items => [...]}`.
|
||
|
||
- Initial schema comments were DER-oriented (migration path). Updated to emphasize compact UPER + portability.
|
||
|
||
- Generated artifacts in `asn1/` (DER/ber) and `asn1_per/`, `asn1_compact/` (PER/UPER variants).
|
||
|
||
- Key files:
|
||
- `asn1/GajumaruSerialization.asn` (source)
|
||
- `src/gmser_asn1_rlp.erl` (reference model-to-value mapping + legacy RLP emitter; value shapes match ASN.1)
|
||
- Tests in `test/gmser_chain_objects_tests.erl` and inside `gmser_asn1_rlp` for equivalence.
|
||
|
||
## 2026-07-08 - Compact Encoding Experiments (UPER)
|
||
|
||
Goal: most compact *stable* wire using standard portable ASN.1 (not custom non-portable rules, not RLP).
|
||
|
||
- Tried standard DER → too verbose (tiny case: ~36 bytes vs legacy RLP 5 bytes).
|
||
- Switched to **UPER (Unaligned PER)** — the most compact *standard* ASN.1 encoding rule with good canonical/deterministic properties.
|
||
- Compiled via `asn1ct:compile(..., [uper])`.
|
||
- Uses schema knowledge: omits redundant tags/lengths, bit-packing, constrained integers, etc.
|
||
- Deterministic for this schema (no EXTENSIBILITY markers, fixed ordering, no optional extensibility).
|
||
|
||
- Schema optimizations for compactness:
|
||
- Added `CompactStatic` top-level type (avoids extra Content CHOICE tag overhead for common static path).
|
||
- `staticFields` (pure `SEQUENCE OF Value`) — no IA5String names on wire.
|
||
- Constrained `tag`/`vsn` (INTEGER (0..65535), (0..255)) for better packing.
|
||
- Prefer concrete SEQUENCES (e.g. `SignedTx`) or `staticFields` over generic `templateFields` (names add cost).
|
||
- Kept `TemplateFields` for debug/transition only.
|
||
|
||
- Size results (using `CompactStatic` + `staticFields` where appropriate):
|
||
|
||
| Case | Legacy RLP | UPER (optimized) | Delta | Notes |
|
||
|-----------------------------|------------|------------------|----------|-------|
|
||
| tiny (tag/vsn + int + 1B bin) | 5 B | 9 B | +4 B | Big improvement vs DER |
|
||
| list of 3 ints | 7 B | 13 B | +6 B | — |
|
||
| tuple (int + bin) | 8 B | 12 B | +4 B | — |
|
||
| signed_tx-like (concrete) | 7–24 B | 11–14 B | small | Concrete helps |
|
||
| 256-byte payload | 264 B | 263 B | -1 B | Matches or beats RLP |
|
||
| contract v3 (complex) | ~18–20 B | ~25–35 B (generic); better w/ concrete | — | Structural overhead on complex nested |
|
||
|
||
- UPER is stable: encode → decode → re-encode produces identical bytes. Roundtrips work.
|
||
|
||
- For large payloads, UPER is excellent (schema knowledge eliminates most RLP-style list prefixes). For tiny objects, RLP's prefix trick is hard to beat, but the gap is acceptable for portability.
|
||
|
||
- OER (Octet Encoding Rules) also compiled but was larger (18 B on tiny case).
|
||
|
||
## 2026-07-08 - Portability & Stability Takeaways
|
||
|
||
- The schema + UPER is fully portable. Other languages can:
|
||
1. Compile the `.asn` with their ASN.1 tool.
|
||
2. Build a value matching `CompactStatic` / `staticFields` / concrete types.
|
||
3. Call their UPER encoder → identical compact bytes.
|
||
|
||
- No Erlang-specific runtime required for the new wire format.
|
||
- Determinism comes from:
|
||
- UPER canonical packing rules.
|
||
- Constrained types in schema.
|
||
- Explicit staticFields (no map iteration, names omitted).
|
||
- Same rules as legacy for ints (minimal), ordering, etc.
|
||
|
||
- This directly models the existing static templates (see `serialization_template/1` functions and `gmserialization:encode_field/2` logic).
|
||
- Concrete types in schema give best compactness for known objects.
|
||
- Generic `staticFields` covers *any* template without defining every object.
|
||
|
||
## Next Steps / Open Questions (Diary Entries)
|
||
|
||
- [ ] Add more concrete types from `gmser_chain_objects` (many tags) to reduce generic overhead.
|
||
- [ ] Add more ASN.1 constraints (SIZE, value ranges) to help PER pack tighter.
|
||
- [ ] Measure on real on-chain objects (key_block, etc.).
|
||
- [ ] Decide on top-level header for the new format (keep tag/vsn?).
|
||
- [ ] (Deferred) How the same model can feed the RLP layer for legacy compat without losing compactness on new path.
|
||
- [ ] Consider if a custom "ASN.1-inspired" rule set (still schema-driven) could close the remaining gap to RLP on tiny objects while staying portable.
|
||
|
||
## 2026-07-08 - Schema Updated for Bignums
|
||
|
||
- Updated `GajumaruSerialization.asn`:
|
||
- Introduced `BigInt ::= INTEGER (0..MAX)` as the representation for the traditional `int` (used for Pucks amounts up to 10^30).
|
||
- `Value` CHOICE now uses `bigIntValue` for the bignum case.
|
||
- Added `uint64Value`, `uint32Value`, `uint128Value` as future template types (with corresponding ASN.1 subtypes).
|
||
- Updated header comments to document the bignum nature of `int`.
|
||
|
||
- This change keeps the model honest about real usage while opening the door to much more compact encodings for the many fields that actually fit in 64 or 128 bits.
|
||
|
||
- Next: We should extend the Erlang-side `type()` in `gmserialization.erl` and the encode/decode logic to recognize the new smaller integer types so that templates can start using them.
|
||
|
||
---
|
||
|
||
*This file should be kept as a living diary. Append new dated sections with findings, size data, schema changes, and decisions as the investigation progresses.*
|
||
|
||
## 2026-07-08 - Handling of `int` as Bignums (Pucks, amounts, etc.)
|
||
|
||
Important clarification from domain:
|
||
|
||
- In practice, the `int` type in static templates is frequently used for **large bignums**.
|
||
- Example: Amount fields (balances, transaction amounts, etc.) are denominated in "Pucks".
|
||
- Maximum value mentioned: 1 × 10^30.
|
||
- This is ~ 2^99.66, i.e., requires up to ~13 bytes in minimal unsigned encoding.
|
||
|
||
Current legacy handling (in `gmserialization.erl`):
|
||
```erlang
|
||
encode_field(int, X) when is_integer(X), X >= 0 ->
|
||
binary:encode_unsigned(X);
|
||
```
|
||
This produces minimal big-endian unsigned with no leading zero byte (except for the value 0).
|
||
|
||
Implications for ASN.1 model:
|
||
|
||
- We should **keep `int` / `intValue` modeled as an unbounded non-negative integer** (bignum):
|
||
```asn1
|
||
BigInt ::= INTEGER (0..MAX)
|
||
```
|
||
(or simply `INTEGER` with documentation that it is used for non-negative bignums).
|
||
|
||
- Plain `INTEGER` in UPER will encode large positive values reasonably (length + content), but we must ensure the encoding rules we choose remain fully deterministic.
|
||
|
||
- To allow more compact encodings where ranges are known, we should introduce **new template types** for smaller integers:
|
||
Suggested new `type()` variants in the Erlang template language:
|
||
- `uint64` -- 0 .. 2^64-1
|
||
- `uint32` -- 0 .. 2^32-1
|
||
- `uint16`, `uint8`, `uint128` etc. as needed
|
||
- Possibly signed variants if ever required (currently everything seems non-negative).
|
||
|
||
Corresponding in ASN.1 (inside Value CHOICE or as reusable types):
|
||
```asn1
|
||
Uint64 ::= INTEGER (0..18446744073709551615)
|
||
Uint32 ::= INTEGER (0..4294967295)
|
||
Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
|
||
```
|
||
|
||
- In the schema's Value CHOICE we can evolve to:
|
||
```asn1
|
||
Value ::= CHOICE {
|
||
bigIntValue [0] BigInt, -- the classic "int" for Pucks etc.
|
||
uint64Value [7] Uint64,
|
||
uint32Value [8] Uint32,
|
||
...
|
||
-- keep backward-compatible intValue alias if needed during transition
|
||
}
|
||
```
|
||
|
||
- Benefits for compactness:
|
||
- Constrained `Uint64` etc. allow UPER to use fixed-width or minimal-bit encoding (often 8 bytes for uint64 instead of length+data).
|
||
- Still fully portable and deterministic.
|
||
|
||
- Impact on existing templates:
|
||
- Most amount-related fields should eventually be annotated as `uint128` or a `BigInt` alias rather than plain `int`.
|
||
- Small counters, versions, indices can use `uint32` / `uint64`.
|
||
- This may require extending the `type()` in `gmserialization.erl` and the encoders.
|
||
|
||
## 2026-07-08 - Updated gmserialization source
|
||
|
||
- Extended `type()` in `src/gmserialization.erl` with `uint128`, `uint64`, `uint32`, `uint16`, `uint8` (keeping `int` for bignums).
|
||
- Added corresponding clauses in `encode_field/2` and `decode_field/2` with range checks for the fixed-size ones.
|
||
- Updated `src/gmser_asn1_rlp.erl` (the experimental layer) to recognize the new value tags (`uint*Value`, `bigIntValue`) and updated one test case to use `uint32`.
|
||
- Updated `doc/static.md` example types.
|
||
- `int` remains fully backward compatible for bignum use.
|
||
- All existing tests + new type usage pass.
|
||
|
||
- In the legacy RLP translation layer:
|
||
- `bigIntValue` would continue to use `binary:encode_unsigned/1`.
|
||
- Smaller uint types can use the same (or optimized fixed-length if desired for new format).
|
||
|
||
Next action items:
|
||
- Update `GajumaruSerialization.asn` to introduce `BigInt`, `Uint*` types and adjust Value.
|
||
- Decide on naming in the Erlang template DSL (`int` remains bignum alias? or rename?).
|
||
- Add example in the schema for a balance field.
|
||
- Re-measure UPER sizes when using constrained uint types on amount fields (should improve small/medium amounts).
|
||
|
||
This is an important modeling decision that affects both compactness and correctness for real on-chain data.
|
||
|
||
## 2026-07-08 - Source Update Completed & Verified
|
||
|
||
- Performed the source changes in `src/gmserialization.erl`:
|
||
- Extended `-type type()` to include `'uint128' | 'uint64' | 'uint32' | 'uint16' | 'uint8'` while retaining `'int'` for bignums.
|
||
- Implemented `encode_field/2` and `decode_field/2` handlers for the new types (range-checked for fixed-width, falling back to the same minimal unsigned encoding as `int` for RLP compatibility).
|
||
- Synced `src/gmser_asn1_rlp.erl`:
|
||
- Added support for new ASN.1 value variants (`{uint*Value, ...}`, `{bigIntValue, ...}`) in `encode_asn1_value/1`.
|
||
- Updated test data and comments to use the new types.
|
||
- Updated `doc/static.md` to reflect the extended type language.
|
||
- Verified:
|
||
- Legacy templates using `int` continue to work unchanged.
|
||
- New types (e.g. `{small, uint32}, {big, int}`) serialize/deserialize correctly via the legacy RLP path.
|
||
- All 7 equivalence tests in `gmser_asn1_rlp` still pass.
|
||
- `gmser_chain_objects_tests` (12 tests) still pass.
|
||
- The ASN.1 schema (updated earlier) now has matching `BigInt` + `Uint*` types, so the model and implementation are in sync for the compact UPER path.
|
||
|
||
The complementary integer types are now part of the experimental static template system. This enables the ASN.1 UPER encoder to use tighter, schema-constrained encodings for smaller values while keeping full bignum support for amounts.
|
||
|
||
Next diary items (still open):
|
||
- Extend more concrete types in the schema.
|
||
- Add actual UPER-based encode path in the library (beyond the RLP layer).
|
||
- Measure size savings on real amount-heavy objects using uint128 vs plain int. |