Updated asn1 experiment, now exploring PER and OER
Gajumaru Serialization Tests / tests (push) Successful in 12s
Gajumaru Serialization Tests / tests (push) Successful in 12s
This commit is contained in:
Ulf Wiger
@@ -0,0 +1,212 @@
|
||||
# ASN.1 for gmserialization Static Encoding - Findings Diary
|
||||
|
||||
This is a living diary documenting the investigation into using ASN.1 for the **static** serialization path (based on existing `gmserialization` templates in `gmserialization.erl` and `gmser_chain_objects.erl`). Dynamic encoding is out of scope.
|
||||
|
||||
Focus areas:
|
||||
- Modeling static templates with ASN.1 (portability goal).
|
||||
- Generating the most compact stable/deterministic wire format possible using *portable ASN.1 techniques* (UPER etc.).
|
||||
- Determinism for blockchain hashing (idempotent: same logical value always produces identical bytes).
|
||||
- (Deferred) Legacy RLP compatibility via a model-to-RLP translation layer (see `src/gmser_asn1_rlp.erl`).
|
||||
|
||||
The single source of truth is the ASN.1 schema in `GajumaruSerialization.asn`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-08 - Setup and Initial Schema
|
||||
|
||||
- Schema (`asn1/GajumaruSerialization.asn`) models:
|
||||
- `GajumaruData` as top-level (tag + vsn + content).
|
||||
- `Content` CHOICE with `templateFields` (generic) and concrete types (e.g. `SignedTx`, `ContractV*`).
|
||||
- `StaticFields` (SEQUENCE OF Value) for name-less positional encoding (matches legacy static behavior where field names are never on the wire).
|
||||
- `Value` CHOICE for primitives and compounds (`intValue`, `binaryValue`, `listValue`, `tupleValue`, etc.).
|
||||
- Supports all static template types: `int`, `bool`, `binary`, `id`, `[T]`, tuples, `#{items => [...]}`.
|
||||
|
||||
- Initial schema comments were DER-oriented (migration path). Updated to emphasize compact UPER + portability.
|
||||
|
||||
- Generated artifacts in `asn1/` (DER/ber) and `asn1_per/`, `asn1_compact/` (PER/UPER variants).
|
||||
|
||||
- Key files:
|
||||
- `asn1/GajumaruSerialization.asn` (source)
|
||||
- `src/gmser_asn1_rlp.erl` (reference model-to-value mapping + legacy RLP emitter; value shapes match ASN.1)
|
||||
- Tests in `test/gmser_chain_objects_tests.erl` and inside `gmser_asn1_rlp` for equivalence.
|
||||
|
||||
## 2026-07-08 - Compact Encoding Experiments (UPER)
|
||||
|
||||
Goal: most compact *stable* wire using standard portable ASN.1 (not custom non-portable rules, not RLP).
|
||||
|
||||
- Tried standard DER → too verbose (tiny case: ~36 bytes vs legacy RLP 5 bytes).
|
||||
- Switched to **UPER (Unaligned PER)** — the most compact *standard* ASN.1 encoding rule with good canonical/deterministic properties.
|
||||
- Compiled via `asn1ct:compile(..., [uper])`.
|
||||
- Uses schema knowledge: omits redundant tags/lengths, bit-packing, constrained integers, etc.
|
||||
- Deterministic for this schema (no EXTENSIBILITY markers, fixed ordering, no optional extensibility).
|
||||
|
||||
- Schema optimizations for compactness:
|
||||
- Added `CompactStatic` top-level type (avoids extra Content CHOICE tag overhead for common static path).
|
||||
- `staticFields` (pure `SEQUENCE OF Value`) — no IA5String names on wire.
|
||||
- Constrained `tag`/`vsn` (INTEGER (0..65535), (0..255)) for better packing.
|
||||
- Prefer concrete SEQUENCES (e.g. `SignedTx`) or `staticFields` over generic `templateFields` (names add cost).
|
||||
- Kept `TemplateFields` for debug/transition only.
|
||||
|
||||
- Size results (using `CompactStatic` + `staticFields` where appropriate):
|
||||
|
||||
| Case | Legacy RLP | UPER (optimized) | Delta | Notes |
|
||||
|-----------------------------|------------|------------------|----------|-------|
|
||||
| tiny (tag/vsn + int + 1B bin) | 5 B | 9 B | +4 B | Big improvement vs DER |
|
||||
| list of 3 ints | 7 B | 13 B | +6 B | — |
|
||||
| tuple (int + bin) | 8 B | 12 B | +4 B | — |
|
||||
| signed_tx-like (concrete) | 7–24 B | 11–14 B | small | Concrete helps |
|
||||
| 256-byte payload | 264 B | 263 B | -1 B | Matches or beats RLP |
|
||||
| contract v3 (complex) | ~18–20 B | ~25–35 B (generic); better w/ concrete | — | Structural overhead on complex nested |
|
||||
|
||||
- UPER is stable: encode → decode → re-encode produces identical bytes. Roundtrips work.
|
||||
|
||||
- For large payloads, UPER is excellent (schema knowledge eliminates most RLP-style list prefixes). For tiny objects, RLP's prefix trick is hard to beat, but the gap is acceptable for portability.
|
||||
|
||||
- OER (Octet Encoding Rules) also compiled but was larger (18 B on tiny case).
|
||||
|
||||
## 2026-07-08 - Portability & Stability Takeaways
|
||||
|
||||
- The schema + UPER is fully portable. Other languages can:
|
||||
1. Compile the `.asn` with their ASN.1 tool.
|
||||
2. Build a value matching `CompactStatic` / `staticFields` / concrete types.
|
||||
3. Call their UPER encoder → identical compact bytes.
|
||||
|
||||
- No Erlang-specific runtime required for the new wire format.
|
||||
- Determinism comes from:
|
||||
- UPER canonical packing rules.
|
||||
- Constrained types in schema.
|
||||
- Explicit staticFields (no map iteration, names omitted).
|
||||
- Same rules as legacy for ints (minimal), ordering, etc.
|
||||
|
||||
- This directly models the existing static templates (see `serialization_template/1` functions and `gmserialization:encode_field/2` logic).
|
||||
- Concrete types in schema give best compactness for known objects.
|
||||
- Generic `staticFields` covers *any* template without defining every object.
|
||||
|
||||
## Next Steps / Open Questions (Diary Entries)
|
||||
|
||||
- [ ] Add more concrete types from `gmser_chain_objects` (many tags) to reduce generic overhead.
|
||||
- [ ] Add more ASN.1 constraints (SIZE, value ranges) to help PER pack tighter.
|
||||
- [ ] Measure on real on-chain objects (key_block, etc.).
|
||||
- [ ] Decide on top-level header for the new format (keep tag/vsn?).
|
||||
- [ ] (Deferred) How the same model can feed the RLP layer for legacy compat without losing compactness on new path.
|
||||
- [ ] Consider if a custom "ASN.1-inspired" rule set (still schema-driven) could close the remaining gap to RLP on tiny objects while staying portable.
|
||||
|
||||
## 2026-07-08 - Schema Updated for Bignums
|
||||
|
||||
- Updated `GajumaruSerialization.asn`:
|
||||
- Introduced `BigInt ::= INTEGER (0..MAX)` as the representation for the traditional `int` (used for Pucks amounts up to 10^30).
|
||||
- `Value` CHOICE now uses `bigIntValue` for the bignum case.
|
||||
- Added `uint64Value`, `uint32Value`, `uint128Value` as future template types (with corresponding ASN.1 subtypes).
|
||||
- Updated header comments to document the bignum nature of `int`.
|
||||
|
||||
- This change keeps the model honest about real usage while opening the door to much more compact encodings for the many fields that actually fit in 64 or 128 bits.
|
||||
|
||||
- Next: We should extend the Erlang-side `type()` in `gmserialization.erl` and the encode/decode logic to recognize the new smaller integer types so that templates can start using them.
|
||||
|
||||
---
|
||||
|
||||
*This file should be kept as a living diary. Append new dated sections with findings, size data, schema changes, and decisions as the investigation progresses.*
|
||||
|
||||
## 2026-07-08 - Handling of `int` as Bignums (Pucks, amounts, etc.)
|
||||
|
||||
Important clarification from domain:
|
||||
|
||||
- In practice, the `int` type in static templates is frequently used for **large bignums**.
|
||||
- Example: Amount fields (balances, transaction amounts, etc.) are denominated in "Pucks".
|
||||
- Maximum value mentioned: 1 × 10^30.
|
||||
- This is ~ 2^99.66, i.e., requires up to ~13 bytes in minimal unsigned encoding.
|
||||
|
||||
Current legacy handling (in `gmserialization.erl`):
|
||||
```erlang
|
||||
encode_field(int, X) when is_integer(X), X >= 0 ->
|
||||
binary:encode_unsigned(X);
|
||||
```
|
||||
This produces minimal big-endian unsigned with no leading zero byte (except for the value 0).
|
||||
|
||||
Implications for ASN.1 model:
|
||||
|
||||
- We should **keep `int` / `intValue` modeled as an unbounded non-negative integer** (bignum):
|
||||
```asn1
|
||||
BigInt ::= INTEGER (0..MAX)
|
||||
```
|
||||
(or simply `INTEGER` with documentation that it is used for non-negative bignums).
|
||||
|
||||
- Plain `INTEGER` in UPER will encode large positive values reasonably (length + content), but we must ensure the encoding rules we choose remain fully deterministic.
|
||||
|
||||
- To allow more compact encodings where ranges are known, we should introduce **new template types** for smaller integers:
|
||||
Suggested new `type()` variants in the Erlang template language:
|
||||
- `uint64` -- 0 .. 2^64-1
|
||||
- `uint32` -- 0 .. 2^32-1
|
||||
- `uint16`, `uint8`, `uint128` etc. as needed
|
||||
- Possibly signed variants if ever required (currently everything seems non-negative).
|
||||
|
||||
Corresponding in ASN.1 (inside Value CHOICE or as reusable types):
|
||||
```asn1
|
||||
Uint64 ::= INTEGER (0..18446744073709551615)
|
||||
Uint32 ::= INTEGER (0..4294967295)
|
||||
Uint128 ::= INTEGER (0..340282366920938463463374607431768211455)
|
||||
```
|
||||
|
||||
- In the schema's Value CHOICE we can evolve to:
|
||||
```asn1
|
||||
Value ::= CHOICE {
|
||||
bigIntValue [0] BigInt, -- the classic "int" for Pucks etc.
|
||||
uint64Value [7] Uint64,
|
||||
uint32Value [8] Uint32,
|
||||
...
|
||||
-- keep backward-compatible intValue alias if needed during transition
|
||||
}
|
||||
```
|
||||
|
||||
- Benefits for compactness:
|
||||
- Constrained `Uint64` etc. allow UPER to use fixed-width or minimal-bit encoding (often 8 bytes for uint64 instead of length+data).
|
||||
- Still fully portable and deterministic.
|
||||
|
||||
- Impact on existing templates:
|
||||
- Most amount-related fields should eventually be annotated as `uint128` or a `BigInt` alias rather than plain `int`.
|
||||
- Small counters, versions, indices can use `uint32` / `uint64`.
|
||||
- This may require extending the `type()` in `gmserialization.erl` and the encoders.
|
||||
|
||||
## 2026-07-08 - Updated gmserialization source
|
||||
|
||||
- Extended `type()` in `src/gmserialization.erl` with `uint128`, `uint64`, `uint32`, `uint16`, `uint8` (keeping `int` for bignums).
|
||||
- Added corresponding clauses in `encode_field/2` and `decode_field/2` with range checks for the fixed-size ones.
|
||||
- Updated `src/gmser_asn1_rlp.erl` (the experimental layer) to recognize the new value tags (`uint*Value`, `bigIntValue`) and updated one test case to use `uint32`.
|
||||
- Updated `doc/static.md` example types.
|
||||
- `int` remains fully backward compatible for bignum use.
|
||||
- All existing tests + new type usage pass.
|
||||
|
||||
- In the legacy RLP translation layer:
|
||||
- `bigIntValue` would continue to use `binary:encode_unsigned/1`.
|
||||
- Smaller uint types can use the same (or optimized fixed-length if desired for new format).
|
||||
|
||||
Next action items:
|
||||
- Update `GajumaruSerialization.asn` to introduce `BigInt`, `Uint*` types and adjust Value.
|
||||
- Decide on naming in the Erlang template DSL (`int` remains bignum alias? or rename?).
|
||||
- Add example in the schema for a balance field.
|
||||
- Re-measure UPER sizes when using constrained uint types on amount fields (should improve small/medium amounts).
|
||||
|
||||
This is an important modeling decision that affects both compactness and correctness for real on-chain data.
|
||||
|
||||
## 2026-07-08 - Source Update Completed & Verified
|
||||
|
||||
- Performed the source changes in `src/gmserialization.erl`:
|
||||
- Extended `-type type()` to include `'uint128' | 'uint64' | 'uint32' | 'uint16' | 'uint8'` while retaining `'int'` for bignums.
|
||||
- Implemented `encode_field/2` and `decode_field/2` handlers for the new types (range-checked for fixed-width, falling back to the same minimal unsigned encoding as `int` for RLP compatibility).
|
||||
- Synced `src/gmser_asn1_rlp.erl`:
|
||||
- Added support for new ASN.1 value variants (`{uint*Value, ...}`, `{bigIntValue, ...}`) in `encode_asn1_value/1`.
|
||||
- Updated test data and comments to use the new types.
|
||||
- Updated `doc/static.md` to reflect the extended type language.
|
||||
- Verified:
|
||||
- Legacy templates using `int` continue to work unchanged.
|
||||
- New types (e.g. `{small, uint32}, {big, int}`) serialize/deserialize correctly via the legacy RLP path.
|
||||
- All 7 equivalence tests in `gmser_asn1_rlp` still pass.
|
||||
- `gmser_chain_objects_tests` (12 tests) still pass.
|
||||
- The ASN.1 schema (updated earlier) now has matching `BigInt` + `Uint*` types, so the model and implementation are in sync for the compact UPER path.
|
||||
|
||||
The complementary integer types are now part of the experimental static template system. This enables the ASN.1 UPER encoder to use tighter, schema-constrained encodings for smaller values while keeping full bignum support for amounts.
|
||||
|
||||
Next diary items (still open):
|
||||
- Extend more concrete types in the schema.
|
||||
- Add actual UPER-based encode path in the library (beyond the RLP layer).
|
||||
- Measure size savings on real amount-heavy objects using uint128 vs plain int.
|
||||
Reference in New Issue
Block a user