QPQ-AG/gmserialization
11
0
Fork 0
Code Issues Pull Requests 7 Actions Packages Projects Releases 2 Wiki Activity

Proof-of-concept implementation of ASN.1 mapping #62

Open
uwiger wants to merge 12 commits from uw-asn1 into master
pull from: uw-asn1
merge into: :master
Conversation 0 Commits 12 Files Changed 39
+13445 -20
uwiger commented 2026-07-07 17:07:03 +09:00
Owner
Copy Link
Copy Source

This branch contains a proof-of-concept exploration of representing
the RLP (gmser_rlp) and gmserialization layers using ASN.1 with
DER encoding.

Goals of the experiment:

  • Model the static serialization templates and dynamic types in ASN.1.
  • Demonstrate reliable detection of legacy RLP data vs. new DER data
    (first byte >= 0xC0 for RLP lists vs. 0x30 for DER SEQUENCE).
  • Evaluate wire-size overhead of standard DER TLV encoding compared
    to the extremely compact RLP prefix encoding.
  • Verify that the DER path produces deterministic output suitable
    for hashing (idempotent serialization).

Key files:

  • GajumaruSerialization.asn -- the ASN.1 schema
  • GajumaruSerialization.{erl,hrl} -- generated encoder/decoder
  • detect_demo.erl, size_comparison.erl -- supporting test code

Findings (high level):

  • Detection via first byte works cleanly.
  • DER is fully deterministic for hashing when the same value is given.
  • Small objects pay a significant size overhead (often 3-6x) due to
    TLV tags/lengths and field names in the generic path.
  • Concrete SEQUENCE definitions are much more compact than the
    generic templateFields representation.
  • Large payloads have acceptable relative overhead.

This is NOT production code. It was created to answer feasibility
questions around ASN.1 modeling, format detection, determinism,
and size characteristics.

Branch created from uw-new-acs for isolated experimentation.

This branch contains a proof-of-concept exploration of representing the RLP (gmser_rlp) and gmserialization layers using ASN.1 with DER encoding. Goals of the experiment: - Model the static serialization templates and dynamic types in ASN.1. - Demonstrate reliable detection of legacy RLP data vs. new DER data (first byte >= 0xC0 for RLP lists vs. 0x30 for DER SEQUENCE). - Evaluate wire-size overhead of standard DER TLV encoding compared to the extremely compact RLP prefix encoding. - Verify that the DER path produces deterministic output suitable for hashing (idempotent serialization). Key files: - GajumaruSerialization.asn -- the ASN.1 schema - GajumaruSerialization.{erl,hrl} -- generated encoder/decoder - detect_demo.erl, size_comparison.erl -- supporting test code Findings (high level): - Detection via first byte works cleanly. - DER is fully deterministic for hashing when the same value is given. - Small objects pay a significant size overhead (often 3-6x) due to TLV tags/lengths and field names in the generic path. - Concrete SEQUENCE definitions are much more compact than the generic templateFields representation. - Large payloads have acceptable relative overhead. This is NOT production code. It was created to answer feasibility questions around ASN.1 modeling, format detection, determinism, and size characteristics. Branch created from uw-new-acs for isolated experimentation.
uwiger added 9 commits 2026-07-07 17:07:03 +09:00
Add support for extended-auth account types
Gajumaru Serialization Tests / tests (push) Successful in 11s
Details
7a8e840793
Add auth tx and account sig store
Gajumaru Serialization Tests / tests (push) Successful in 11s
Details
dbfc013c8a
Add missing tag for proposal_gossip_tx
Gajumaru Serialization Tests / tests (push) Successful in 11s
Details
a368c64f7e
Add is_account/1 and account_pubkey/1 helpers
Gajumaru Serialization Tests / tests (push) Successful in 11s
Details
a84dcc880d 
Extended-auth account ids use an {account, Subtype} tag. These helpers
let callers test account membership and extract the 32-byte pubkey
without pattern-matching on specialize/1 results.
Add account_auth_update_tx tag
Gajumaru Serialization Tests / tests (push) Successful in 11s
Details
79b43b0665
Add ac_receipt and AC side tx tags from ac793dcaf6
Gajumaru Serialization Tests / tests (push) Successful in 12s
Details
faa20bd00b 
Cherry-pick the chain-object tags from ac793dcaf6 while keeping the
extended-auth account types (account_sig_store, auth_tx, etc.).
Add ac_acct_state serialization tag
Gajumaru Serialization Tests / tests (push) Successful in 10s
Details
0e157f824a
Fix ac_acct_state tag and restore key_block mapping
Gajumaru Serialization Tests / tests (push) Successful in 11s
Details
c80505c810 
The previous commit accidentally removed key_block (tag 100) instead of
adding ac_acct_state (tag 99).
PoC: ASN.1 / DER serialization experiment
Gajumaru Serialization Tests / tests (push) Successful in 10s
Details
6d7ab1e4ad 
This branch contains a proof-of-concept exploration of representing
the RLP (gmser_rlp) and gmserialization layers using ASN.1 with
DER encoding.

Goals of the experiment:
- Model the static serialization templates and dynamic types in ASN.1.
- Demonstrate reliable detection of legacy RLP data vs. new DER data
  (first byte >= 0xC0 for RLP lists vs. 0x30 for DER SEQUENCE).
- Evaluate wire-size overhead of standard DER TLV encoding compared
  to the extremely compact RLP prefix encoding.
- Verify that the DER path produces deterministic output suitable
  for hashing (idempotent serialization).

Key files:
- GajumaruSerialization.asn   -- the ASN.1 schema
- GajumaruSerialization.{erl,hrl} -- generated encoder/decoder
- detect_demo.erl, size_comparison.erl -- supporting test code

Findings (high level):
- Detection via first byte works cleanly.
- DER is fully deterministic for hashing when the same value is given.
- Small objects pay a significant size overhead (often 3-6x) due to
  TLV tags/lengths and field names in the generic path.
- Concrete SEQUENCE definitions are much more compact than the
  generic templateFields representation.
- Large payloads have acceptable relative overhead.

This is NOT production code. It was created to answer feasibility
questions around ASN.1 modeling, format detection, determinism,
and size characteristics.

Branch created from uw-new-acs for isolated experimentation.
uwiger requested review from zxq9 2026-07-07 17:07:03 +09:00
uwiger added 1 commit 2026-07-07 17:50:31 +09:00
Add thin ASN.1-driven RLP production layer + equivalence tests
Gajumaru Serialization Tests / tests (push) Successful in 10s
Details
9817af8a46 
Implements gmser_asn1_rlp:encode/1 that accepts terms shaped according
to the GajumaruSerialization.asn model (e.g. GajumaruData with
templateFields or concrete CHOICEs like signedTx) and emits exactly
the same RLP bytes as the legacy gmserialization + gmser_rlp stack.

Purpose:
- Use ASN.1 as a formal, multi-language friendly schema and type model.
- Provide a thin, portable "production layer" that other languages can
  implement (ASN.1-generated types + equivalent RLP emitter).
- Preserve the existing compact RLP wire format and on-the-wire
  equivalence (no DER bloat for the legacy path).

The implementation walks the ASN.1-shaped value and applies the same
encoding rules as gmserialization:encode_field/2 (minimal unsigned
integers, positional values for static maps, etc.) followed by
gmser_rlp:encode/1.

Includes EUnit equivalence tests covering:
- Simple fields, zero/empty values
- Lists and tuples
- List-of-tuples (e.g. type_info)
- Concrete signedTx
- ContractV3 (including bool in structured data)

All tests assert byte-for-byte identity with legacy serialization and
that the resulting bytes are still accepted by legacy decoders.

This continues the proof-of-concept on the uw-asn1 branch.
uwiger added 1 commit 2026-07-08 17:53:43 +09:00
Updated asn1 experiment, now exploring PER and OER
Gajumaru Serialization Tests / tests (push) Successful in 12s
Details
4036133476
uwiger added 1 commit 2026-07-22 18:27:24 +09:00
Add ASN.1 schema export from static serialization templates.
Gajumaru Serialization Tests / tests (push) Successful in 48m24s
Details
b2e9af0acb 
Introduce gmser_schema_export to emit portable ASN.1 type definitions
from tag/version/template triples (RLP remains the wire format). Export
tag/1 and rev_tag/1 for discovery and naming, with docs and eunit coverage.
All checks were successful
Gajumaru Serialization Tests / tests (push) Successful in 48m24s
Details
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin uw-asn1:uw-asn1
git checkout uw-asn1
Sign in to join this conversation.
Reviewers
No Reviewers
zxq9
Labels
Clear labels
bug
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 pig lipstick
Muggle-facing enhancements
 question
More information is needed
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Dagadunn Pickmick apex132 dimitar.p.ivanov flambard (Markus Flambard) gchewy hans_sv pharpend spivee (Jarvis Carroll) thescientress (Cecille de Jesus) uwiger (Ulf Wiger) zxq9 (Craig Everett)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: QPQ-AG/gmserialization#62