Ulf Wiger
6d7ab1e4ad
Gajumaru Serialization Tests / tests (push) Successful in 10s
This branch contains a proof-of-concept exploration of representing
the RLP (gmser_rlp) and gmserialization layers using ASN.1 with
DER encoding.
Goals of the experiment:
- Model the static serialization templates and dynamic types in ASN.1.
- Demonstrate reliable detection of legacy RLP data vs. new DER data
(first byte >= 0xC0 for RLP lists vs. 0x30 for DER SEQUENCE).
- Evaluate wire-size overhead of standard DER TLV encoding compared
to the extremely compact RLP prefix encoding.
- Verify that the DER path produces deterministic output suitable
for hashing (idempotent serialization).
Key files:
- GajumaruSerialization.asn -- the ASN.1 schema
- GajumaruSerialization.{erl,hrl} -- generated encoder/decoder
- detect_demo.erl, size_comparison.erl -- supporting test code
Findings (high level):
- Detection via first byte works cleanly.
- DER is fully deterministic for hashing when the same value is given.
- Small objects pay a significant size overhead (often 3-6x) due to
TLV tags/lengths and field names in the generic path.
- Concrete SEQUENCE definitions are much more compact than the
generic templateFields representation.
- Large payloads have acceptable relative overhead.
This is NOT production code. It was created to answer feasibility
questions around ASN.1 modeling, format detection, determinism,
and size characteristics.
Branch created from uw-new-acs for isolated experimentation.
226 lines
7.7 KiB
Plaintext
226 lines
7.7 KiB
Plaintext
-- GajumaruSerialization.asn
|
|
--
|
|
-- ASN.1 model for the structures serialized by gmserialization.erl
|
|
-- (the layer on top of RLP).
|
|
--
|
|
-- Purpose:
|
|
-- * Provide a formal, toolable description of the data.
|
|
-- * Enable migration from the legacy RLP-based format to DER.
|
|
--
|
|
-- Detection of legacy vs new:
|
|
-- Legacy data (produced by gmserialization:serialize / gmser_chain_objects:serialize
|
|
-- or gmser_dyn:serialize) is always an RLP *list* at the top level.
|
|
-- This means the first byte is in the range 0xC0 .. 0xFF.
|
|
--
|
|
-- DER-encoded data using the types below will start with 0x30 (SEQUENCE,
|
|
-- constructed, universal tag) for the outermost GajumaruData (or any
|
|
-- top-level SEQUENCE we define). 0x30 < 0xC0, so a single-byte prefix
|
|
-- check reliably distinguishes the two formats.
|
|
--
|
|
-- Usage in Erlang (after compiling with asn1ct):
|
|
-- {ok, Mod} = asn1ct:compile(GajumaruSerialization.asn, [ber, der]).
|
|
-- {ok, Term} = 'GajumaruSerialization':decode('GajumaruData', DerBinary).
|
|
--
|
|
-- Notes on type mapping from gmserialization.erl:
|
|
-- - int -> INTEGER (new data uses canonical DER INTEGER)
|
|
-- - binary -> OCTET STRING
|
|
-- - bool -> BOOLEAN
|
|
-- - id -> Id (SEQUENCE) (cleaner than the legacy packed 33-byte form)
|
|
-- - [T] -> SEQUENCE OF T
|
|
-- - {T1,...} -> SEQUENCE (fixed-size, order matters)
|
|
-- - #{items := [{K,T},...]} -> SEQUENCE with the fields in template order
|
|
-- (static maps carry *values only*, no keys on the wire)
|
|
-- - tag + vsn are preserved as the first two fields for compatibility
|
|
-- with existing dispatch logic.
|
|
--
|
|
-- Legacy integers had strict unsigned-minimal no-leading-zero (except for 0)
|
|
-- and RLP-level rules. New DER data does not need to follow those rules.
|
|
--
|
|
-- WARNING:
|
|
-- Any cryptographic hash or signature computed over a serialized object
|
|
-- will change when switching from RLP to DER for that object. Plan a
|
|
-- coordinated upgrade.
|
|
|
|
GajumaruSerialization DEFINITIONS
|
|
AUTOMATIC TAGS ::=
|
|
BEGIN
|
|
|
|
EXPORTS ALL;
|
|
|
|
-- ============================================================
|
|
-- Top-level wrapper used for new DER data.
|
|
-- This is what a decoder will see first.
|
|
-- ============================================================
|
|
|
|
GajumaruData ::= SEQUENCE {
|
|
tag INTEGER,
|
|
vsn INTEGER,
|
|
content Content
|
|
}
|
|
|
|
-- Content can be a specific structured type (preferred) or a generic
|
|
-- representation of a template-driven object.
|
|
Content ::= CHOICE {
|
|
-- Generic fallback that can represent any [{Field, Type}] template
|
|
-- without having a pre-defined SEQUENCE for every object.
|
|
templateFields [0] TemplateFields,
|
|
|
|
-- Examples of concrete versioned types (extend as needed)
|
|
account [1] Account,
|
|
signedTx [2] SignedTx,
|
|
contract [3] ContractCode
|
|
-- Add more alternatives for other tags from gmser_chain_objects
|
|
}
|
|
|
|
-- ============================================================
|
|
-- Generic template-driven representation
|
|
-- (useful during transition or for unregistered types)
|
|
-- ============================================================
|
|
|
|
TemplateFields ::= SEQUENCE OF TemplateField
|
|
|
|
TemplateField ::= SEQUENCE {
|
|
-- Field name is included for debuggability / generic processing.
|
|
-- In the original static encoding the name is NOT on the wire;
|
|
-- only position and type matter. We include it here for convenience.
|
|
name IA5String OPTIONAL,
|
|
value Value
|
|
}
|
|
|
|
Value ::= CHOICE {
|
|
intValue [0] INTEGER,
|
|
boolValue [1] BOOLEAN,
|
|
binaryValue [2] OCTET STRING,
|
|
idValue [3] Id,
|
|
listValue [4] SEQUENCE OF Value,
|
|
tupleValue [5] SEQUENCE OF Value,
|
|
mapValue [6] SEQUENCE OF KeyValue -- only needed if you want to
|
|
-- represent dynamic-style maps
|
|
}
|
|
|
|
KeyValue ::= SEQUENCE {
|
|
key Value,
|
|
val Value
|
|
}
|
|
|
|
-- ============================================================
|
|
-- Common types
|
|
-- ============================================================
|
|
|
|
Id ::= SEQUENCE {
|
|
-- Corresponds to the simple tags in gmser_id (account=1, name=2, etc.)
|
|
-- and the extended account subtype (high bit in legacy).
|
|
type INTEGER (0..255),
|
|
value OCTET STRING (SIZE (32))
|
|
}
|
|
|
|
-- ============================================================
|
|
-- Concrete object examples (derived from usage in the codebase)
|
|
-- Add / evolve per version as you introduce new vsns.
|
|
-- ============================================================
|
|
|
|
-- Example: a very simple account-like object used in tests
|
|
Account ::= SEQUENCE {
|
|
foo INTEGER,
|
|
bar OCTET STRING
|
|
}
|
|
|
|
-- Simplified signed transaction
|
|
SignedTx ::= SEQUENCE {
|
|
signatures SEQUENCE OF OCTET STRING,
|
|
transaction OCTET STRING
|
|
}
|
|
|
|
-- Contract code objects (see gmser_contract_code.erl)
|
|
-- We model the three versions that exist today.
|
|
ContractCode ::= CHOICE {
|
|
v1 [0] ContractV1,
|
|
v2 [1] ContractV2,
|
|
v3 [2] ContractV3
|
|
}
|
|
|
|
ContractV1 ::= SEQUENCE {
|
|
sourceHash OCTET STRING,
|
|
-- typeInfo is a list of 4-tuples in legacy:
|
|
-- {typeHash, name, argType, outType}
|
|
typeInfo SEQUENCE OF TypeInfoV1,
|
|
byteCode OCTET STRING
|
|
}
|
|
|
|
ContractV2 ::= SEQUENCE {
|
|
sourceHash OCTET STRING,
|
|
typeInfo SEQUENCE OF TypeInfoV1,
|
|
byteCode OCTET STRING,
|
|
compilerVersion OCTET STRING
|
|
}
|
|
|
|
ContractV3 ::= SEQUENCE {
|
|
sourceHash OCTET STRING,
|
|
typeInfo SEQUENCE OF TypeInfoV3,
|
|
byteCode OCTET STRING,
|
|
compilerVersion OCTET STRING,
|
|
payable BOOLEAN
|
|
}
|
|
|
|
TypeInfoV1 ::= SEQUENCE {
|
|
typeHash OCTET STRING,
|
|
name OCTET STRING,
|
|
argType OCTET STRING,
|
|
outType OCTET STRING
|
|
}
|
|
|
|
TypeInfoV3 ::= SEQUENCE {
|
|
typeHash OCTET STRING,
|
|
name OCTET STRING,
|
|
payable BOOLEAN,
|
|
argType OCTET STRING,
|
|
outType OCTET STRING
|
|
}
|
|
|
|
-- ============================================================
|
|
-- Notes for implementers
|
|
-- ============================================================
|
|
-- 1. Detection (recommended decoder entry point):
|
|
--
|
|
-- decode(Binary) ->
|
|
-- case Binary of
|
|
-- <<B, _/binary>> when B >= 16#C0 ->
|
|
-- decode_legacy_rlp(Binary); % existing gmser_* path
|
|
-- _ ->
|
|
-- {ok, Term} =
|
|
-- 'GajumaruSerialization':decode('GajumaruData', Binary),
|
|
-- Term
|
|
-- end.
|
|
--
|
|
-- This works because:
|
|
-- - All current top-level output from serialize() is an RLP list
|
|
-- (first byte 0xC0-0xFF).
|
|
-- - GajumaruData and the concrete choices above are SEQUENCEs
|
|
-- (first byte 0x30 for short form, or 0x30 0x81/0x82... for long).
|
|
--
|
|
-- 2. When you add a new object type or version, prefer adding a
|
|
-- concrete SEQUENCE alternative in the Content CHOICE rather than
|
|
-- always falling back to TemplateFields. This gives you better
|
|
-- validation and generated types.
|
|
--
|
|
-- 3. INTEGER in DER is signed and uses a different minimal encoding
|
|
-- than the legacy unsigned big-endian no leading zero form. This is
|
|
-- fine for new data.
|
|
--
|
|
-- 4. If you need to preserve exact legacy integer wire bytes inside
|
|
-- the new format (e.g. for some hash preimage reason), you can
|
|
-- carry selected integers as OCTET STRING (legacy unsigned minimal bytes).
|
|
--
|
|
-- 5. Dynamic encoding (gmser_dyn) is intentionally not modeled here
|
|
-- in full, because it is runtime-schema driven (type codes 246-255,
|
|
-- labels, alt/switch, etc.). You can still use the generic
|
|
-- TemplateFields + Value for some dynamic cases, or model specific
|
|
-- message schemas as additional CHOICE arms.
|
|
--
|
|
-- 6. Compile with DER for canonical output:
|
|
-- asn1ct:compile(GajumaruSerialization, [der]).
|
|
--
|
|
-- The ber option is also accepted; der implies the stricter rules.
|
|
|
|
END
|