Skip to content

Output schema v3

Schema name: output_schema_v3
Step indexing meaning: post_election_pre_mutation

The output schema defines the on-disk outputs produced by headless batch runs. Replay remains backward-compatible with both schema v2 and schema v3, but the active writer contract is v3.

Specification

Run directory layout

Per run directory (e.g. .../data/simulation_output/<ts>/run_<i>/):

  • meta.yaml – schema + run metadata, plus config reference
  • config_used.yaml – canonical batch config (stored at batch root; meta.yaml points to it via config_ref)
  • static.json – static model info (grid size, preference-group metadata via personality_groups, file patterns)
  • steps.parquet
  • area_steps.parquet
  • agents.parquet
  • votes.parquet
  • static_cell_areas.parquet – typed cell-to-area overlay
  • static_cell_agents.parquet – typed cell/area/agent overlay
  • grids/grid_<k>.npy snapshots (index k zero-padded to len(str(num_steps)))
  • grid_001 and grid_{S} election-time snapshots are always present
  • grid_000 (pre-election convenience snapshot) is written only when store_grid=true
  • additional sparse snapshots follow store_grid + grid_interval

Legacy replay-side .npy overlays (area_borders.npy, agents_per_cell.npy, area_strings_per_cell.npy, agent_strings_per_cell.npy) are not required artifacts for current schema-v3 headless runs.

Voting rule identification (reproducibility)

For rule-comparison runs, rule_idx is the canonical voting-rule selector. To make rule_idx unambiguous across code changes, schema v3 stores:

  • meta.yaml: run.rule_idx, run.rule_name, run.rule_impl_name
  • static.json: voting_rules.names, voting_rules.impl_names, plus the selected index/name

The distance function (distance_idx) is also recorded for auditability:

  • meta.yaml: run.distance_idx, run.distance_name, run.distance_impl_name
  • static.json: distance_functions.names, distance_functions.impl_names, plus the selected index/name

Timing semantics (important)

Recorded step t (where t starts at 1) corresponds to the election-time state:

  • The grid shown/used is the state after applying mutation from step t-1 (for t>1).
  • Elections and rewards/learning happen on this state during step t.
  • No mutation occurs during step t; mutation from step t is applied at the start of step t+1.
  • area_color_* in area_steps.parquet reflects the election-time distribution for step t.
  • color_* in steps.parquet reflects the global election-time distribution for step t, computed from exact grid counts (not from averaging area distributions).
  • Grid snapshots (grids/grid_*.npy) are the election-time state for step t and match the distributions.
  • First/last election-time snapshots are always present at t=1 and t=S.
  • When snapshots are stored sparsely (grid_interval > 1), replay uses carry-forward semantics: for step t it shows the latest available grid_k with k <= t.

post_election_pre_mutation means: post-election/reward for step t, pre-mutation of step t (applied at step t+1).

Replay step 0

Replay starts in a grid-only step 0 state:

  • It loads grids/grid_000.npy (pad-aware filename; if present) and shows it as the initial grid.
  • It does not populate model/area time series until the first replay step() call.

Shared identifiers

All Parquet tables include:

  • run_seed (int32): concrete RNG seed used for the run
  • rule_idx (int16): voting rule index used for the run

Tables

steps.parquet

Primary key: (run_seed, rule_idx, step)

column dtype notes
run_seed int32 run identifier (seed)
rule_idx int16 voting rule index
step int32 1..S
collective_assets float32 model sum of assets
gini_index int16 0–100
turnout float32 global population-based turnout (%)
mean_altruism float32 mean altruism_factor
mean_dissatisfaction float32 mean dissatisfaction_value
color_0..color_{C-1} float32 optional, pre-mutation

area_steps.parquet

Merged area-state + election table.

Primary key: (run_seed, rule_idx, step, area_id)

column dtype notes
run_seed int32
rule_idx int16
step int32
area_id int32
eligible_voters int32 agents eligible for election in this area/step
participants int32 number who voted
turnout float32 participants/area_num_agents * 100
fee_pool float32 matches simulation internal type
winning_option_id int32 option row index into model.options
elected_color_0..elected_color_{C-1} int16 Area.voted_ordering
dist_to_reality float32 distance(real_order, voted_order)
puzzle_distance float32 distance(puzzle_order, voted_order); NaN when puzzle mode is off
gini_index int16 area gini 0–100
area_color_0..area_color_{C-1} float32 pre-mutation distribution

Quality-gate interpretation:

  • Canonical gate metric in analysis is quality_distance (derived, mode-aware).
  • Source is puzzle_distance when quality_target_mode=puzzle, else dist_to_reality.
  • Both raw columns are retained for diagnostics and replay context.

agents.parquet

Agent snapshot table (agent state only).

Primary key: (run_seed, rule_idx, step, agent_id)

column dtype notes
run_seed int32
rule_idx int16
step int32
agent_id int32
assets float32 matches simulation internal type
num_elections_participated int32 cumulative counter across all areas/steps
personality_group_idx int16 stored preference-group index
eligible_for_election boolean eligibility flag in this area/step election
participating boolean participation decision in this area/step
election_fee float32 charged fee in this area/step
reward_personal float32 reward/penalty amount
election_delta_abs float32 realized absolute asset delta
election_delta_rel float32 realized relative asset delta
participation_baseline float32 EMA baseline for participation learning
participation_signal float32 participation learning signal (mode-dependent)
participation_signal_group_component float32 centered/group component participation signal
participation_signal_fee_component float32 explicit fee component of participation signal
q_participation float32 learned participation propensity (q)
participation_probability float32 current participation probability from q
altruism_factor float32 agent altruism_factor
dissatisfaction_value float32 dissatisfaction (distance)
dissatisfaction_baseline float32 EMA baseline for dissatisfaction
dissatisfaction_signal float32 baseline-corrected dissatisfaction signal

Semantics: the row for step t represents the agent’s final state after it participated in all elections it was eligible for during step t.

votes.parquet

Vote signal table (participants only). This is the single source of election-contextual agent values (belief/confidence).

Primary key: (run_seed, rule_idx, step, area_id, agent_id)

column dtype notes
run_seed int32
rule_idx int16
step int32
area_id int32 disambiguates overlapping areas
agent_id int32
participating boolean always true (rows only for participants)
confidence float32 agent confidence at vote time in this area
voted_altruistically boolean True/False/null at vote time
estim_dst_color_0..estim_dst_color_{C-1} float32 estimated area color distribution at vote time
rank_1_option_id Int32 option row index into model.options
rank_1_oppose_score float32 lower = better
rank_2_option_id Int32
rank_2_oppose_score float32
rank_3_option_id Int32
rank_3_oppose_score float32

Notes:

  • votes.parquet uses a fixed 3-rank wide layout to reduce row counts.
  • If fewer than 3 options exist, remaining rank_* fields should be null.
  • If estimate distributions are missing/invalid at vote time, estim_dst_color_* is written as NaN (fail-visible), not zeros.
  • If no agent participates in a run/step, votes.parquet may be empty (0 rows) but remains schema-valid with typed columns.
  • For tied oppose scores, top-k rank extraction uses seeded RNG tie-breaking (unbiased); id-based tie fallback is not used.