Event System Integration

Overview

The module event system is designed to handle common integration pain points in distributed module architectures. This document covers all integration scenarios, reliability guarantees, and best practices.

Integration Pain Points Addressed

1. Event Delivery Reliability

Problem: Events can be lost if modules are slow or channels are full.

Solution:

Channel Buffering: 100-event buffer per module (configurable)
Non-Blocking Delivery: Uses try_send to avoid blocking the publisher
Channel Full Handling: Events are dropped with warning (module is slow, not dead)
Channel Closed Detection: Automatically removes dead modules from subscriptions
Delivery Statistics: Track success/failure rates per module

Code:

#![allow(unused)]
fn main() {
// EventManager tracks delivery statistics
let stats = event_manager.get_delivery_stats("module_id").await;
// Returns: Option<(successful_deliveries, failed_deliveries, channel_full_count)>
}

2. Event Ordering and Timing

Problem: Events might arrive out of order or modules might miss events during startup.

Solution:

ModuleLoaded Timing: Only published AFTER module subscribes (startup complete)
Hotloaded Modules: Automatically receive all already-loaded modules when subscribing
Consistent Ordering: Subscription → ModuleLoaded events (guaranteed order)

Flow:

Module loads → Recorded in loaded_modules
Module subscribes → Receives all already-loaded modules
ModuleLoaded published → After subscription (startup complete)

3. Event Channel Backpressure

Problem: Fast publishers can overwhelm slow consumers.

Solution:

Bounded Channels: 100-event buffer prevents unbounded memory growth
Non-Blocking: Publisher never blocks, events dropped if channel full
Statistics Tracking: Monitor channel full events to identify slow modules
Automatic Cleanup: Dead modules automatically removed

Monitoring:

#![allow(unused)]
fn main() {
let stats = event_manager.get_delivery_stats("module_id").await;
if let Some((_, _, channel_full_count)) = stats {
    if channel_full_count > 100 {
        warn!("Module {} is slow, dropping events", module_id);
    }
}
}

4. Missing Events During Startup

Problem: Modules that start later miss events from earlier modules.

Solution:

Hotloaded Module Support: Newly subscribing modules receive all already-loaded modules
Event Replay: ModuleLoaded events sent to newly subscribing modules
Consistent State: All modules have consistent view of loaded modules

5. Event Type Coverage

Problem: Not all events have corresponding payloads or are published.

Solution:

Complete Coverage: All EventType variants have corresponding EventPayload variants
Governance Events: All governance events are published
Network Events: All network events are published
Lifecycle Events: All lifecycle events are published

Event Categories

Core Blockchain Events

NewBlock: Block connected to chain
NewTransaction: Transaction in mempool
BlockDisconnected: Block disconnected (reorg)
ChainReorg: Chain reorganization

Governance Events

GovernanceProposalCreated: Proposal created
GovernanceProposalVoted: Vote cast
GovernanceProposalMerged: Proposal merged
GovernanceForkDetected: Fork detected

Network Events

PeerConnected: Peer connected
PeerDisconnected: Peer disconnected
PeerBanned: Peer banned
MessageReceived: Network message received
BroadcastStarted: Broadcast started
BroadcastCompleted: Broadcast completed

Module Lifecycle Events

ModuleLoaded: Module loaded (after subscription)
ModuleUnloaded: Module unloaded
ModuleCrashed: Module crashed
ModuleHealthChanged: Health status changed

Maintenance Events

DataMaintenance: Unified cleanup/flush (replaces StorageFlush + DataCleanup)
MaintenanceStarted: Maintenance started
MaintenanceCompleted: Maintenance completed
HealthCheck: Health check performed

Resource Management Events

DiskSpaceLow: Disk space low
ResourceLimitWarning: Resource limit warning

Event Delivery Guarantees

At-Most-Once Delivery

Events are delivered at most once per subscriber
If channel is full, event is dropped (not retried)
If channel is closed, module is removed from subscriptions

Best-Effort Delivery

Events are delivered on a best-effort basis
No guaranteed delivery (modules can be slow/dead)
Statistics track delivery success/failure rates

Ordering Guarantees

Events are delivered in order per module (single channel)
No cross-module ordering guarantees
ModuleLoaded events are ordered: subscription → ModuleLoaded

Error Handling

Channel Full

Event is dropped with warning
Module subscription is NOT removed (module is slow, not dead)
Statistics track channel full count

Channel Closed

Module subscription is removed
Statistics track failed delivery count
Module is automatically cleaned up

Serialization Errors

Event is dropped with warning
Module subscription is NOT removed
Error is logged for debugging

Monitoring and Debugging

Delivery Statistics

#![allow(unused)]
fn main() {
// Get statistics for a module
let stats = event_manager.get_delivery_stats("module_id").await;
// Returns: Option<(successful, failed, channel_full)>

// Get statistics for all modules
let all_stats = event_manager.get_all_delivery_stats().await;
// Returns: HashMap<module_id, (successful, failed, channel_full)>

// Reset statistics (for testing)
event_manager.reset_delivery_stats("module_id").await;
}

Event Subscribers

#![allow(unused)]
fn main() {
// Get list of subscribers for an event type
let subscribers = event_manager.get_subscribers(EventType::NewBlock).await;
// Returns: Vec<module_id>
}

Best Practices

For Module Developers

Subscribe Early: Subscribe to events as soon as possible after handshake
Handle Events Quickly: Keep event handlers fast and non-blocking
Monitor Statistics: Check delivery statistics to ensure events are received
Handle ModuleLoaded: Always handle ModuleLoaded to know about other modules
Graceful Shutdown: Handle NodeShutdown and DataMaintenance (urgency: “high”)