Skip to content

Protocol Buffers in AICO

This document provides guidance for developers working with Protocol Buffers (protobuf) in the AICO system.

Overview

AICO uses Protocol Buffers as the unified message format for all communication between components. Protocol Buffers provide:

  • High Performance: Binary serialization is faster and more compact than JSON
  • Strong Typing: Compile-time type checking reduces errors
  • Cross-Platform: Consistent message format across all platforms and languages
  • Schema Evolution: Built-in versioning and backward compatibility
  • Code Generation: Automatic generation of serialization/deserialization code

Directory Structure

All Protocol Buffer definitions are located in the /proto/ directory with the following structure:

/proto/
  ├── aico_core_envelope.proto      # Core message envelope
  ├── aico_core_common.proto        # Common types and utilities
  ├── aico_core_api_gateway.proto   # API Gateway messages
  ├── aico_core_logging.proto       # Logging system messages
  ├── aico_core_plugin_system.proto # Plugin system messages
  ├── aico_core_update_system.proto # Update system messages
  ├── aico_conversation.proto       # Conversation messages
  ├── aico_emotion.proto            # Emotion simulation messages
  ├── aico_integration.proto        # Cross-module integration
  ├── aico_modelservice.proto       # Model service messages
  ├── aico_personality.proto        # Personality simulation
  └── aico_ui.proto                 # UI-related messages

Code Generation

Prerequisites

  1. Install the Protocol Buffers compiler (protoc): 

    # macOS
brew install protobuf

# Ubuntu/Debian
sudo apt-get install protobuf-compiler

# Windows (via chocolatey)
choco install protoc

  2. Install language-specific plugins: 

    # Python
pip install protobuf mypy-protobuf

# Dart
dart pub global activate protoc_plugin

# JavaScript/TypeScript
npm install -g protoc-gen-js protoc-gen-grpc-web

Generating Code

Note: All commands assume you're starting from the AICO project root directory. For Python, you must ensure the -I (include) path points to both your proto directory and your Python venv's site-packages, where the Google well-known .proto files are located. See below for the exact command and troubleshooting.

Python (Backend & Shared)

From the project root, run:

# macOS/Linux
protoc -I=proto -I=.venv/lib/python3.13/site-packages --python_out=shared/aico/proto proto/*.proto

# Windows
protoc -I=proto -I=.venv/Lib/site-packages --python_out=shared/aico/proto proto/*.proto
  • If you use a different venv location or OS, adjust the -I path accordingly.
  • If you get errors about missing google/protobuf/*.proto files, ensure your venv's site-packages/google/protobuf/ contains the .proto files. See the troubleshooting section below.

Note: you can also use the CLI to compile the proto files: 

aico dev protoc

Dart (Flutter Frontend)

cd proto
protoc -I=. --dart_out=../frontend/lib/generated ./*.proto

JavaScript/TypeScript (Studio Admin Interface)

cd proto
protoc -I=. --js_out=import_style=commonjs,binary:../studio/src/generated --grpc-web_out=import_style=commonjs,mode=grpcwebtext:../studio/src/generated ./*.proto

Development Guidelines

Message Format Evolution

When updating message formats:

  1. Follow Protocol Buffers best practices for backward compatibility:
  2. Never remove or renumber fields, only add new ones
  3. Use the reserved keyword for deprecated fields

  4. Keep field numbers consistent across versions

  5. Update version numbers in the message metadata when making changes

  6. Document changes in commit messages and update relevant architecture documentation

Common Patterns

Message Envelope

All messages should use the common envelope structure defined in core/envelope.proto:

message Envelope {
  Metadata metadata = 1;
  google.protobuf.Any payload = 2;
}

message Metadata {
  string message_id = 1;
  google.protobuf.Timestamp timestamp = 2;
  string source = 3;
  string message_type = 4;
  string version = 5;
}

Timestamps

Always use google.protobuf.Timestamp for timestamp fields:

import "google/protobuf/timestamp.proto";

message YourMessage {
  google.protobuf.Timestamp created_at = 1;
}

Enumerations

Define enumerations with an UNKNOWN = 0 default value:

enum Priority {
  UNKNOWN = 0;
  LOW = 1;
  MEDIUM = 2;
  HIGH = 3;
}

Extensibility

Use oneof for fields that can have multiple types:

message Result {
  oneof value {
    string text_value = 1;
    int32 numeric_value = 2;
    bool boolean_value = 3;
  }
}

Testing

  1. Unit Testing: Test serialization/deserialization of messages
  2. Integration Testing: Test message passing between different components
  3. Cross-Language Testing: Verify compatibility between Python, Dart, and JavaScript implementations

Integration with Message Bus

The ZeroMQ message bus uses these Protocol Buffer definitions for serialization and deserialization of all messages. See the Message Bus Architecture document for more details.

Integration with API Gateway

The API Gateway performs minimal transformation between external formats (JSON, gRPC) and internal Protocol Buffer messages. See the API Gateway Architecture document for more details.

Local-First and Federated Architecture Considerations

Protocol Buffers support AICO's local-first, federated architecture by:

  1. Efficiency: Compact binary format reduces network bandwidth for device synchronization
  2. Consistency: Same message format used across all devices and platforms
  3. Versioning: Built-in schema evolution supports gradual updates across federated devices
  4. Security: Binary format with clear structure reduces attack surface compared to text-based formats

References