pipecat/AGENTS.md

AGENTS.md

Project Overview

Pipecat is an open-source Python framework for building real-time voice and multimodal conversational AI agents. The codebase is organized around a pipeline architecture where data flows through connected services (STT → LLM → TTS).

Development Environment Setup

Prerequisites

• Minimum Python Version: 3.10
• Recommended Python Version: 3.12
• Package Manager: uv (recommended) or pip

Setup Commands

# Clone the repository
git clone https://github.com/pipecat-ai/pipecat.git
cd pipecat

# Install dependencies with uv (recommended)
uv sync --group dev --all-extras \
  --no-extra gstreamer \
  --no-extra krisp \
  --no-extra local \
  --no-extra ultravox

# Or with pip
pip install -e ".[dev]"

# Install pre-commit hooks
uv run pre-commit install

# Set up environment variables
cp env.example .env

Build and Test Commands

Running Tests

# Run all tests
uv run pytest

# Run specific test file
uv run pytest tests/test_name.py

# Run tests with coverage
uv run pytest --cov=pipecat --cov-report=html

Code Quality

# Format code (required before commits)
uv run ruff format

# Lint code
uv run ruff check

# Type checking
uv run mypy src/pipecat

# Run pre-commit checks manually
uv run pre-commit run --all-files

Documentation

# Build API documentation
cd docs/api
./build-docs.sh

# Build docs manually
sphinx-build -b html . _build/html -W --keep-going

Code Style Guidelines

Python Standards

• Formatting: Strict PEP 8 via Ruff
• Docstrings: Google-style format
• Type Hints: Required for all public APIs
• Import Organization: Automated via Ruff

Docstring Conventions

• Classes: Describe purpose + __init__ with complete Args: section
• Dataclasses: Use Parameters: section, no __init__ docstring
• Methods: Include Args: and Returns: sections
• Properties: Must have Returns: section
• Examples: Use Examples: section with :: syntax

File Organization

src/pipecat/           # Main package
├── processors/        # Frame processors
├── services/          # AI service integrations
├── transports/        # Communication layers
├── frames/            # Data frame definitions
└── pipeline/          # Pipeline orchestration

examples/foundational/ # Step-by-step tutorials
tests/                 # Test suite

Testing Instructions

Test Structure

• Unit Tests: Test individual components in isolation
• Integration Tests: Test service interactions
• Example Tests: Validate foundational examples work

Adding Tests

# Test naming convention
test_<component>_<functionality>.py

# Run specific test pattern
uv run pytest -k "test_pipeline"

# Run with debugging
uv run pytest -s -vv tests/test_name.py::test_function

Pre-commit Requirements

All commits must pass:

• Ruff formatting
• Ruff linting
• Type checking
• Basic test suite

Dependency Management

Using uv (Recommended)

# Add runtime dependency
uv add package-name

# Add optional dependency
uv add --optional service package-name

# Add development dependency
uv add --group dev package-name

# Update lockfile
uv lock

# Sync dependencies
uv sync

Important Notes

• Always commit both pyproject.toml and uv.lock together
• Never manually edit uv.lock - it's auto-generated
• Use extras for optional service dependencies (e.g., [openai], [cartesia])

Project Structure Guidelines

Service Integration

When adding new AI services:

1. Create service class in src/pipecat/services/<provider>/
2. Follow existing patterns (e.g., STTService, LLMService)
3. Add to appropriate extras in pyproject.toml
4. Include tests in tests/
5. Add documentation examples

Frame Processing

For custom processors:

1. Inherit from FrameProcessor
2. Implement process_frame() method. ALWAYS explicitly call await super().process_frame(frame, direction) at the top of this method.
3. Handle frame direction (FrameDirection.UPSTREAM/DOWNSTREAM)
4. Add proper type hints and docstrings

Transport Implementation

For new transport layers:

1. Inherit from BaseTransport
2. Implement required abstract methods
3. Handle connection lifecycle
4. Support both input and output streams

Security Considerations

API Keys

• Never commit API keys to the repository
• Use environment variables for all secrets
• Reference env.example for required variables
• Use .env files for local development

Input Validation

• Validate all external inputs (audio, text, API responses)
• Sanitize user data before processing
• Handle rate limiting for external services
• Implement proper timeout handling

Performance Guidelines

Memory Management

• Clean up resources in transport disconnection handlers
• Use async context managers for service connections
• Implement proper frame lifecycle management

Latency Optimization

• Choose appropriate STT services for latency requirements
• Use streaming TTS when possible
• Implement connection pooling for HTTP services
• Consider WebRTC for real-time applications

Common Patterns

Error Handling

@transport.event_handler("on_error")
async def on_error(transport, error):
    logger.error(f"Transport error: {error}")

    # Shutdown the pipeline
    await task.queue_frame(EndFrame())
 

Service Configuration

# Use environment variables for configuration
service = OpenAILLMService(
    api_key=os.getenv("OPENAI_API_KEY", ""),
    model="gpt-4o",
    params={"temperature": 0.7}
)

Pipeline Assembly

pipeline = Pipeline([
    transport.input(),
    stt_service,
    context_aggregator.user(),
    llm_service,
    tts_service,
    transport.output(),
    context_aggregator.assistant(),
])

Commit and PR Guidelines

Commit Message Format

<type>(<scope>): <description>

[optional body]

[optional footer]

Types: feat, fix, docs, style, refactor, test, chore

PR Requirements

• All tests must pass
• Code must be properly formatted (Ruff)
• Include appropriate tests for new functionality
• Update documentation if needed
• Reference related issues in description

Review Process

1. Automated checks must pass
2. Manual code review by maintainers
3. Documentation review for user-facing changes
4. Integration testing for service additions

Troubleshooting

Common Issues

• Import errors: Run uv sync to ensure dependencies are installed
• Test failures: Check environment variables in .env
• Format errors: Run uv run ruff format before committing
• Type errors: Ensure all public methods have type hints

Development Tips

• Use foundational examples as starting points for testing
• Check existing services for integration patterns
• Run tests frequently during development
• Use IDE integration for Ruff formatting

Getting Help

• Documentation: docs.pipecat.ai
• Issues: GitHub Issues