# Application Scenarios (/docs/application_scenarios)

The TEN framework is designed to make AI feature integration and application development more efficient and flexible. Developers can create and deploy AI function extensions quickly, combining them to form customized applications or services. Typical AI scenarios include:

* **Intelligent Speech Processing**: Extensions for speech recognition (STT), speech synthesis (TTS), and sentiment analysis can be combined to create conversational systems.

* **Visual Recognition Systems**: Extensions for image recognition and video analysis can be used to build security or retail solutions with face recognition, object detection, and motion tracking.

* **Recommendation Systems and Data Analysis**: Extensions for machine learning and data processing can be combined to create personalized recommendation systems and analytical tools.

* **Interactive Education and Training Platforms**: Extensions for interactive teaching, intelligent tutoring, and assessments can be assembled into comprehensive educational technology solutions.

* **Health Monitoring and Diagnostic Systems**: Medical image analysis and AI-driven diagnostic extensions can be used to develop systems for disease monitoring and early diagnosis.

Through the TEN framework, developers can create, share, and extend AI extensions, enabling rapid development and innovation. The flexible architecture ensures scalability and maintainability, providing a solid foundation for modern AI solutions.


# Goal (/docs/goal)

The vision of the TEN framework is to establish a modular, flexible, and scalable platform that enhances development speed, maintainability, and scalability. The framework is designed to support best engineering practices for modular development, enabling developers to create robust, efficient, and adaptable applications by leveraging comprehensive mechanisms provided by the platform.


# Welcome to TEN (/docs)

import { Card, Cards } from 'fumadocs-ui/components/card'
import { Callout } from 'fumadocs-ui/components/callout'

Welcome to TEN! TEN stands for **Transformative Extensions Network**, an open-source framework for real-time multimodal conversational AI.

## Vision

The TEN framework is designed to establish a modular, flexible, and scalable platform that enhances development speed, maintainability, and scalability. The framework supports best engineering practices for modular development, enabling you to create robust, efficient, and adaptable applications by leveraging comprehensive mechanisms provided by the platform.

## Get Started

<Cards>
  <Card title="Quick Start with TEN Agent Examples" description="Get TEN Agent up and running in under 10 minutes" href="/docs/ten_agent_examples/getting_started" />

  <Card title="TEN Framework Docs" description="Deep dive into the framework's capabilities" href="/docs/ten_framework/getting-started/quick-start" />

  <Card title="TEN VAD Docs" description="Low-latency, high-performance and lightweight Voice Activity Detector" href="/docs/ten_vad" />

  <Card title="TEN Turn Detection Docs" description="Turn detection for full-duplex dialogue communication" href="/docs/ten_turn_detection" />

  <Card title="Examples & Tutorials" description="Learn by building real-world agents" href="https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples" />
</Cards>

## Why TEN?

<Cards>
  <Card title="High-Performance Real-Time">
    Native support for multimodal interactions with low latency. Optimized for complex audio-visual AI applications.
  </Card>

  <Card title="Multi-Language Support">
    Build extensions in C++, Go, Python, or JavaScript/TypeScript. Run seamlessly on Windows, Mac, Linux, and mobile.
  </Card>

  <Card title="Edge-Cloud Integration">
    Combine edge and cloud deployments flexibly. Balance cost, latency, and performance for your use case.
  </Card>

  <Card title="Beyond Model Limitations">
    Create complex AI apps with drag-and-drop simplicity. Integrate LLMs with databases, RAG, audio-visual tools, and more.
  </Card>

  <Card title="Real-Time State Management">
    Dynamic agent state management enables responsive behavior adjustment in real time.
  </Card>

  <Card title="Modular & Extensible">
    Build reusable extensions that work together. No glue code needed.
  </Card>
</Cards>

## Choose Your Path

<Callout type="info" title="New to TEN?">
  Start with the [Getting Started guide](/docs/ten_agent_examples/getting_started) to run your first voice agent in minutes.
</Callout>

### For Application Developers

Build voice agents and conversational AI applications:

* [TEN Agent Examples Overview](/docs/ten_agent_examples/overview) - Understand the agent architecture
* [Getting Started](/docs/ten_agent_examples/getting_started) - Set up and run your first agent
* [Customize Agent](/docs/ten_agent_examples/customize_agent/modify-main) - Configure and personalize your agent

### For Extension Developers

Create custom extensions and integrations:

* [Extension Development](/docs/ten_agent_examples/extension_dev) - Build your own extensions
* [API Reference](/docs/ten_agent_examples/api-reference) - Events and configuration schemas
* [TEN Framework API](/docs/ten_framework/api) - Complete framework API documentation
* [TEN Cloud Store](/docs/ten_framework/cloud_store) - Share and discover extensions

### For Framework Contributors

Contribute to the TEN ecosystem:

* [TEN Framework Overview](/docs/ten_framework/concept_overview) - Architecture deep dive
* [Development Guide](/docs/ten_framework/development/development_process) - Development best practices
* [Testing & Debugging](/docs/ten_framework/testing_and_debugging/for_users_of_ten_framework) - Quality assurance

## Community & Support

* [GitHub: TEN-Framework](https://github.com/TEN-framework/ten-framework) - Core framework repository
* [FAQs](/docs/ten_agent_examples/faqs) - Common questions and troubleshooting


# Key Concepts (/docs/key_concepts)

## Modularity

1. **Defined Module Boundaries**: The framework allows for clear definition of control and data plane boundaries, which can be statically declared. This facilitates easy inspection and post-processing using offline tools.

2. **Dynamic Service Composition**: Modules within a process can be dynamically composed at runtime, without the need for static compilation, enabling flexible service configuration.

3. **Streamlined Module Integration**: In scenarios with multiple module compositions, only declarations are required, without modifying any module code or writing glue code.

4. **Decentralized Module Coupling**: The architecture allows for modules to be developed independently by different teams, with no inter-module coupling, facilitating dynamic replacement.

5. **Flexible Module Interface Communication**: Supports two types of interface communication—one requiring consistent parameters on both sides, and another allowing differences in parameters with platform-provided bridging mechanisms. This flexibility is crucial for independent module development and deployment.

## Orchestration

1. **Directed Cyclic Graph (DCG) Topologies**: Supports complex graph structures for module composition, enabling 1-to-n, n-to-1, and n-to-m interaction relationships in both control and data planes.

2. **Cross-Process and Thread Orchestration**: Allows coordination of module instances across different processes and threads to form complete service scenarios.

![cross\_process\_and\_thread\_orchestration](https://ten-framework-assets.s3.amazonaws.com/doc-assets/cross_process_and_thread_orchestration.png)

3. **Dynamic Orchestration Configuration**: Orchestration can be configured dynamically at runtime or pre-configured before service execution.

## Flexible Configuration Settings

1. **Hierarchical Configuration Support**: Different module instances can have varying configuration possibilities, adaptable across different hierarchical contexts.

2. **Versatile Configuration Capabilities**: Configurations can be preset, dynamically changed during runtime, or distributed as updates.

## Diverse Releasing Capabilities

1. **Support for Binary & Source Release**: Modules can be distributed as source or binary, with both modes coexisting for the same module.

2. **Integration With Cloud Marketplace**: Modules, whether source or binary, can be published to the cloud marketplace for easy download and integration into projects.

## Multiple Instances Capability

1. **Multiple Instances of the Platform**: The platform avoids the use of global variables, supporting multiple instances within a single business process.

2. **Multiple Instances of Each Module**: Modules are developed without reliance on global variables, allowing multiple instances of the same module to run without conflict.

3. **Multiple Instances of Orchestrated Pipelines**: Supports concurrent or sequential execution of multiple orchestrated pipelines within a single process.

## Diverse Distribution Capability

1. **Dynamic Switching of Distribution Configurations**: Modules can be distributed across different processes and threads for high availability, high concurrency, and easier hot-updating.

2. **Decoupling of Distribution Configuration**: Distribution configurations are decoupled from both module code and compilation/build processes, simplifying deployment.

## Flexibility in Input/Output Support

1. **Support for Control and Data Planes**: The framework supports both command invocation and raw data transmission (audio/video/data) between modules.

2. **Support for Inter-process and Inter-thread Communication**: Modules can handle external input/output operations across processes and threads simultaneously.

3. **RPC Essence**: External input/output operations have RPC characteristics, enabling seamless module integration across different processes.

4. **Trigger Multiple Downstream Components**: A single action can trigger multiple downstream modules, with aggregated responses returned to the original module.

5. **Declarative External Input/Output**: Interfaces can be statically declared, allowing for static checks by offline tools.

6. **Simplified External Input/Output Code**: Function call interfaces, ideally auto-generated, simplify coding for module developers.

7. **Support for Non-serializable Data**: The architecture supports transmitting non-serializable data, with tools automatically recognizing and tagging such usage.

8. **Transparent Inter-process Communication**: Underlying transmission logic is abstracted away by the platform, simplifying cross-process interactions.

9. **Support for Synchronous and Asynchronous Control Plane Operations**: The control plane supports both synchronous and asynchronous interactions, allowing for bidirectional command and response flows.

10. **Support for Restricted Execution Context**: Enables synchronous interaction between modules within the same thread.

## Freedom in Developing Modules

1. **Freedom to Use Third-party Libraries**: Modules can freely utilize third-party libraries to achieve desired functionalities.

2. **Freedom to Utilize Operating System Features**: Developers can freely create and terminate native threads within modules, offering maximum development flexibility.

## Programming Language Support

1. **Support for Multiple Programming Languages**: The framework supports modules written in various languages (e.g., C/C++, Go, Java, Python), enabling diverse development environments.

   ![Programming Language Support](https://ten-framework-assets.s3.amazonaws.com/doc-assets/various_language_support.png)

2. **Cross-language Module Integration**: Modules written in different languages can run together within the same process.

3. **Unified Interaction Across Languages**: Modules have consistent interfaces and behaviors across different languages.

4. **Unified Data Types Across Languages**: The platform provides unified data types, ensuring consistent data exchange across different programming languages.

## Rich External Interaction Capabilities

1. **UI Provider**: The architecture supports backend services providing UI components for the frontend.

2. **RESTful Interface**: Backend services can expose RESTful interfaces for client interactions.

3. **Non-intrusive Integration with Existing Services**: The framework can be integrated into existing services without requiring a complete rewrite.

4. **Single-threaded Operation**: The framework can function as a standalone process or as a thread within an existing service, avoiding conflicts with existing business logic.

## Testing Friendly

1. **Diverse Testing Mechanisms**: The framework supports various testing strategies, including unit tests, integration tests, module testing, and black-box testing.

2. **Independent Module Testing**: Modules can be tested independently, facilitating thorough and effective testing.

## Offline Processing Capability

1. **Package Manager**: The framework includes CLI and package managers for managing modules and their combinations.

2. **Orchestration Tools**: GUI tools are available for offline development and orchestration, enhancing the developer experience.

3. **Visual Debugging Tools**: The architecture supports visual debugging tools to help visualize module interactions.

## Robustness

1. **High Concurrency of Orchestrated Pipelines**: The platform supports at least 100 concurrent instances of orchestrated pipelines within a single process.

2. **High Concurrency of RESTful Interfaces**: The platform can support over 20,000 RESTful clients in a single process.

3. **High Concurrency of Non-RESTful Interfaces**: The platform can support over 1,000 non-RESTful clients in a single process.


# TEN Turn Detection (/docs/ten_turn_detection)

# TEN Turn Detection

Intelligent model for detecting conversation turns in human-AI dialogue systems, identifying three key utterance states for natural conversation flow.

## Core Functionality

Classifies user text into three states:

* **Finished**: Complete thought expressed; user expects a response
* **Unfinished**: Clearly unfinished utterance where user pauses but intends to continue
* **Wait**: User requests AI to pause or stop speaking

## Technology

* **Model**: Transformer-based (Qwen2.5-7B)
* **Languages**: English and Chinese
* **Inference**: Production-ready deployment scripts included

## Performance Metrics

Strong accuracy across all detection categories:

| Metric               | Accuracy |
| -------------------- | -------- |
| Finished (English)   | 90.64%   |
| Unfinished (English) | 98.44%   |
| Wait State           | 91%      |

Performance exceeds comparable open-source alternatives.

## Use Cases

* **Full-Duplex Conversation** — Enable interruption detection and natural turn-taking
* **Conversation Flow Management** — Determine when to wait for user continuation vs. respond
* **Pause/Stop Handling** — Recognize when users request the AI to pause or stop
* **Multi-Language Support** — Deploy in English and Chinese speaking applications

## Getting Started

```python
# Basic usage
from ten_turn_detection import TurnDetection

detector = TurnDetection()
state = detector.detect("user input text")
# Returns: "finished", "unfinished", or "wait"
```

## Integration with TEN Agent

TEN Turn Detection works seamlessly with TEN VAD for complete conversation management:

1. **TEN VAD** detects speech activity (when user is speaking)
2. **TEN Turn Detection** analyzes utterance to determine conversation state
3. **Agent** responds appropriately based on detected turn state

This enables natural, responsive voice conversations with proper turn-taking.

## Datasets

Bilingual test datasets (TEN-Turn-TestSet) are available for:

* Validation and testing
* Fine-tuning on domain-specific language
* Benchmarking model performance

## Resources

* **GitHub Repository**: [TEN-framework/ten-turn-detection](https://github.com/TEN-framework/ten-turn-detection)
* **Model Weights**: Available via Hugging Face
* **Datasets**: Bilingual test datasets included

## License

Apache 2.0 (see repository for details)


# TEN VAD (/docs/ten_vad)

# TEN VAD - Voice Activity Detector

TEN VAD is a lightweight, high-performance Voice Activity Detector (VAD) designed for real-time speech detection with superior accuracy and minimal computational overhead.

## Overview

TEN VAD delivers exceptional precision compared to industry alternatives like WebRTC VAD and Silero VAD, while consuming significantly fewer computational resources and memory.

**Key Performance Metrics:**

* Real-Time Factor: 0.015 on AMD Ryzen (exceptionally fast)
* Library Size: Only 306KB (minimal footprint)
* Superior precision compared to WebRTC and Silero VAD
* Frame-level speech activity detection for fine-grained control

## Features

### Multi-Platform Support

* **Desktop**: Linux, Windows, macOS
* **Mobile**: Android, iOS
* **Web**: Browser integration via WebAssembly

### Multi-Language Bindings

Build VAD detection in your preferred language:

* Python (with numpy)
* C/C++
* Java (with JNI support for Android)
* Go
* JavaScript/WebAssembly

### Technology Stack

* **ONNX Model Format** for cross-platform deployment and optimization
* **WebAssembly Support** for seamless browser integration
* **JNI Bindings** for native Android support

## Use Cases

* **Real-time Voice Assistants** — Detect speech onset and offset for responsive interaction
* **Voice Conferencing** — Identify active speakers in multi-party conversations
* **Speech Recognition Preprocessing** — Optimize ASR pipeline by filtering silence
* **Mobile Applications** — Lightweight detection on resource-constrained devices
* **Browser-Based Applications** — Client-side speech detection without server round-trips

## Performance Highlights

TEN VAD outperforms comparable alternatives across diverse hardware platforms:

* Minimal computational complexity
* Reduced memory usage
* Fast inference time (Real-Time Factor \< 1)
* Accurate frame-level detection

This makes it ideal for:

* Edge deployment scenarios
* Real-time conversational AI
* Mobile and web applications
* Multi-stream processing

## Getting Started

### Installation

```bash
# Python
pip install ten-vad

# JavaScript/Node.js
npm install ten-vad
```

### Basic Usage

```python
from ten_vad import VAD

# Initialize VAD
vad = VAD()

# Process audio frames
for frame in audio_stream:
    is_speech = vad.detect(frame)
    if is_speech:
        print("Speech detected!")
```

## Integration with TEN Agent

TEN VAD is a core component of TEN Agent's speech processing pipeline:

* Enables natural interruption handling in voice interactions
* Improves responsiveness by detecting speech boundaries
* Reduces unnecessary processing of silence

Use TEN VAD with the TEN framework for:

* **Voice Activity Detection** — Identify when users are speaking
* **Turn Detection** — Combined with TEN Turn Detection for intelligent conversation flow
* **Barge-in Support** — Enable users to interrupt AI responses naturally

## Resources

* **GitHub Repository**: [TEN-framework/ten-vad](https://github.com/TEN-framework/ten-vad)
* **Documentation**: Full API reference and examples
* **Community**: 1.6k stars, 131 forks, active Discord and WeChat communities

## Latest Updates (2025)

Recent releases include:

* Python inference with ONNX models on Linux and macOS
* Full Golang support across major operating systems
* Java support on multiple platforms including Android
* Enhanced model accuracy and performance optimizations

## License

TEN VAD is open-source and community-driven. Check the repository for licensing details.


# Architecture Flow (/docs/ten_agent_examples/architecture_flow)

The whole system contains multiple components that work together to provide a seamless experience. The main components are:

* **TEN Agent App**: The main application that orchestrates the extensions and manages the data flow between them. It runs as a background process. Depending on the graph configuration, the agent starts the required extensions and handles the data flow between them.
* **Front-end UI**: A web-based interface to interact with the agent. It allows users to configure the agent, start/stop the agent, and talk to the agent.
* **Web Server**: A simple Golang web server that serves the HTTP requests. It is responsible for handling the incoming requests and starting/stopping agent processes. It also passes parameter like `graph_name` to determine which graph to use.

The flow of the system is as follows:

<img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/eight.svg" alt="Architecture" />


# FAQs (/docs/ten_agent_examples/faqs)

import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'
import { Callout } from 'fumadocs-ui/components/callout'

Find answers to common questions and solutions to typical issues when working with TEN Agent.

## Installation & Setup

<Accordions>
  <Accordion title="Why do I see '/app/agent/bin/start: not found'?">
    This error typically occurs on Windows due to line-ending issues. Git may convert LF (Unix) line endings to CRLF (Windows), which breaks shell scripts.

    **Solution:**

    1. Configure Git to keep LF line endings:

    ```bash title="Terminal"
    git config --global core.autocrlf false
    ```

    2. Delete your local copy and re-clone the repository:

    ```bash title="Terminal"
    git clone https://github.com/TEN-framework/TEN-Agent.git
    ```

    **Alternative:** Download the ZIP file directly from GitHub instead of cloning.
  </Accordion>

  <Accordion title="How do I verify my internet connection?">
    Make sure both HTTPS and general connectivity are working properly.

    **Test HTTPS connection:**

    ```bash title="Terminal"
    ping www.google.com

    # Expected output:
    PING google.com (198.18.1.94): 56 data bytes
    64 bytes from 198.18.1.94: icmp_seq=0 ttl=64 time=0.099 ms
    64 bytes from 198.18.1.94: icmp_seq=1 ttl=64 time=0.121 ms
    ```

    **Test HTTP connection:**

    ```bash title="Terminal"
    curl www.google.com

    # Expected output (redirect response is normal):
    <html>
    <head><title>301 Moved Permanently</title></head>
    <body>
    <h1>301 Moved Permanently</h1>
    </body>
    </html>
    ```

    <Callout type="info">
      If you're behind a corporate firewall or proxy, you may need to configure Docker and Git to use your proxy settings.
    </Callout>
  </Accordion>
</Accordions>

## Configuration

<Accordions>
  <Accordion title="How do I refresh environment variables?">
    When you update your `.env` file, the running server won't automatically pick up the changes.

    **To apply new environment variables:**

    1. Stop the running server (Ctrl+C)
    2. Save your changes to the `.env` file
    3. Restart the server with `task run`

    <Callout type="info">
      Inside the Docker container, you don't need to run `source .env` - the server will read the file on startup.
    </Callout>
  </Accordion>

  <Accordion title="Where do I get the required API keys?">
    You'll need API keys from these services:

    | Service    | Purpose                 | Get Key                                                                                                                       |
    | ---------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
    | Agora      | Real-time communication | [Agora Console](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project) |
    | OpenAI     | LLM processing          | [OpenAI Platform](https://platform.openai.com/api-keys)                                                                       |
    | Deepgram   | Speech-to-text          | [Deepgram Console](https://console.deepgram.com/)                                                                             |
    | ElevenLabs | Text-to-speech          | [ElevenLabs](https://elevenlabs.io/)                                                                                          |

    Most services offer free tiers or trial credits for getting started.
  </Accordion>
</Accordions>

## Docker Issues

<Accordions>
  <Accordion title="Docker build is very slow on Apple Silicon Mac">
    If you're using an M1/M2/M3 Mac and experiencing slow builds, check your Docker settings.

    **Solution:**

    1. Open Docker Desktop settings
    2. Go to **General** or **Features in development**
    3. **Uncheck** "Use Rosetta for x86\_64/amd64 emulation"

    <Callout type="warn">
      This may result in slightly slower build times on ARM, but runtime performance will be better. When you deploy to x64 servers, performance will be normal.
    </Callout>
  </Accordion>

  <Accordion title="Container fails to start or exits immediately">
    If the Docker container exits right after starting, check these common causes:

    1. **Port conflicts**: Make sure ports 3000, 8080, and 49483 are not in use
    2. **Resource limits**: Ensure Docker has at least 4GB RAM allocated
    3. **Missing .env file**: Verify `.env` exists and has valid API keys

    **Debug steps:**

    ```bash title="Terminal"
    # Check container logs
    docker logs ten_agent_dev

    # Check if ports are in use
    lsof -i :3000
    lsof -i :8080
    ```
  </Accordion>
</Accordions>

## Runtime Issues

<Accordions>
  <Accordion title="Agent doesn't respond to voice input">
    If the agent starts but doesn't respond to your voice:

    1. **Check microphone permissions** in your browser
    2. **Verify API keys** are correctly set in `.env`
    3. **Check browser console** for WebRTC errors
    4. **Ensure Agora credentials** are valid and the project has RTC enabled

    <Callout type="info">
      Open your browser's developer tools (F12) and check the Console tab for error messages.
    </Callout>
  </Accordion>

  <Accordion title="How do I check if my agent is running correctly?">
    After running `task run`, verify these endpoints:

    * **Frontend**: [http://localhost:3000](http://localhost:3000)
    * **TMAN Designer**: [http://localhost:49483](http://localhost:49483)
    * **API Server**: [http://localhost:8080](http://localhost:8080)

    If any of these don't load, check the container logs for errors.
  </Accordion>
</Accordions>

## Still Need Help?

If you can't find an answer to your question:

* Check the [GitHub Issues](https://github.com/TEN-framework/TEN-Agent/issues) for similar problems
* Join the community discussions on GitHub
* Review the [Getting Started guide](/docs/ten_agent_examples/getting_started) for proper setup steps


# Getting Started (/docs/ten_agent_examples/getting_started)

import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
import { Callout } from 'fumadocs-ui/components/callout'
import { Step, Steps } from 'fumadocs-ui/components/steps'

Get TEN Agent up and running in just three steps.

<Steps>
  <Step>
    ## Prerequisites

    Before you begin, make sure you have the following ready:

    <Tabs items={['API Keys', 'Installations', 'System Requirements']}>
      <Tab>
        You'll need accounts and API keys from these services:

        * **Agora** - [App ID](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project) and [App Certificate](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project) (free minutes every month)
        * **OpenAI** - [API key](https://openai.com/index/openai-api/) (or any compatible LLM)
        * **Deepgram** - [ASR key](https://deepgram.com/) (free credits available with signup)
        * **ElevenLabs** - [TTS key](https://elevenlabs.io/) (free credits available with signup)
      </Tab>

      <Tab>
        Install these tools on your machine:

        * [Docker](https://www.docker.com/) / [Docker Compose](https://docs.docker.com/compose/)
        * [Node.js (LTS) v18+](https://nodejs.org/en)
      </Tab>

      <Tab>
        Minimum hardware requirements:

        * **CPU**: 2 cores or more
        * **RAM**: 4 GB or more
      </Tab>
    </Tabs>

    <Callout title="macOS: Docker setting on Apple Silicon" type="warn">
      Uncheck "Use Rosetta for x86/amd64 emulation" in Docker settings. This may result in slower build times on ARM, but performance will be normal when deployed to x64 servers.
    </Callout>
  </Step>

  <Step>
    ## Build the Agent

    ### Clone and configure

    Clone the repository and set up your environment variables:

    ```bash title="Terminal"
    git clone https://github.com/TEN-framework/ten-framework.git

    cd ai_agents

    cp ./.env.example ./.env
    ```

    ### Add your API keys

    Open `.env` and fill in your API keys:

    ```bash title=".env"
    # Agora credentials
    AGORA_APP_ID=your_app_id_here
    AGORA_APP_CERTIFICATE=your_certificate_here

    # ElevenLabs TTS
    ELEVENLABS_TTS_KEY=your_key_here

    # OpenAI
    OPENAI_API_KEY=your_key_here

    # Deepgram ASR
    DEEPGRAM_API_KEY=your_key_here
    ```

    ### Start development containers

    ```bash title="Terminal"
    docker compose up -d
    ```

    ### Build and run the agent

    Enter the container and build the default voice assistant:

    ```bash title="Terminal"
    docker exec -it ten_agent_dev bash
    ```

    Inside the container:

    ```bash title="Inside Container"
    # Navigate to the voice assistant example
    cd agents/examples/voice-assistant

    # Install dependencies and build (~5-8 minutes)
    task install

    # Start the agent
    task run
    ```

    <Callout type="info">
      The frontend will be available at [http://localhost:3000](http://localhost:3000)
    </Callout>

    <Callout title="Want to try other examples?" type="info">
      Check the `/examples` folder for more agent configurations:

      ```bash
      cd agents/examples/<example_name>
      task install
      ```
    </Callout>
  </Step>

  <Step>
    ## Customize with TMAN Designer

    Use the visual designer to customize your agent:

    1. Open [localhost:49483](http://localhost:49483) in your browser
    2. Right-click to load a graph (e.g., Voice Assistant)
    3. Configure API keys and preferences for each extension
    4. Right-click and select **Manage Apps** to open the Apps Manager
    5. Click **Run App** to start your customized TEN Agent

    <Callout type="info">
      If you already configured API keys in the `.env` file, you can skip step 3.
    </Callout>
  </Step>
</Steps>


# TEN Agent Examples Overview (/docs/ten_agent_examples/overview)

TEN Agent Examples is your comprehensive guide to building conversational AI agents with the TEN Framework. These examples demonstrate best practices for creating real-time, multimodal voice agents integrated with modern LLMs like Gemini 2.0 Live and OpenAI Realtime API.

## What You'll Learn

The TEN Agent Examples section provides practical, hands-on documentation to help you:

* **Get started quickly** — Set up and run your first voice agent in under 10 minutes
* **Understand architecture** — Learn how TEN Agent orchestrates ASR, LLM, TTS, and RTC components
* **Customize agents** — Extend and modify agent behavior through the modular "main" extension
* **Build extensions** — Create custom extensions for LLMs, speech services, and external APIs
* **Deploy agents** — Package and deploy your agents in production environments

## Documentation Structure

### Getting Started

Start here if you're new to TEN Agent. You'll set up your development environment and run your first voice agent.

### Architecture & Design

Understand how TEN Agent works under the hood, including real-time communication flow, event handling, and component interaction.

### Customization Guide

Learn how to modify and extend agents to fit your specific needs through the flexible main extension pattern.

### Extension Development

Build your own extensions for LLMs, STT/TTS providers, and custom integrations.

### API Reference

Complete reference documentation for events, schemas, and configuration options.

### Tutorials & Examples

Real-world examples demonstrating specific use cases and patterns.

## Key Concepts

**Real-Time Voice Agents**: TEN Agent supports full-duplex conversation with natural interruption handling, delivering seamless voice interactions.

**Modular Architecture**: Extend functionality through self-contained extensions. The core agent logic resides in the "main" extension, which you can customize for your use case.

**Multi-Language Support**: Build extensions in Python, Node.js, Go, or C++. Mix and match languages in a single agent.

**Production-Ready**: TEN Agent provides the infrastructure and patterns needed for deploying conversational AI at scale.


# RTC First - Design Rationale (/docs/ten_agent_examples/rtc_first_design)

This document explains why our system adopts an **RTC First** strategy, instead of building primarily on WebSockets.
For developers, this means you get a **low-latency, media-first pipeline** that is already built in, with no need to implement device capture, codecs, or transport logic from scratch.

***

## Why RTC First?

* **Low latency:** RTC avoids TCP head-of-line blocking by using UDP + SRTP.
* **Media-native:** Direct microphone, camera, and screen capture.
* **Codec integration:** Opus, VP8/VP9/AV1, H.264 with hardware acceleration.
* **Synchronization:** Audio, video, and data channels aligned in one session.
* **Cross-platform:** Works on browsers, mobile, and native apps.

By contrast, **WebSockets** are best for structured data exchange but lack:

* Native media capture
* Encoding/decoding support
* Bandwidth adaptation
* Built-in synchronization

***

## RTC vs WebSocket

| Feature           | RTC                                         | WebSocket                      |
| ----------------- | ------------------------------------------- | ------------------------------ |
| Transport         | UDP (SRTP, DTLS, ICE, congestion control)   | TCP (single ordered stream)    |
| Latency           | \~50–150ms (optimized for real-time)        | Higher (head-of-line blocking) |
| Media Support     | Native mic/camera/screen capture            | None (raw binary/text only)    |
| Encoding/Decoding | Built-in codecs (Opus, VP8, H.264, AV1)     | External / manual              |
| Synchronization   | Multi-stream A/V sync + data channels       | Manual implementation          |
| Adaptation        | Bandwidth estimation, adaptive bitrate, FEC | Not supported                  |
| Best For          | Calls, streaming, AI agents, real-time apps | Chat, signaling, async events  |

***

Our system includes an RTC module powered by [Agora RTC](https://www.agora.io/en/) as the default provider, so developers don’t need to worry about device capture, encoding/decoding, or transport internals.

From a practical perspective:

* **Setup is minimal:** RTC is ready out of the box.
* **WebSockets remain useful:** Ideal for signaling, configuration, or occasional async updates.
* **RTC is the recommended channel:** For real-time, multimodal interaction where latency, synchronization, and quality matter most.

By adopting **RTC First**, we ensure:

* Real-time performance across devices and platforms.
* A consistent developer experience with built-in media pipelines.
* Flexibility to extend with async features without losing RTC guarantees.

In short, developers can focus on **building features**, not rebuilding a transport stack.


# Build System (/docs/ten_framework/build_system)

## Build System

### C++

The TEN framework uses the `ten_gn` build system to compile C++ apps and C++ extensions. `ten_gn` is a build system based on Google's GN.

### Golang

The TEN framework uses the standard go command to build TEN framework Golang projects.

## Building

### C++ App

To build a TEN framework C++ app, run the following commands in the root directory of the TEN framework C++ app.

`<os>` can be one of the following:

* linux
* mac
* win

`<arch>` can be one of the following:

* x86
* x64
* arm
* arm64

`<build_type>` can be one of the following:

* debug
* release

```shell
ten_gn gen <os> <arch> <build_type>
ten_gn build <os> <arch> <build_type>
```

### C++ Addon

The term `addon` here includes the following:

* extension
* extension group
* protocol

The method to build TEN framework C++ addons is the same as for building TEN framework C++ apps.

### Golang App

To build a TEN Golang project, run the following command in the root directory of the TEN Golang project:

```shell
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```


# Cloud Store (/docs/ten_framework/cloud_store)

The TEN Cloud Store, similar to Google Play on Android and the App Store on iOS, offers a variety of TEN packages. These packages can be downloaded into a TEN app to aid and enhance development.

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

## Information Stored in the Database for TEN Packages

1. **Package Type**

   A string representing the type of package, such as `extension`.

2. **Package Name**

   A string representing the name of the package, such as `builtin_extension`.

3. **Package Version**

   A string following semantic versioning (semver) that specifies the package version, such as `3.0.0`.

4. **Dependencies**

   An array of objects, each containing the following fields:

   * **Package Type**

     A string, matching the top-level `package type`.

   * **Package Name**

     A string, matching the top-level `package name`.

   * **Version**

     A string following semver that specifies the version requirements for the package, such as `~3.0.0`.


# Overview (/docs/ten_framework/concept_overview)

![TEN Framework](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_framework.png)

## TEN Runtime

The TEN runtime is an execution environment designed for running applications within the TEN framework. It provides a flexible architecture that supports extensions developed in multiple programming languages and handles various types of data streams. By managing the lifecycle of extensions, data flows, and threads, the TEN runtime offers developers a powerful tool for building a wide range of applications and services.

## App

An App in the TEN framework can operate as a standalone process or as a thread within an existing process. This flexibility allows for versatile deployment options depending on the needs of the application.

## Extension Group

An Extension Group is a mechanism that designates a specific thread for execution. All extensions within a given group are executed on the same thread, ensuring consistent and synchronized processing.

## Extension

An extension is the fundamental building block of the TEN framework. Developers can create extensions in various programming languages and combine them to build different applications and scenarios. The framework's design supports cross-language collaboration, enabling extensions written in different languages to work together seamlessly within the same application or service.

For example, a developer might use C++ to create an extension for real-time communication (RTC) due to its performance benefits in handling audio and video data, while developing an AI extension in Python to take advantage of its robust libraries for data analysis and machine learning. These extensions can then be integrated into a single application, leveraging the strengths of each language.

## Graph

A Graph in the TEN framework describes the data flow between extensions. It orchestrates how data moves from one extension to another, defining the participants and the flow of data between them. For example, you can route the output of a Speech-to-Text (STT) extension to a Large Language Model (LLM) extension for further processing.

The TEN framework supports four main types of data flows between extensions:

* Command
* Data
* Video Frame
* Audio Frame

By defining these data flows within a graph, developers can create inter-extension communication and unidirectional data streams, particularly useful for handling audio and video data.

## Relationship Among App, Group, and Extension

* **App**:

  An app can execute multiple graphs, which can be either statically predefined or dynamically assembled.

* **Graph**:

  A graph is formed by multiple extensions working together to create a meaningful scenario. Each graph instance operates as a session within the app.

* **Extension Group**:

  The concept of an extension group is analogous to a thread. Extensions written in the same language and within the same extension group run on the same thread during runtime. Developers do not need to manage threads directly; they simply "declare" the group to which each extension belongs.

* **Extension**:

  Each extension within the framework is assigned a unique ID, structured as: `app-uri/graph-name/group-name/extension-name`

![Hierarchical Relationship of Concepts](https://ten-framework-assets.s3.amazonaws.com/doc-assets/hierarchical_relationship_of_concepts.png)

## TEN Cloud Store

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

The TEN Cloud Store functions similarly to Google Play Store or Apple’s App Store, providing a marketplace for extensions. Developers can share their extensions or download those created by others. These extensions can be integrated into TEN apps, facilitating development and expanding functionality.

## TEN Manager

The TEN Manager is a tool that simplifies the management of extensions. It handles tasks such as uploading, sharing, and installing extensions, automatically managing dependencies between them and their environment. This makes the installation and publication of extensions convenient and efficient, streamlining the development process within the TEN framework.


# Dependencies (/docs/ten_framework/dependencies)

The TEN framework utilizes several third-party libraries. Some are used specifically for testing, while others are integrated into the TEN runtime. Below is a description of these libraries, along with any necessary modifications required for their use within the TEN framework.

## Google gn

Instance ID

| OS    | Arch  | Instance ID                                   |
| ----- | ----- | --------------------------------------------- |
| Linux | x64   | BzX0zkfwFeUn9MaDVqm6FugmTIy-hFpgNUx43O1fN00C  |
| Linux | arm64 | rT\_12w1Iv6ug8CJ4j0VQekA0qTDq6CwoAqGWasIKFcEC |
| Win   | x64   | 1QlqF0FPVt82ba5f48HxHpv5xPqOmyaThoR3TicuJ8QC  |

Download directly from [Google GN webpage](https://chrome-infra-packages.appspot.com/p/gn/gn/).

Commit ID

| OS  | Arch      | Commit ID                                |
| --- | --------- | ---------------------------------------- |
| Mac | universal | 18602f6cf1168cf78302024043edc02e8bad2ffb |

Clone and build from the [Gn repository](https://gn.googlesource.com/gn).

## Google ninja

Version: 1.12.1

Download directly from [Ninja release page](https://github.com/ninja-build/ninja/releases).

## yyjson

Version: 0.10.0

[MIT license](https://github.com/ibireme/yyjson/blob/master/LICENSE)

This is used in the TEN framework core to parse and generate JSON data. Please refer to `third_party/yyjson` for details.

## libuv

Version: 1.50.0

[MIT license](https://github.com/libuv/libuv#licensing)

This is one of the event loop libraries used in the TEN runtime. Please refer to `third_party/libuv` for details.

## msgpack-c

Version: 6.1.0

[Boost Software License, Version 1.0](https://github.com/msgpack/msgpack-c#license)

A MessagePack library for C. Please refer to `third_party/msgpack` for details.

## libwebsockets

Version: 4.3.2

[MIT license](https://github.com/warmcat/libwebsockets/blob/main/LICENSE)

Canonical libwebsockets.org networking library. Please refer to `third_party/libwebsockets` for details.

```diff
--- /home/wei/MyData/Temp/libwebsockets-4.3.2/CMakeLists.txt
+++ libwebsockets/CMakeLists.txt
@@ -851,6 +851,7 @@
  add_definitions(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE)
  # Fail the build if any warnings
  add_compile_options(/W3 /WX)
+ add_compile_options(/Zc:preprocessor /wd4819)
  # Unbreak MSVC broken preprocessor __VA_ARGS__ behaviour
  if (MSVC_VERSION GREATER 1925)
   add_compile_options(/Zc:preprocessor /wd5105)
```

Apply the following patch.

```diff
diff --git a/third_party/libwebsockets/cmake/lws_config.h.in b/third_party/libwebsockets/cmake/lws_config.h.in
index f3f4a9d79..e8d36c3ae 100644
--- a/third_party/libwebsockets/cmake/lws_config.h.in
+++ b/third_party/libwebsockets/cmake/lws_config.h.in
@@ -238,3 +238,9 @@
 #cmakedefine LWS_WITH_PLUGINS_API
 #cmakedefine LWS_HAVE_RTA_PREF

+// NOTE (Liu): The libwebsockets library will use external variables from mbedtls
+// if 'LWS_WITH_MBEDTLS' is enabled. On Windows, variable declarations in the header
+// file must start with '__declspec(dllimport)', otherwise, the
+// symbols cannot be accessed.
+// Refer to third_party/mbedtls/include/mbedtls/export.h.
+#define USING_SHARED_MBEDTLS
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
index e8a0cb2d4..84a164e90 100755
--- a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
+++ b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
@@ -183,7 +183,12 @@ int ssl_pm_new(SSL *ssl)
         mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, version);
     } else {
         mbedtls_ssl_conf_max_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
-    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, 1);
+
+    // NOTE: mbedtls added 'ssl_conf_version_check()' since v3.1.0, and
+    // mbedtls only enables TLS 1.2 by default. So the 'min_tls_version' and
+    // 'max_tls_version' must be '0x303', or 'mbedtls_ssl_setup' will
+    // fail.
+    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
     }

     mbedtls_ssl_conf_rng(&ssl_pm->conf, mbedtls_ctr_drbg_random, &ssl_pm->ctr_drbg);
```

```diff
diff --git a/third_party/libwebsockets/CMakeLists.txt b/third_party/libwebsockets/CMakeLists.txt
index 92638143a..746f9b6a6 100644
--- a/third_party/libwebsockets/CMakeLists.txt
+++ b/third_party/libwebsockets/CMakeLists.txt
@@ -547,9 +547,12 @@ SET(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/lib")

 SET(LWS_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}")

-# Put absolute path of dynamic libraries into the object code. Some
-# architectures, notably Mac OS X, need this.
-SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
+# Commented out to avoid using absolute paths in the install_name on macOS.
+# When CMAKE_INSTALL_NAME_DIR is set to an absolute path, binaries that link
+# against libwebsockets will hardcode this absolute path, making the library
+# difficult to relocate and potentially causing "library not found" errors at
+# runtime if the library is installed in a different location.
+# SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
```

### Fix for linking mbedtls on Windows

Apply the following patch if the CMake version is higher than 3.24, as `find_package` supports the `GLOBAL` keyword since 3.24.

```diff
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
index e34151724..79b15089d 100644
--- a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
+++ b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
-        find_library(MBEDTLS_LIBRARY mbedtls)
-        find_library(MBEDX509_LIBRARY mbedx509)
-        find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+        # find_library(MBEDTLS_LIBRARY mbedtls)
+        # find_library(MBEDX509_LIBRARY mbedx509)
+        # find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+        # set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+
+        # Set the custom search path.
+        set(MBEDTLS_DIR "${MBEDTLS_BUILD_PATH}/cmake")
+
+        // NOTE (Liu): We should use 'find_package' rather than 'find_library'
+        // to search for the mbedtls libraries. 'find_library' only finds a
+        // library, shared or static, without any settings from the external
+        // library.

-        set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+        // The 'lib/tls/CMakeLists.txt' depends on the mbedtls libraries, so we set the scope to 'GLOBAL' here.
+        find_package(MbedTLS GLOBAL)
+        set(LWS_MBEDTLS_LIBRARIES MbedTLS::mbedcrypto MbedTLS::mbedx509 MbedTLS::mbedtls)
```

```diff
diff --git a/third_party/libwebsockets/BUILD.gn b/third_party/libwebsockets/BUILD.gn
index 4c89c2e2e..e6d641b9c 100644
--- a/third_party/libwebsockets/BUILD.gn
+++ b/third_party/libwebsockets/BUILD.gn
@@ -84,6 +84,10 @@ cmake_project("websockets") {
   }

   options += [
+  // The libwebsockets will use the 'MBEDTLS_BUILD_PATH' variable to set the
+  // search path of 'find_package'.
+  "MBEDTLS_BUILD_PATH=" + rebase_path("$root_gen_dir/cmake/mbedtls"),
+
```

And remove `#define USING_SHARED_MBEDTLS` in `third_party/libwebsockets/cmake/lws_config.h.in`.

## curl

Version: 8.1.2

[MIT-like license](https://github.com/curl/curl/blob/master/COPYING)

A command-line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP. libcurl offers a myriad of powerful features.

Please refer to `third_party/curl` for details.

### Patches for curl

Patch `lib/CMakeLists.txt`, to change the shared library name from `_imp.lib` to `.lib`.

```cmake
if(WIN32)
  if(BUILD_SHARED_LIBS)
    if(MSVC)
      // Add "_imp" as a suffix before the extension to avoid conflicting with
      // the statically linked "libcurl.lib"
      //set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX "_imp.lib")
      set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX ".lib")
    endif()
  endif()
endif()
```

Export `Curl_ws_done` in `lib/ws.h`, because this function needs to be called to prevent memory leaks.

```c
CURL_EXTERN void Curl_ws_done(struct Curl_easy *data);
// ^^^^^^^^
```

## clingo

Version: 5.8.0

[MIT license](https://github.com/potassco/clingo/blob/master/LICENSE.md)

A grounder and solver for logic programs.

Please refer to `third_party/clingo` for details.

ten\_manager actually uses the clingo-sys crate to call clingo, and clingo-sys includes its own clingo source code. However, this included clingo source code is version 5.6.2, which has compilation errors in a cmake 4 environment. Therefore, we need to use cargo's patch feature to use the latest local version of clingo source code (which has fixed the cmake 4 compilation errors). Additionally, since clingo-sys already includes clingo source code, we don't actually need the third\_party/clingo folder. However, because clingo-sys still compiles clingo as a shared library on Windows, we currently still need to use the third\_party/clingo folder to compile clingo's shared library on Windows, and then use gn scripts to copy the Windows shared library to the final output folder of ten\_manager. In the future, we should directly use the Windows shared library compiled from the clingo source code included in clingo-sys for copying, which would allow us to remove the third\_party/clingo folder.

## FFmpeg

Version: 6.0

[GPL or LGPL license](https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md)

The FFmpeg codebase is mainly LGPL-licensed with optional components licensed under GPL. Please refer to the LICENSE file for detailed information.

Used in ffmpeg extensions, primarily for testing purposes. Please refer to `third_party/ffmpeg` for details.

## libbacktrace

Version: 1.0

[BSD license](https://github.com/ianlancetaylor/libbacktrace/blob/master/LICENSE)

A C library that may be linked into a C/C++ program to produce symbolic backtraces.

> ⚠️ **Note:**
> We have significantly modified `libbacktrace` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/backtrace` for details.

## uthash

Version: 2.3.0

[BSD license](https://github.com/troydhanson/uthash/blob/master/LICENSE)

C macros for hash tables and more.

> ⚠️ **Note:**
> We have significantly modified `uthash` to conform to the naming conventions and folder structure of the TEN framework. Please refer to the files under `core/include/ten_utils/container` that have `uthash` mentioned in the file headers.

## uuid4

[MIT license](https://github.com/gpakosz/uuid4/blob/master/LICENSE.MIT)
[WTFPLv2 license](https://github.com/gpakosz/uuid4/blob/master/LICENSE.WTFPLv2)

UUID v4 generation in C.

> ⚠️ **Note:**
> We have significantly modified `uuid4` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/lib/sys/general/uuid.c` for details.

## zf\_log

[MIT license](https://github.com/wonder-mice/zf_log/blob/master/LICENSE)

Core logging library for C/ObjC/C++.

> ⚠️ **Note:**
> We have significantly modified `zf_log` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/log` for details.


# Message System (/docs/ten_framework/message_system)

## Message Types

In the TEN framework, messages are categorized into four types:

1. Command
2. Data
3. Video Frame
4. Audio Frame

The primary difference between a command and non-command message types is that commands have a result, while non-commands do not.

Corresponding to these types, extensions have four message callbacks:

1. `OnCmd`
2. `OnData`
3. `OnVideoFrame`
4. `OnAudioFrame`

Here is a classification diagram of TEN framework messages:

```text
┌── has result
│   └── Command
│       ├── TEN framework built-in command
│       │    => message name starts with `ten:`
│       └── Non-TEN framework built-in command
│            => message name does not start with `ten:`
└── no result
    ├── Data
    │   ├── TEN framework built-in data
    │   │    => message name starts with `ten:`
    │   └── Non-TEN framework built-in data
    │        => message name does not start with `ten:`
    ├── Video Frame
    │   ├── TEN framework built-in video frame
    │   │    => message name starts with `ten:`
    │   └── Non-TEN framework built-in video frame
    │        => message name does not start with `ten:`
    └── Audio Frame
        ├── TEN framework built-in audio frame
        │    => message name starts with `ten:`
        └── Non-TEN framework built-in audio frame
             => message name does not start with `ten:`
```

## Message Name

A message name in the TEN framework is used to differentiate messages of the same type but with different purposes. Extensions use different message names to determine the actions they need to take.

Naming rules for message names are as follows:

1. The first character must be a letter (`a-z`, `A-Z`) or an underscore (`_`).
2. Subsequent characters can be letters (`a-z`, `A-Z`), digits (`0-9`), or underscores (`_`).

## Message Flow

![Message Flow](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_flow.png)

The TEN framework supports three message flow patterns:

1. **One message, one result.**

   Suitable when an action produces a single result.

2. **One message, multiple results.**

   Suitable when an action produces multiple results.

3. **Multiple messages.**

   Suitable for actions that do not require results.

## Extension Message Interface

The `api` field in an extension's manifest presents the extension's external message interface. The `api` field in the manifest can contain the following types of API information:

* `cmd_in`
* `cmd_out`
* `data_in`
* `data_out`
* `audio_frame_in`
* `audio_frame_out`
* `video_frame_in`
* `video_frame_out`

An example of these API definitions is as follows:

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

The message pairing mechanism between different extensions in the TEN framework is similar to function calls in traditional programming languages. Here's a simple analogy:

1. The message name is akin to a function name, and the pairing mechanism between extensions' messages is like the function call mechanism.
2. Message properties are similar to function parameters.
3. When the TEN framework determines that an output message `foo` from extension A pairs with an input message `foo` from extension B, it's analogous to a function pointer in extension A pointing to the function `foo` in extension B.
4. The pairing is based solely on the message name, not the message properties. This means the TEN framework does not support a mechanism analogous to function overloading.

Therefore, the message name is crucial for the input/output messages of each extension, serving as the control plane interface for that extension.

Although the TEN framework provides a language-agnostic calling mechanism at the single-message level, it also allows static and dynamic checks on the calling relationships between extensions through the TEN schema. This helps detect potential issues, such as differing interpretations of a property type between a source and a destination extension. Users can define their message names and determine the calling relationships between extensions through graphs. The TEN framework also offers static check tools to ensure these relationships do not cause any problems.

## Message Ownership Concept in the TEN Framework

In the TEN framework, once a message is successfully sent, the extension that sent the message can no longer use it. This concept is crucial for maintaining message safety and thread safety within the framework.

![Message Processing Follows the Concept of Ownership](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_ownership.png)

There are two directions in which messages can be sent: **send** and **return**. Only command messages have a return direction; data, audio\_frame, and video\_frame messages do not. After a message is sent, it may go through various processing steps, and allowing an extension to continue using the message could lead to thread safety issues. For instance, another extension handling the message might be running in a different thread. Even if the initial extension continues to use the message without any visible issues, this behavior is undefined and should not be relied upon by developers.

The TEN runtime enforces various safeguards to ensure that an extension cannot use a message after it has been successfully sent.

### Message Ownership Transfer to the Extension

When the TEN framework delivers a message to an extension via the `OnCmd`, `OnData`, `OnVideoFrame`, or `OnAudioFrame` callbacks, the framework transfers ownership of the message to the extension. This transfer of ownership means that the TEN framework guarantees that only the receiving extension has access to the message, ensuring message safety and thread safety. The extension can freely use the message and must return ownership to the TEN framework when it no longer needs the message. After returning ownership, the extension must not interact with the message any further.

### Message Ownership Transfer to the TEN Framework

When an extension sends a message back to the TEN runtime using APIs like `SendCmd`, `ReturnResult`, `SendData`, `SendVideoFrame`, or `SendAudioFrame`, the extension transfers ownership of the message back to the TEN framework.

### Ownership of Data Related to Messages

The concept of ownership extends beyond the message itself to all resources associated with that message. Just like the message, each resource's ownership can only belong to one extension at a time, and it mirrors the message's ownership. This design ensures both message and thread safety within the TEN framework.

For example, if extension A owns a message and uses a buffer within that message, it must return ownership of both the message and the buffer to the TEN runtime when it no longer needs them.

### Copy Semantics by Default

To prevent issues such as data races caused by multiple extensions handling the same resource simultaneously, the TEN framework defaults to **copy semantics** when transmitting data across TEN runtime and extension boundaries. Copying ensures that ownership is separated, maintaining safety.

### Borrowing Semantics

The TEN framework also supports **borrowing semantics**, allowing an extension to use a resource without taking ownership. This approach enhances resource utilization while maintaining message and thread safety. Borrowing APIs must be used in conjunction with GiveBack APIs to ensure the safety and integrity of resources.

When an extension returns ownership of a message to the TEN runtime, the runtime will only accept the message if all associated resources have also been returned. For instance, if extension A borrows a data message's buffer but does not return the buffer's ownership before returning the message itself, the TEN runtime will reject the message (e.g., `SendData` will fail).

### Borrowing API

Extensions can directly access resources within a message using borrowing APIs, allowing temporary use without transferring ownership.

### GiveBack API

The GiveBack APIs are used to return the ownership of resources back to the TEN framework, ensuring that resources are safely managed and freed when no longer needed.

## Command

TEN clients and TEN extensions can send commands to other TEN extensions.

If a command is used in a situation where the result isn't needed, it's fine. TEN runtime will automatically discard any unnecessary results.

### Result

A result is the response to a command. For example, when TEN extension A sends command X to TEN extension B, after processing the command, TEN extension B will send a result back to TEN extension A. The result may include any meaningful details, allowing TEN extension A to gain more information about the execution outcome.

A typical result's JSON representation might look like this:

```json
{
  "ten": {
    "status_code": 400
  }
}
```


# Preparation (/docs/ten_framework/preparation)

The TEN framework supports Windows, Linux, and Mac as development environments, but it is recommended to use Linux or Mac.

## Development Environment

### Linux

Using Ubuntu as an example, you must first install the following packages:

* gcc: One of the C/C++ compilers supported by the TEN framework.
* clang: Another C/C++ compiler supported by the TEN framework.
* clang-format: A code formatting tool used by the TEN framework during runtime.
* clang-tidy: A static code analyzer.
* cmake: The build system. The version must be greater than 3.13.
* Python3: Required for the TEN framework Python binding.
* pytest: Used for the TEN framework's integration testing.

### Mac

To set up your Mac development environment, you need the following:

* XCode
* CocoaPods
* Personal Account

Follow these instructions to prepare the environment:

```shell
brew install llvm googletest doxygen ninja clang-format
brew install include-what-you-use
```

Since the version of `cmake` installed via `brew` might be outdated, if so, you need to install `cmake` from the official website and set up a symbolic link.

```shell
ln -s /Applications/CMake.app/Contents/bin/cmake /usr/local/bin/cmake
```

If `cmake` cannot find `clang-tidy`, you can resolve the issue with the following command:

```shell
ln -sf /usr/local/opt/llvm/bin/clang-tidy /usr/local/bin/clang-tidy
```

To build various language bindings for the TEN framework, you can install the required languages using the following command:

```shell
brew install python golang
```

### Windows

If you need to develop the TEN framework on Microsoft Windows, you can manually install the following packages:

* Visual Studio Community Version (up to 2022)

  Be sure to select the clang-related tools and add the clang binary path to the `PATH` environment variable.

* cmake

* python

> ⚠️ **Note:**
> When installing `cmake`/`python` on Windows, the installation path should not contain spaces.

## ten\_gn

The TEN framework uses `ten_gn` as its build system. `ten_gn` is a build system based on Google GN. The source code for `ten_gn` is located in the `core/ten_gn` directory of the TEN framework repository. Add the `ten_gn` directory path to the system `PATH` in your development environment.

## Using Docker Containers

We provide pre-written Docker files that allow you to create a container with all the necessary packages required to build the TEN framework from source.

### Ubuntu 22.04

Navigate to `tools/docker_for_building/ubuntu/22.04` and run the following commands to create and enter the build environment.

```shell
docker-compose up -d
docker-compose run ten-building-ubuntu-2204
```


# Profiling (/docs/ten_framework/profiling)

## TEN Framework GO Project

Use `gperftools` to profile your program's CPU usage and heap memory usage. The following steps describe how to prepare the environment, use `gperftools` to profile your program, and analyze the performance data.

### Installing `gperftools`

To install `gperftools` on Ubuntu, run the following commands:

```shell
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

### Profiling CPU Usage with `gperftools`

To profile the CPU usage of your program, run the following commands:

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/.../cpu

exec /.../program
```

When the program exits, a CPU performance data file will be generated. The data will be stored in the `/.../cpu.0001.prof` file.

If you want to generate CPU performance data periodically, you can set the `CPUPROFILE_FREQUENCY` environment variable. For example, to generate CPU performance data every 30 seconds, run the following command:

```shell
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/.../cpu CPUPROFILE_FREQUENCY=30 exec /.../program
```

#### Analyzing CPU Performance Data

To analyze the CPU performance data, run the following commands:

```shell
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### Profiling Heap Memory Usage with `gperftools`

To profile your program's heap memory usage, run the following commands:

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

It is recommended to use the `HEAP_PROFILE_TIME_INTERVAL` environment variable to periodically generate heap memory performance data. All heap memory performance data will be stored in the `HEAPPROFILE` directory.

#### Analyzing Heap Memory Performance Data

Each generated heap memory performance data file will have a `.heap` extension. To analyze the heap memory performance data, you can convert the `.heap` files to a human-readable text format. For example, to convert a `.heap` file in `/.../heap` to a text format, run the following command:

```shell
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

The performance data will look like this:

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

You can also convert the `.heap` file to other formats like PDF or SVG. For example, to convert a `.heap` file in `/.../heap` to PDF format, run the following command:

```shell
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

The performance data in PDF format will be similar to the text output but visualized graphically.

Additionally, you can compare two sets of heap memory performance data. For example, to compare the heap memory performance data in `/.../heap.0001.heap` and `/.../heap.0002.heap`, run the following command:

```shell
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

The PDF will show the differences in heap memory usage between the two data sets.

If you are generating heap memory performance data periodically, we provide a script to analyze all the heap memory performance data in a directory and generate an Excel file showing the heap memory usage trends. For example, to analyze all heap memory performance data in `/.../heap`, run the following command:

```shell
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

The script will generate raw performance data with symbols in the `/.../raw` directory, human-readable text performance data in the `/.../text` directory, and an Excel file in `/.../heap.xlsx`.

You can plot a line chart to display the heap memory usage trends.

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### Profiling with `pprof`

We recommend using `pprof` to profile programs written in Go. The easiest way to profile a Go program is by using `pprof_app_go` as your TEN application. When activated, the application reads environment variables to decide whether to activate the `pprof` server and heap profiler.

```shell
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### Analyzing `pprof` Performance Data

To convert heap performance data to a text format, use the following command:

```shell
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

To analyze all heap performance data and generate an Excel file, run:

```shell
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

You can plot a line chart to display the heap memory usage trends:

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

This will give you a visual representation of Go heap performance data trends.


# Version System (/docs/ten_framework/version_system)

The TEN framework follows the semver specification.

## Version Information of TEN Package

The version information of a TEN package is recorded in the `version` field of its own `manifest.json` file. For every TEN package, this `version` field is mandatory.

```json
{
  "version": "1.0.0"
}
```

## Version Requirement of Dependency

> ⚠️ **Note:**
> If no version requirement is specified for a dependency, the latest version is implied.

```text
└── ten_packages/
    ├── extension/
    ├── extension_group/
    ├── protocol/
    └── system/
```

These packages are distinguished by the value of the `type` field in their `manifest.json`.

```json
{
  "type": "system" // "extension", "extension_group", "protocol"
}
```

In a TEN package, the version requirements for the dependent TEN packages are specified as follows.

```json
{
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.1"
    },
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "1.0.3"
    },
    {
      "type": "extension",
      "name": "default_http_extension_cpp",
      "version": "1.0.0"
    },
    {
      "type": "extension_group",
      "name": "default_extension_group",
      "version": "1.0.0"
    }
  ]
}
```

## Version Requirements Specification

A version requirements string can contain multiple version requirement strings, separated by commas. Each version requirement string supports the following operators:

* `>`, `>=`, `=`

  Manually specify a version range or an exact dependency version.

* `~`

  Used to specify a minimum version with some flexibility for updates. If the major, minor, and patch versions are specified, or only the major and minor versions are specified, only patch-level changes are allowed. If only the major version is specified, both minor and patch-level changes are allowed.

* `^`

  The `^` operator can be omitted, meaning that if a version requirement string does not include any operator, it defaults to `^`. For example, the string `0.1.12` represents `^0.1.12`. Although `0.1.12` looks like a specific version, it actually specifies a version requirement that allows SemVer-compatible updates.

  Updates are allowed as long as they do not modify the leftmost non-zero number in the major, minor, or patch combination. In this case, if you upgrade the TEN package, it will update to version `0.1.13` (if it's the latest `0.1.z` version), but not to `0.2.0`. If the version string specified is `1.0`, it will update to `1.1` (if it's the latest `1.y` version), but not to `2.0`.

  > ⚠️ **Note:**
  > Versions `0.0.x` are considered incompatible with any other versions.


# Build System (/docs/ten_framework/0.11.59/build_system)

## Build System

### C++

The TEN framework uses the `ten_gn` build system to compile C++ apps and C++ extensions. `ten_gn` is a build system based on Google's GN.

### Golang

The TEN framework uses the standard go command to build TEN framework Golang projects.

## Building

### C++ App

To build a TEN framework C++ app, run the following commands in the root directory of the TEN framework C++ app.

`<os>` can be one of the following:

* linux
* mac
* win

`<arch>` can be one of the following:

* x86
* x64
* arm
* arm64

`<build_type>` can be one of the following:

* debug
* release

```shell
ten_gn gen <os> <arch> <build_type>
ten_gn build <os> <arch> <build_type>
```

### C++ Addon

The term `addon` here includes the following:

* extension
* extension group
* protocol

The method to build TEN framework C++ addons is the same as for building TEN framework C++ apps.

### Golang App

To build a TEN Golang project, run the following command in the root directory of the TEN Golang project:

```shell
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```


# Cloud Store (/docs/ten_framework/0.11.59/cloud_store)

The TEN Cloud Store, similar to Google Play on Android and the App Store on iOS, offers a variety of TEN packages. These packages can be downloaded into a TEN app to aid and enhance development.

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

## Information Stored in the Database for TEN Packages

1. **Package Type**

   A string representing the type of package, such as `extension`.

2. **Package Name**

   A string representing the name of the package, such as `builtin_extension`.

3. **Package Version**

   A string following semantic versioning (semver) that specifies the package version, such as `3.0.0`.

4. **Dependencies**

   An array of objects, each containing the following fields:

   * **Package Type**

     A string, matching the top-level `package type`.

   * **Package Name**

     A string, matching the top-level `package name`.

   * **Version**

     A string following semver that specifies the version requirements for the package, such as `~3.0.0`.


# Overview (/docs/ten_framework/0.11.59/concept_overview)

![TEN Framework](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_framework.png)

## TEN Runtime

The TEN runtime is an execution environment designed for running applications within the TEN framework. It provides a flexible architecture that supports extensions developed in multiple programming languages and handles various types of data streams. By managing the lifecycle of extensions, data flows, and threads, the TEN runtime offers developers a powerful tool for building a wide range of applications and services.

## App

An App in the TEN framework can operate as a standalone process or as a thread within an existing process. This flexibility allows for versatile deployment options depending on the needs of the application.

## Extension Group

An Extension Group is a mechanism that designates a specific thread for execution. All extensions within a given group are executed on the same thread, ensuring consistent and synchronized processing.

## Extension

An extension is the fundamental building block of the TEN framework. Developers can create extensions in various programming languages and combine them to build different applications and scenarios. The framework's design supports cross-language collaboration, enabling extensions written in different languages to work together seamlessly within the same application or service.

For example, a developer might use C++ to create an extension for real-time communication (RTC) due to its performance benefits in handling audio and video data, while developing an AI extension in Python to take advantage of its robust libraries for data analysis and machine learning. These extensions can then be integrated into a single application, leveraging the strengths of each language.

## Graph

A Graph in the TEN framework describes the data flow between extensions. It orchestrates how data moves from one extension to another, defining the participants and the flow of data between them. For example, you can route the output of a Speech-to-Text (STT) extension to a Large Language Model (LLM) extension for further processing.

The TEN framework supports four main types of data flows between extensions:

* Command
* Data
* Video Frame
* Audio Frame

By defining these data flows within a graph, developers can create inter-extension communication and unidirectional data streams, particularly useful for handling audio and video data.

## Relationship Among App, Group, and Extension

* **App**:

  An app can execute multiple graphs, which can be either statically predefined or dynamically assembled.

* **Graph**:

  A graph is formed by multiple extensions working together to create a meaningful scenario. Each graph instance operates as a session within the app.

* **Extension Group**:

  The concept of an extension group is analogous to a thread. Extensions written in the same language and within the same extension group run on the same thread during runtime. Developers do not need to manage threads directly; they simply "declare" the group to which each extension belongs.

* **Extension**:

  Each extension within the framework is assigned a unique ID, structured as: `app-uri/graph-name/group-name/extension-name`

![Hierarchical Relationship of Concepts](https://ten-framework-assets.s3.amazonaws.com/doc-assets/hierarchical_relationship_of_concepts.png)

## TEN Cloud Store

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

The TEN Cloud Store functions similarly to Google Play Store or Apple’s App Store, providing a marketplace for extensions. Developers can share their extensions or download those created by others. These extensions can be integrated into TEN apps, facilitating development and expanding functionality.

## TEN Manager

The TEN Manager is a tool that simplifies the management of extensions. It handles tasks such as uploading, sharing, and installing extensions, automatically managing dependencies between them and their environment. This makes the installation and publication of extensions convenient and efficient, streamlining the development process within the TEN framework.


# Dependencies (/docs/ten_framework/0.11.59/dependencies)

The TEN framework utilizes several third-party libraries. Some are used specifically for testing, while others are integrated into the TEN runtime. Below is a description of these libraries, along with any necessary modifications required for their use within the TEN framework.

## Google gn

Instance ID

| OS    | Arch  | Instance ID                                   |
| ----- | ----- | --------------------------------------------- |
| Linux | x64   | BzX0zkfwFeUn9MaDVqm6FugmTIy-hFpgNUx43O1fN00C  |
| Linux | arm64 | rT\_12w1Iv6ug8CJ4j0VQekA0qTDq6CwoAqGWasIKFcEC |
| Win   | x64   | 1QlqF0FPVt82ba5f48HxHpv5xPqOmyaThoR3TicuJ8QC  |

Download directly from [Google GN webpage](https://chrome-infra-packages.appspot.com/p/gn/gn/).

Commit ID

| OS  | Arch      | Commit ID                                |
| --- | --------- | ---------------------------------------- |
| Mac | universal | 18602f6cf1168cf78302024043edc02e8bad2ffb |

Clone and build from the [Gn repository](https://gn.googlesource.com/gn).

## Google ninja

Version: 1.12.1

Download directly from [Ninja release page](https://github.com/ninja-build/ninja/releases).

## yyjson

Version: 0.10.0

[MIT license](https://github.com/ibireme/yyjson/blob/master/LICENSE)

This is used in the TEN framework core to parse and generate JSON data. Please refer to `third_party/yyjson` for details.

## libuv

Version: 1.50.0

[MIT license](https://github.com/libuv/libuv#licensing)

This is one of the event loop libraries used in the TEN runtime. Please refer to `third_party/libuv` for details.

## msgpack-c

Version: 6.1.0

[Boost Software License, Version 1.0](https://github.com/msgpack/msgpack-c#license)

A MessagePack library for C. Please refer to `third_party/msgpack` for details.

## libwebsockets

Version: 4.3.2

[MIT license](https://github.com/warmcat/libwebsockets/blob/main/LICENSE)

Canonical libwebsockets.org networking library. Please refer to `third_party/libwebsockets` for details.

```diff
--- /home/wei/MyData/Temp/libwebsockets-4.3.2/CMakeLists.txt
+++ libwebsockets/CMakeLists.txt
@@ -851,6 +851,7 @@
  add_definitions(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE)
  # Fail the build if any warnings
  add_compile_options(/W3 /WX)
+ add_compile_options(/Zc:preprocessor /wd4819)
  # Unbreak MSVC broken preprocessor __VA_ARGS__ behaviour
  if (MSVC_VERSION GREATER 1925)
   add_compile_options(/Zc:preprocessor /wd5105)
```

Apply the following patch.

```diff
diff --git a/third_party/libwebsockets/cmake/lws_config.h.in b/third_party/libwebsockets/cmake/lws_config.h.in
index f3f4a9d79..e8d36c3ae 100644
--- a/third_party/libwebsockets/cmake/lws_config.h.in
+++ b/third_party/libwebsockets/cmake/lws_config.h.in
@@ -238,3 +238,9 @@
 #cmakedefine LWS_WITH_PLUGINS_API
 #cmakedefine LWS_HAVE_RTA_PREF

+// NOTE (Liu): The libwebsockets library will use external variables from mbedtls
+// if 'LWS_WITH_MBEDTLS' is enabled. On Windows, variable declarations in the header
+// file must start with '__declspec(dllimport)', otherwise, the
+// symbols cannot be accessed.
+// Refer to third_party/mbedtls/include/mbedtls/export.h.
+#define USING_SHARED_MBEDTLS
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
index e8a0cb2d4..84a164e90 100755
--- a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
+++ b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
@@ -183,7 +183,12 @@ int ssl_pm_new(SSL *ssl)
         mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, version);
     } else {
         mbedtls_ssl_conf_max_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
-    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, 1);
+
+    // NOTE: mbedtls added 'ssl_conf_version_check()' since v3.1.0, and
+    // mbedtls only enables TLS 1.2 by default. So the 'min_tls_version' and
+    // 'max_tls_version' must be '0x303', or 'mbedtls_ssl_setup' will
+    // fail.
+    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
     }

     mbedtls_ssl_conf_rng(&ssl_pm->conf, mbedtls_ctr_drbg_random, &ssl_pm->ctr_drbg);
```

```diff
diff --git a/third_party/libwebsockets/CMakeLists.txt b/third_party/libwebsockets/CMakeLists.txt
index 92638143a..746f9b6a6 100644
--- a/third_party/libwebsockets/CMakeLists.txt
+++ b/third_party/libwebsockets/CMakeLists.txt
@@ -547,9 +547,12 @@ SET(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/lib")

 SET(LWS_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}")

-# Put absolute path of dynamic libraries into the object code. Some
-# architectures, notably Mac OS X, need this.
-SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
+# Commented out to avoid using absolute paths in the install_name on macOS.
+# When CMAKE_INSTALL_NAME_DIR is set to an absolute path, binaries that link
+# against libwebsockets will hardcode this absolute path, making the library
+# difficult to relocate and potentially causing "library not found" errors at
+# runtime if the library is installed in a different location.
+# SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
```

### Fix for linking mbedtls on Windows

Apply the following patch if the CMake version is higher than 3.24, as `find_package` supports the `GLOBAL` keyword since 3.24.

```diff
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
index e34151724..79b15089d 100644
--- a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
+++ b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
-        find_library(MBEDTLS_LIBRARY mbedtls)
-        find_library(MBEDX509_LIBRARY mbedx509)
-        find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+        # find_library(MBEDTLS_LIBRARY mbedtls)
+        # find_library(MBEDX509_LIBRARY mbedx509)
+        # find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+        # set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+
+        # Set the custom search path.
+        set(MBEDTLS_DIR "${MBEDTLS_BUILD_PATH}/cmake")
+
+        // NOTE (Liu): We should use 'find_package' rather than 'find_library'
+        // to search for the mbedtls libraries. 'find_library' only finds a
+        // library, shared or static, without any settings from the external
+        // library.

-        set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+        // The 'lib/tls/CMakeLists.txt' depends on the mbedtls libraries, so we set the scope to 'GLOBAL' here.
+        find_package(MbedTLS GLOBAL)
+        set(LWS_MBEDTLS_LIBRARIES MbedTLS::mbedcrypto MbedTLS::mbedx509 MbedTLS::mbedtls)
```

```diff
diff --git a/third_party/libwebsockets/BUILD.gn b/third_party/libwebsockets/BUILD.gn
index 4c89c2e2e..e6d641b9c 100644
--- a/third_party/libwebsockets/BUILD.gn
+++ b/third_party/libwebsockets/BUILD.gn
@@ -84,6 +84,10 @@ cmake_project("websockets") {
   }

   options += [
+  // The libwebsockets will use the 'MBEDTLS_BUILD_PATH' variable to set the
+  // search path of 'find_package'.
+  "MBEDTLS_BUILD_PATH=" + rebase_path("$root_gen_dir/cmake/mbedtls"),
+
```

And remove `#define USING_SHARED_MBEDTLS` in `third_party/libwebsockets/cmake/lws_config.h.in`.

## curl

Version: 8.1.2

[MIT-like license](https://github.com/curl/curl/blob/master/COPYING)

A command-line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP. libcurl offers a myriad of powerful features.

Please refer to `third_party/curl` for details.

### Patches for curl

Patch `lib/CMakeLists.txt`, to change the shared library name from `_imp.lib` to `.lib`.

```cmake
if(WIN32)
  if(BUILD_SHARED_LIBS)
    if(MSVC)
      // Add "_imp" as a suffix before the extension to avoid conflicting with
      // the statically linked "libcurl.lib"
      //set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX "_imp.lib")
      set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX ".lib")
    endif()
  endif()
endif()
```

Export `Curl_ws_done` in `lib/ws.h`, because this function needs to be called to prevent memory leaks.

```c
CURL_EXTERN void Curl_ws_done(struct Curl_easy *data);
// ^^^^^^^^
```

## clingo

Version: 5.8.0

[MIT license](https://github.com/potassco/clingo/blob/master/LICENSE.md)

A grounder and solver for logic programs.

Please refer to `third_party/clingo` for details.

ten\_manager actually uses the clingo-sys crate to call clingo, and clingo-sys includes its own clingo source code. However, this included clingo source code is version 5.6.2, which has compilation errors in a cmake 4 environment. Therefore, we need to use cargo's patch feature to use the latest local version of clingo source code (which has fixed the cmake 4 compilation errors). Additionally, since clingo-sys already includes clingo source code, we don't actually need the third\_party/clingo folder. However, because clingo-sys still compiles clingo as a shared library on Windows, we currently still need to use the third\_party/clingo folder to compile clingo's shared library on Windows, and then use gn scripts to copy the Windows shared library to the final output folder of ten\_manager. In the future, we should directly use the Windows shared library compiled from the clingo source code included in clingo-sys for copying, which would allow us to remove the third\_party/clingo folder.

## FFmpeg

Version: 6.0

[GPL or LGPL license](https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md)

The FFmpeg codebase is mainly LGPL-licensed with optional components licensed under GPL. Please refer to the LICENSE file for detailed information.

Used in ffmpeg extensions, primarily for testing purposes. Please refer to `third_party/ffmpeg` for details.

## libbacktrace

Version: 1.0

[BSD license](https://github.com/ianlancetaylor/libbacktrace/blob/master/LICENSE)

A C library that may be linked into a C/C++ program to produce symbolic backtraces.

> ⚠️ **Note:**
> We have significantly modified `libbacktrace` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/backtrace` for details.

## uthash

Version: 2.3.0

[BSD license](https://github.com/troydhanson/uthash/blob/master/LICENSE)

C macros for hash tables and more.

> ⚠️ **Note:**
> We have significantly modified `uthash` to conform to the naming conventions and folder structure of the TEN framework. Please refer to the files under `core/include/ten_utils/container` that have `uthash` mentioned in the file headers.

## uuid4

[MIT license](https://github.com/gpakosz/uuid4/blob/master/LICENSE.MIT)
[WTFPLv2 license](https://github.com/gpakosz/uuid4/blob/master/LICENSE.WTFPLv2)

UUID v4 generation in C.

> ⚠️ **Note:**
> We have significantly modified `uuid4` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/lib/sys/general/uuid.c` for details.

## zf\_log

[MIT license](https://github.com/wonder-mice/zf_log/blob/master/LICENSE)

Core logging library for C/ObjC/C++.

> ⚠️ **Note:**
> We have significantly modified `zf_log` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/log` for details.


# Message System (/docs/ten_framework/0.11.59/message_system)

## Message Types

In the TEN framework, messages are categorized into four types:

1. Command
2. Data
3. Video Frame
4. Audio Frame

The primary difference between a command and non-command message types is that commands have a result, while non-commands do not.

Corresponding to these types, extensions have four message callbacks:

1. `OnCmd`
2. `OnData`
3. `OnVideoFrame`
4. `OnAudioFrame`

Here is a classification diagram of TEN framework messages:

```text
┌── has result
│   └── Command
│       ├── TEN framework built-in command
│       │    => message name starts with `ten:`
│       └── Non-TEN framework built-in command
│            => message name does not start with `ten:`
└── no result
    ├── Data
    │   ├── TEN framework built-in data
    │   │    => message name starts with `ten:`
    │   └── Non-TEN framework built-in data
    │        => message name does not start with `ten:`
    ├── Video Frame
    │   ├── TEN framework built-in video frame
    │   │    => message name starts with `ten:`
    │   └── Non-TEN framework built-in video frame
    │        => message name does not start with `ten:`
    └── Audio Frame
        ├── TEN framework built-in audio frame
        │    => message name starts with `ten:`
        └── Non-TEN framework built-in audio frame
             => message name does not start with `ten:`
```

## Message Name

A message name in the TEN framework is used to differentiate messages of the same type but with different purposes. Extensions use different message names to determine the actions they need to take.

Naming rules for message names are as follows:

1. The first character must be a letter (`a-z`, `A-Z`) or an underscore (`_`).
2. Subsequent characters can be letters (`a-z`, `A-Z`), digits (`0-9`), or underscores (`_`).

## Message Flow

![Message Flow](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_flow.png)

The TEN framework supports three message flow patterns:

1. **One message, one result.**

   Suitable when an action produces a single result.

2. **One message, multiple results.**

   Suitable when an action produces multiple results.

3. **Multiple messages.**

   Suitable for actions that do not require results.

## Extension Message Interface

The `api` field in an extension's manifest presents the extension's external message interface. The `api` field in the manifest can contain the following types of API information:

* `cmd_in`
* `cmd_out`
* `data_in`
* `data_out`
* `audio_frame_in`
* `audio_frame_out`
* `video_frame_in`
* `video_frame_out`

An example of these API definitions is as follows:

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

The message pairing mechanism between different extensions in the TEN framework is similar to function calls in traditional programming languages. Here's a simple analogy:

1. The message name is akin to a function name, and the pairing mechanism between extensions' messages is like the function call mechanism.
2. Message properties are similar to function parameters.
3. When the TEN framework determines that an output message `foo` from extension A pairs with an input message `foo` from extension B, it's analogous to a function pointer in extension A pointing to the function `foo` in extension B.
4. The pairing is based solely on the message name, not the message properties. This means the TEN framework does not support a mechanism analogous to function overloading.

Therefore, the message name is crucial for the input/output messages of each extension, serving as the control plane interface for that extension.

Although the TEN framework provides a language-agnostic calling mechanism at the single-message level, it also allows static and dynamic checks on the calling relationships between extensions through the TEN schema. This helps detect potential issues, such as differing interpretations of a property type between a source and a destination extension. Users can define their message names and determine the calling relationships between extensions through graphs. The TEN framework also offers static check tools to ensure these relationships do not cause any problems.

## Message Ownership Concept in the TEN Framework

In the TEN framework, once a message is successfully sent, the extension that sent the message can no longer use it. This concept is crucial for maintaining message safety and thread safety within the framework.

![Message Processing Follows the Concept of Ownership](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_ownership.png)

There are two directions in which messages can be sent: **send** and **return**. Only command messages have a return direction; data, audio\_frame, and video\_frame messages do not. After a message is sent, it may go through various processing steps, and allowing an extension to continue using the message could lead to thread safety issues. For instance, another extension handling the message might be running in a different thread. Even if the initial extension continues to use the message without any visible issues, this behavior is undefined and should not be relied upon by developers.

The TEN runtime enforces various safeguards to ensure that an extension cannot use a message after it has been successfully sent.

### Message Ownership Transfer to the Extension

When the TEN framework delivers a message to an extension via the `OnCmd`, `OnData`, `OnVideoFrame`, or `OnAudioFrame` callbacks, the framework transfers ownership of the message to the extension. This transfer of ownership means that the TEN framework guarantees that only the receiving extension has access to the message, ensuring message safety and thread safety. The extension can freely use the message and must return ownership to the TEN framework when it no longer needs the message. After returning ownership, the extension must not interact with the message any further.

### Message Ownership Transfer to the TEN Framework

When an extension sends a message back to the TEN runtime using APIs like `SendCmd`, `ReturnResult`, `SendData`, `SendVideoFrame`, or `SendAudioFrame`, the extension transfers ownership of the message back to the TEN framework.

### Ownership of Data Related to Messages

The concept of ownership extends beyond the message itself to all resources associated with that message. Just like the message, each resource's ownership can only belong to one extension at a time, and it mirrors the message's ownership. This design ensures both message and thread safety within the TEN framework.

For example, if extension A owns a message and uses a buffer within that message, it must return ownership of both the message and the buffer to the TEN runtime when it no longer needs them.

### Copy Semantics by Default

To prevent issues such as data races caused by multiple extensions handling the same resource simultaneously, the TEN framework defaults to **copy semantics** when transmitting data across TEN runtime and extension boundaries. Copying ensures that ownership is separated, maintaining safety.

### Borrowing Semantics

The TEN framework also supports **borrowing semantics**, allowing an extension to use a resource without taking ownership. This approach enhances resource utilization while maintaining message and thread safety. Borrowing APIs must be used in conjunction with GiveBack APIs to ensure the safety and integrity of resources.

When an extension returns ownership of a message to the TEN runtime, the runtime will only accept the message if all associated resources have also been returned. For instance, if extension A borrows a data message's buffer but does not return the buffer's ownership before returning the message itself, the TEN runtime will reject the message (e.g., `SendData` will fail).

### Borrowing API

Extensions can directly access resources within a message using borrowing APIs, allowing temporary use without transferring ownership.

### GiveBack API

The GiveBack APIs are used to return the ownership of resources back to the TEN framework, ensuring that resources are safely managed and freed when no longer needed.

## Command

TEN clients and TEN extensions can send commands to other TEN extensions.

If a command is used in a situation where the result isn't needed, it's fine. TEN runtime will automatically discard any unnecessary results.

### Result

A result is the response to a command. For example, when TEN extension A sends command X to TEN extension B, after processing the command, TEN extension B will send a result back to TEN extension A. The result may include any meaningful details, allowing TEN extension A to gain more information about the execution outcome.

A typical result's JSON representation might look like this:

```json
{
  "ten": {
    "status_code": 400
  }
}
```


# Preparation (/docs/ten_framework/0.11.59/preparation)

The TEN framework supports Windows, Linux, and Mac as development environments, but it is recommended to use Linux or Mac.

## Development Environment

### Linux

Using Ubuntu as an example, you must first install the following packages:

* gcc: One of the C/C++ compilers supported by the TEN framework.
* clang: Another C/C++ compiler supported by the TEN framework.
* clang-format: A code formatting tool used by the TEN framework during runtime.
* clang-tidy: A static code analyzer.
* cmake: The build system. The version must be greater than 3.13.
* Python3: Required for the TEN framework Python binding.
* pytest: Used for the TEN framework's integration testing.

### Mac

To set up your Mac development environment, you need the following:

* XCode
* CocoaPods
* Personal Account

Follow these instructions to prepare the environment:

```shell
brew install llvm googletest doxygen ninja clang-format
brew install include-what-you-use
```

Since the version of `cmake` installed via `brew` might be outdated, if so, you need to install `cmake` from the official website and set up a symbolic link.

```shell
ln -s /Applications/CMake.app/Contents/bin/cmake /usr/local/bin/cmake
```

If `cmake` cannot find `clang-tidy`, you can resolve the issue with the following command:

```shell
ln -sf /usr/local/opt/llvm/bin/clang-tidy /usr/local/bin/clang-tidy
```

To build various language bindings for the TEN framework, you can install the required languages using the following command:

```shell
brew install python golang
```

### Windows

If you need to develop the TEN framework on Microsoft Windows, you can manually install the following packages:

* Visual Studio Community Version (up to 2022)

  Be sure to select the clang-related tools and add the clang binary path to the `PATH` environment variable.

* cmake

* python

> ⚠️ **Note:**
> When installing `cmake`/`python` on Windows, the installation path should not contain spaces.

## ten\_gn

The TEN framework uses `ten_gn` as its build system. `ten_gn` is a build system based on Google GN. The source code for `ten_gn` is located in the `core/ten_gn` directory of the TEN framework repository. Add the `ten_gn` directory path to the system `PATH` in your development environment.

## Using Docker Containers

We provide pre-written Docker files that allow you to create a container with all the necessary packages required to build the TEN framework from source.

### Ubuntu 22.04

Navigate to `tools/docker_for_building/ubuntu/22.04` and run the following commands to create and enter the build environment.

```shell
docker-compose up -d
docker-compose run ten-building-ubuntu-2204
```


# Profiling (/docs/ten_framework/0.11.59/profiling)

## TEN Framework GO Project

Use `gperftools` to profile your program's CPU usage and heap memory usage. The following steps describe how to prepare the environment, use `gperftools` to profile your program, and analyze the performance data.

### Installing `gperftools`

To install `gperftools` on Ubuntu, run the following commands:

```shell
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

### Profiling CPU Usage with `gperftools`

To profile the CPU usage of your program, run the following commands:

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/.../cpu

exec /.../program
```

When the program exits, a CPU performance data file will be generated. The data will be stored in the `/.../cpu.0001.prof` file.

If you want to generate CPU performance data periodically, you can set the `CPUPROFILE_FREQUENCY` environment variable. For example, to generate CPU performance data every 30 seconds, run the following command:

```shell
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/.../cpu CPUPROFILE_FREQUENCY=30 exec /.../program
```

#### Analyzing CPU Performance Data

To analyze the CPU performance data, run the following commands:

```shell
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### Profiling Heap Memory Usage with `gperftools`

To profile your program's heap memory usage, run the following commands:

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

It is recommended to use the `HEAP_PROFILE_TIME_INTERVAL` environment variable to periodically generate heap memory performance data. All heap memory performance data will be stored in the `HEAPPROFILE` directory.

#### Analyzing Heap Memory Performance Data

Each generated heap memory performance data file will have a `.heap` extension. To analyze the heap memory performance data, you can convert the `.heap` files to a human-readable text format. For example, to convert a `.heap` file in `/.../heap` to a text format, run the following command:

```shell
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

The performance data will look like this:

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

You can also convert the `.heap` file to other formats like PDF or SVG. For example, to convert a `.heap` file in `/.../heap` to PDF format, run the following command:

```shell
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

The performance data in PDF format will be similar to the text output but visualized graphically.

Additionally, you can compare two sets of heap memory performance data. For example, to compare the heap memory performance data in `/.../heap.0001.heap` and `/.../heap.0002.heap`, run the following command:

```shell
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

The PDF will show the differences in heap memory usage between the two data sets.

If you are generating heap memory performance data periodically, we provide a script to analyze all the heap memory performance data in a directory and generate an Excel file showing the heap memory usage trends. For example, to analyze all heap memory performance data in `/.../heap`, run the following command:

```shell
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

The script will generate raw performance data with symbols in the `/.../raw` directory, human-readable text performance data in the `/.../text` directory, and an Excel file in `/.../heap.xlsx`.

You can plot a line chart to display the heap memory usage trends.

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### Profiling with `pprof`

We recommend using `pprof` to profile programs written in Go. The easiest way to profile a Go program is by using `pprof_app_go` as your TEN application. When activated, the application reads environment variables to decide whether to activate the `pprof` server and heap profiler.

```shell
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### Analyzing `pprof` Performance Data

To convert heap performance data to a text format, use the following command:

```shell
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

To analyze all heap performance data and generate an Excel file, run:

```shell
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

You can plot a line chart to display the heap memory usage trends:

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

This will give you a visual representation of Go heap performance data trends.


# Version System (/docs/ten_framework/0.11.59/version_system)

The TEN framework follows the semver specification.

## Version Information of TEN Package

The version information of a TEN package is recorded in the `version` field of its own `manifest.json` file. For every TEN package, this `version` field is mandatory.

```json
{
  "version": "1.0.0"
}
```

## Version Requirement of Dependency

> ⚠️ **Note:**
> If no version requirement is specified for a dependency, the latest version is implied.

```text
└── ten_packages/
    ├── extension/
    ├── extension_group/
    ├── protocol/
    └── system/
```

These packages are distinguished by the value of the `type` field in their `manifest.json`.

```json
{
  "type": "system" // "extension", "extension_group", "protocol"
}
```

In a TEN package, the version requirements for the dependent TEN packages are specified as follows.

```json
{
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.1"
    },
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "1.0.3"
    },
    {
      "type": "extension",
      "name": "default_http_extension_cpp",
      "version": "1.0.0"
    },
    {
      "type": "extension_group",
      "name": "default_extension_group",
      "version": "1.0.0"
    }
  ]
}
```

## Version Requirements Specification

A version requirements string can contain multiple version requirement strings, separated by commas. Each version requirement string supports the following operators:

* `>`, `>=`, `=`

  Manually specify a version range or an exact dependency version.

* `~`

  Used to specify a minimum version with some flexibility for updates. If the major, minor, and patch versions are specified, or only the major and minor versions are specified, only patch-level changes are allowed. If only the major version is specified, both minor and patch-level changes are allowed.

* `^`

  The `^` operator can be omitted, meaning that if a version requirement string does not include any operator, it defaults to `^`. For example, the string `0.1.12` represents `^0.1.12`. Although `0.1.12` looks like a specific version, it actually specifies a version requirement that allows SemVer-compatible updates.

  Updates are allowed as long as they do not modify the leftmost non-zero number in the major, minor, or patch combination. In this case, if you upgrade the TEN package, it will update to version `0.1.13` (if it's the latest `0.1.z` version), but not to `0.2.0`. If the version string specified is `1.0`, it will update to `1.1` (if it's the latest `1.y` version), but not to `2.0`.

  > ⚠️ **Note:**
  > Versions `0.0.x` are considered incompatible with any other versions.


# Build System (/docs/ten_framework/0.11.60/build_system)

## Build System

### C++

The TEN framework uses the `ten_gn` build system to compile C++ apps and C++ extensions. `ten_gn` is a build system based on Google's GN.

### Golang

The TEN framework uses the standard go command to build TEN framework Golang projects.

## Building

### C++ App

To build a TEN framework C++ app, run the following commands in the root directory of the TEN framework C++ app.

`<os>` can be one of the following:

* linux
* mac
* win

`<arch>` can be one of the following:

* x86
* x64
* arm
* arm64

`<build_type>` can be one of the following:

* debug
* release

```shell
ten_gn gen <os> <arch> <build_type>
ten_gn build <os> <arch> <build_type>
```

### C++ Addon

The term `addon` here includes the following:

* extension
* extension group
* protocol

The method to build TEN framework C++ addons is the same as for building TEN framework C++ apps.

### Golang App

To build a TEN Golang project, run the following command in the root directory of the TEN Golang project:

```shell
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```


# Cloud Store (/docs/ten_framework/0.11.60/cloud_store)

The TEN Cloud Store, similar to Google Play on Android and the App Store on iOS, offers a variety of TEN packages. These packages can be downloaded into a TEN app to aid and enhance development.

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

## Information Stored in the Database for TEN Packages

1. **Package Type**

   A string representing the type of package, such as `extension`.

2. **Package Name**

   A string representing the name of the package, such as `builtin_extension`.

3. **Package Version**

   A string following semantic versioning (semver) that specifies the package version, such as `3.0.0`.

4. **Dependencies**

   An array of objects, each containing the following fields:

   * **Package Type**

     A string, matching the top-level `package type`.

   * **Package Name**

     A string, matching the top-level `package name`.

   * **Version**

     A string following semver that specifies the version requirements for the package, such as `~3.0.0`.


# Overview (/docs/ten_framework/0.11.60/concept_overview)

![TEN Framework](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_framework.png)

## TEN Runtime

The TEN runtime is an execution environment designed for running applications within the TEN framework. It provides a flexible architecture that supports extensions developed in multiple programming languages and handles various types of data streams. By managing the lifecycle of extensions, data flows, and threads, the TEN runtime offers developers a powerful tool for building a wide range of applications and services.

## App

An App in the TEN framework can operate as a standalone process or as a thread within an existing process. This flexibility allows for versatile deployment options depending on the needs of the application.

## Extension Group

An Extension Group is a mechanism that designates a specific thread for execution. All extensions within a given group are executed on the same thread, ensuring consistent and synchronized processing.

## Extension

An extension is the fundamental building block of the TEN framework. Developers can create extensions in various programming languages and combine them to build different applications and scenarios. The framework's design supports cross-language collaboration, enabling extensions written in different languages to work together seamlessly within the same application or service.

For example, a developer might use C++ to create an extension for real-time communication (RTC) due to its performance benefits in handling audio and video data, while developing an AI extension in Python to take advantage of its robust libraries for data analysis and machine learning. These extensions can then be integrated into a single application, leveraging the strengths of each language.

## Graph

A Graph in the TEN framework describes the data flow between extensions. It orchestrates how data moves from one extension to another, defining the participants and the flow of data between them. For example, you can route the output of a Speech-to-Text (STT) extension to a Large Language Model (LLM) extension for further processing.

The TEN framework supports four main types of data flows between extensions:

* Command
* Data
* Video Frame
* Audio Frame

By defining these data flows within a graph, developers can create inter-extension communication and unidirectional data streams, particularly useful for handling audio and video data.

## Relationship Among App, Group, and Extension

* **App**:

  An app can execute multiple graphs, which can be either statically predefined or dynamically assembled.

* **Graph**:

  A graph is formed by multiple extensions working together to create a meaningful scenario. Each graph instance operates as a session within the app.

* **Extension Group**:

  The concept of an extension group is analogous to a thread. Extensions written in the same language and within the same extension group run on the same thread during runtime. Developers do not need to manage threads directly; they simply "declare" the group to which each extension belongs.

* **Extension**:

  Each extension within the framework is assigned a unique ID, structured as: `app-uri/graph-name/group-name/extension-name`

![Hierarchical Relationship of Concepts](https://ten-framework-assets.s3.amazonaws.com/doc-assets/hierarchical_relationship_of_concepts.png)

## TEN Cloud Store

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

The TEN Cloud Store functions similarly to Google Play Store or Apple’s App Store, providing a marketplace for extensions. Developers can share their extensions or download those created by others. These extensions can be integrated into TEN apps, facilitating development and expanding functionality.

## TEN Manager

The TEN Manager is a tool that simplifies the management of extensions. It handles tasks such as uploading, sharing, and installing extensions, automatically managing dependencies between them and their environment. This makes the installation and publication of extensions convenient and efficient, streamlining the development process within the TEN framework.


# Dependencies (/docs/ten_framework/0.11.60/dependencies)

The TEN framework utilizes several third-party libraries. Some are used specifically for testing, while others are integrated into the TEN runtime. Below is a description of these libraries, along with any necessary modifications required for their use within the TEN framework.

## Google gn

Instance ID

| OS    | Arch  | Instance ID                                   |
| ----- | ----- | --------------------------------------------- |
| Linux | x64   | BzX0zkfwFeUn9MaDVqm6FugmTIy-hFpgNUx43O1fN00C  |
| Linux | arm64 | rT\_12w1Iv6ug8CJ4j0VQekA0qTDq6CwoAqGWasIKFcEC |
| Win   | x64   | 1QlqF0FPVt82ba5f48HxHpv5xPqOmyaThoR3TicuJ8QC  |

Download directly from [Google GN webpage](https://chrome-infra-packages.appspot.com/p/gn/gn/).

Commit ID

| OS  | Arch      | Commit ID                                |
| --- | --------- | ---------------------------------------- |
| Mac | universal | 18602f6cf1168cf78302024043edc02e8bad2ffb |

Clone and build from the [Gn repository](https://gn.googlesource.com/gn).

## Google ninja

Version: 1.12.1

Download directly from [Ninja release page](https://github.com/ninja-build/ninja/releases).

## yyjson

Version: 0.10.0

[MIT license](https://github.com/ibireme/yyjson/blob/master/LICENSE)

This is used in the TEN framework core to parse and generate JSON data. Please refer to `third_party/yyjson` for details.

## libuv

Version: 1.50.0

[MIT license](https://github.com/libuv/libuv#licensing)

This is one of the event loop libraries used in the TEN runtime. Please refer to `third_party/libuv` for details.

## msgpack-c

Version: 6.1.0

[Boost Software License, Version 1.0](https://github.com/msgpack/msgpack-c#license)

A MessagePack library for C. Please refer to `third_party/msgpack` for details.

## libwebsockets

Version: 4.3.2

[MIT license](https://github.com/warmcat/libwebsockets/blob/main/LICENSE)

Canonical libwebsockets.org networking library. Please refer to `third_party/libwebsockets` for details.

```diff
--- /home/wei/MyData/Temp/libwebsockets-4.3.2/CMakeLists.txt
+++ libwebsockets/CMakeLists.txt
@@ -851,6 +851,7 @@
  add_definitions(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE)
  # Fail the build if any warnings
  add_compile_options(/W3 /WX)
+ add_compile_options(/Zc:preprocessor /wd4819)
  # Unbreak MSVC broken preprocessor __VA_ARGS__ behaviour
  if (MSVC_VERSION GREATER 1925)
   add_compile_options(/Zc:preprocessor /wd5105)
```

Apply the following patch.

```diff
diff --git a/third_party/libwebsockets/cmake/lws_config.h.in b/third_party/libwebsockets/cmake/lws_config.h.in
index f3f4a9d79..e8d36c3ae 100644
--- a/third_party/libwebsockets/cmake/lws_config.h.in
+++ b/third_party/libwebsockets/cmake/lws_config.h.in
@@ -238,3 +238,9 @@
 #cmakedefine LWS_WITH_PLUGINS_API
 #cmakedefine LWS_HAVE_RTA_PREF

+// NOTE (Liu): The libwebsockets library will use external variables from mbedtls
+// if 'LWS_WITH_MBEDTLS' is enabled. On Windows, variable declarations in the header
+// file must start with '__declspec(dllimport)', otherwise, the
+// symbols cannot be accessed.
+// Refer to third_party/mbedtls/include/mbedtls/export.h.
+#define USING_SHARED_MBEDTLS
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
index e8a0cb2d4..84a164e90 100755
--- a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
+++ b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
@@ -183,7 +183,12 @@ int ssl_pm_new(SSL *ssl)
         mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, version);
     } else {
         mbedtls_ssl_conf_max_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
-    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, 1);
+
+    // NOTE: mbedtls added 'ssl_conf_version_check()' since v3.1.0, and
+    // mbedtls only enables TLS 1.2 by default. So the 'min_tls_version' and
+    // 'max_tls_version' must be '0x303', or 'mbedtls_ssl_setup' will
+    // fail.
+    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
     }

     mbedtls_ssl_conf_rng(&ssl_pm->conf, mbedtls_ctr_drbg_random, &ssl_pm->ctr_drbg);
```

```diff
diff --git a/third_party/libwebsockets/CMakeLists.txt b/third_party/libwebsockets/CMakeLists.txt
index 92638143a..746f9b6a6 100644
--- a/third_party/libwebsockets/CMakeLists.txt
+++ b/third_party/libwebsockets/CMakeLists.txt
@@ -547,9 +547,12 @@ SET(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/lib")

 SET(LWS_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}")

-# Put absolute path of dynamic libraries into the object code. Some
-# architectures, notably Mac OS X, need this.
-SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
+# Commented out to avoid using absolute paths in the install_name on macOS.
+# When CMAKE_INSTALL_NAME_DIR is set to an absolute path, binaries that link
+# against libwebsockets will hardcode this absolute path, making the library
+# difficult to relocate and potentially causing "library not found" errors at
+# runtime if the library is installed in a different location.
+# SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
```

### Fix for linking mbedtls on Windows

Apply the following patch if the CMake version is higher than 3.24, as `find_package` supports the `GLOBAL` keyword since 3.24.

```diff
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
index e34151724..79b15089d 100644
--- a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
+++ b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
-        find_library(MBEDTLS_LIBRARY mbedtls)
-        find_library(MBEDX509_LIBRARY mbedx509)
-        find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+        # find_library(MBEDTLS_LIBRARY mbedtls)
+        # find_library(MBEDX509_LIBRARY mbedx509)
+        # find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+        # set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+
+        # Set the custom search path.
+        set(MBEDTLS_DIR "${MBEDTLS_BUILD_PATH}/cmake")
+
+        // NOTE (Liu): We should use 'find_package' rather than 'find_library'
+        // to search for the mbedtls libraries. 'find_library' only finds a
+        // library, shared or static, without any settings from the external
+        // library.

-        set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+        // The 'lib/tls/CMakeLists.txt' depends on the mbedtls libraries, so we set the scope to 'GLOBAL' here.
+        find_package(MbedTLS GLOBAL)
+        set(LWS_MBEDTLS_LIBRARIES MbedTLS::mbedcrypto MbedTLS::mbedx509 MbedTLS::mbedtls)
```

```diff
diff --git a/third_party/libwebsockets/BUILD.gn b/third_party/libwebsockets/BUILD.gn
index 4c89c2e2e..e6d641b9c 100644
--- a/third_party/libwebsockets/BUILD.gn
+++ b/third_party/libwebsockets/BUILD.gn
@@ -84,6 +84,10 @@ cmake_project("websockets") {
   }

   options += [
+  // The libwebsockets will use the 'MBEDTLS_BUILD_PATH' variable to set the
+  // search path of 'find_package'.
+  "MBEDTLS_BUILD_PATH=" + rebase_path("$root_gen_dir/cmake/mbedtls"),
+
```

And remove `#define USING_SHARED_MBEDTLS` in `third_party/libwebsockets/cmake/lws_config.h.in`.

## curl

Version: 8.1.2

[MIT-like license](https://github.com/curl/curl/blob/master/COPYING)

A command-line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP. libcurl offers a myriad of powerful features.

Please refer to `third_party/curl` for details.

### Patches for curl

Patch `lib/CMakeLists.txt`, to change the shared library name from `_imp.lib` to `.lib`.

```cmake
if(WIN32)
  if(BUILD_SHARED_LIBS)
    if(MSVC)
      // Add "_imp" as a suffix before the extension to avoid conflicting with
      // the statically linked "libcurl.lib"
      //set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX "_imp.lib")
      set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX ".lib")
    endif()
  endif()
endif()
```

Export `Curl_ws_done` in `lib/ws.h`, because this function needs to be called to prevent memory leaks.

```c
CURL_EXTERN void Curl_ws_done(struct Curl_easy *data);
// ^^^^^^^^
```

## clingo

Version: 5.8.0

[MIT license](https://github.com/potassco/clingo/blob/master/LICENSE.md)

A grounder and solver for logic programs.

Please refer to `third_party/clingo` for details.

ten\_manager actually uses the clingo-sys crate to call clingo, and clingo-sys includes its own clingo source code. However, this included clingo source code is version 5.6.2, which has compilation errors in a cmake 4 environment. Therefore, we need to use cargo's patch feature to use the latest local version of clingo source code (which has fixed the cmake 4 compilation errors). Additionally, since clingo-sys already includes clingo source code, we don't actually need the third\_party/clingo folder. However, because clingo-sys still compiles clingo as a shared library on Windows, we currently still need to use the third\_party/clingo folder to compile clingo's shared library on Windows, and then use gn scripts to copy the Windows shared library to the final output folder of ten\_manager. In the future, we should directly use the Windows shared library compiled from the clingo source code included in clingo-sys for copying, which would allow us to remove the third\_party/clingo folder.

## FFmpeg

Version: 6.0

[GPL or LGPL license](https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md)

The FFmpeg codebase is mainly LGPL-licensed with optional components licensed under GPL. Please refer to the LICENSE file for detailed information.

Used in ffmpeg extensions, primarily for testing purposes. Please refer to `third_party/ffmpeg` for details.

## libbacktrace

Version: 1.0

[BSD license](https://github.com/ianlancetaylor/libbacktrace/blob/master/LICENSE)

A C library that may be linked into a C/C++ program to produce symbolic backtraces.

> ⚠️ **Note:**
> We have significantly modified `libbacktrace` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/backtrace` for details.

## uthash

Version: 2.3.0

[BSD license](https://github.com/troydhanson/uthash/blob/master/LICENSE)

C macros for hash tables and more.

> ⚠️ **Note:**
> We have significantly modified `uthash` to conform to the naming conventions and folder structure of the TEN framework. Please refer to the files under `core/include/ten_utils/container` that have `uthash` mentioned in the file headers.

## uuid4

[MIT license](https://github.com/gpakosz/uuid4/blob/master/LICENSE.MIT)
[WTFPLv2 license](https://github.com/gpakosz/uuid4/blob/master/LICENSE.WTFPLv2)

UUID v4 generation in C.

> ⚠️ **Note:**
> We have significantly modified `uuid4` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/lib/sys/general/uuid.c` for details.

## zf\_log

[MIT license](https://github.com/wonder-mice/zf_log/blob/master/LICENSE)

Core logging library for C/ObjC/C++.

> ⚠️ **Note:**
> We have significantly modified `zf_log` to conform to the naming conventions and folder structure of the TEN framework. Please refer to `core/src/ten_utils/log` for details.


# Message System (/docs/ten_framework/0.11.60/message_system)

## Message Types

In the TEN framework, messages are categorized into four types:

1. Command
2. Data
3. Video Frame
4. Audio Frame

The primary difference between a command and non-command message types is that commands have a result, while non-commands do not.

Corresponding to these types, extensions have four message callbacks:

1. `OnCmd`
2. `OnData`
3. `OnVideoFrame`
4. `OnAudioFrame`

Here is a classification diagram of TEN framework messages:

```text
┌── has result
│   └── Command
│       ├── TEN framework built-in command
│       │    => message name starts with `ten:`
│       └── Non-TEN framework built-in command
│            => message name does not start with `ten:`
└── no result
    ├── Data
    │   ├── TEN framework built-in data
    │   │    => message name starts with `ten:`
    │   └── Non-TEN framework built-in data
    │        => message name does not start with `ten:`
    ├── Video Frame
    │   ├── TEN framework built-in video frame
    │   │    => message name starts with `ten:`
    │   └── Non-TEN framework built-in video frame
    │        => message name does not start with `ten:`
    └── Audio Frame
        ├── TEN framework built-in audio frame
        │    => message name starts with `ten:`
        └── Non-TEN framework built-in audio frame
             => message name does not start with `ten:`
```

## Message Name

A message name in the TEN framework is used to differentiate messages of the same type but with different purposes. Extensions use different message names to determine the actions they need to take.

Naming rules for message names are as follows:

1. The first character must be a letter (`a-z`, `A-Z`) or an underscore (`_`).
2. Subsequent characters can be letters (`a-z`, `A-Z`), digits (`0-9`), or underscores (`_`).

## Message Flow

![Message Flow](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_flow.png)

The TEN framework supports three message flow patterns:

1. **One message, one result.**

   Suitable when an action produces a single result.

2. **One message, multiple results.**

   Suitable when an action produces multiple results.

3. **Multiple messages.**

   Suitable for actions that do not require results.

## Extension Message Interface

The `api` field in an extension's manifest presents the extension's external message interface. The `api` field in the manifest can contain the following types of API information:

* `cmd_in`
* `cmd_out`
* `data_in`
* `data_out`
* `audio_frame_in`
* `audio_frame_out`
* `video_frame_in`
* `video_frame_out`

An example of these API definitions is as follows:

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

The message pairing mechanism between different extensions in the TEN framework is similar to function calls in traditional programming languages. Here's a simple analogy:

1. The message name is akin to a function name, and the pairing mechanism between extensions' messages is like the function call mechanism.
2. Message properties are similar to function parameters.
3. When the TEN framework determines that an output message `foo` from extension A pairs with an input message `foo` from extension B, it's analogous to a function pointer in extension A pointing to the function `foo` in extension B.
4. The pairing is based solely on the message name, not the message properties. This means the TEN framework does not support a mechanism analogous to function overloading.

Therefore, the message name is crucial for the input/output messages of each extension, serving as the control plane interface for that extension.

Although the TEN framework provides a language-agnostic calling mechanism at the single-message level, it also allows static and dynamic checks on the calling relationships between extensions through the TEN schema. This helps detect potential issues, such as differing interpretations of a property type between a source and a destination extension. Users can define their message names and determine the calling relationships between extensions through graphs. The TEN framework also offers static check tools to ensure these relationships do not cause any problems.

## Message Ownership Concept in the TEN Framework

In the TEN framework, once a message is successfully sent, the extension that sent the message can no longer use it. This concept is crucial for maintaining message safety and thread safety within the framework.

![Message Processing Follows the Concept of Ownership](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_ownership.png)

There are two directions in which messages can be sent: **send** and **return**. Only command messages have a return direction; data, audio\_frame, and video\_frame messages do not. After a message is sent, it may go through various processing steps, and allowing an extension to continue using the message could lead to thread safety issues. For instance, another extension handling the message might be running in a different thread. Even if the initial extension continues to use the message without any visible issues, this behavior is undefined and should not be relied upon by developers.

The TEN runtime enforces various safeguards to ensure that an extension cannot use a message after it has been successfully sent.

### Message Ownership Transfer to the Extension

When the TEN framework delivers a message to an extension via the `OnCmd`, `OnData`, `OnVideoFrame`, or `OnAudioFrame` callbacks, the framework transfers ownership of the message to the extension. This transfer of ownership means that the TEN framework guarantees that only the receiving extension has access to the message, ensuring message safety and thread safety. The extension can freely use the message and must return ownership to the TEN framework when it no longer needs the message. After returning ownership, the extension must not interact with the message any further.

### Message Ownership Transfer to the TEN Framework

When an extension sends a message back to the TEN runtime using APIs like `SendCmd`, `ReturnResult`, `SendData`, `SendVideoFrame`, or `SendAudioFrame`, the extension transfers ownership of the message back to the TEN framework.

### Ownership of Data Related to Messages

The concept of ownership extends beyond the message itself to all resources associated with that message. Just like the message, each resource's ownership can only belong to one extension at a time, and it mirrors the message's ownership. This design ensures both message and thread safety within the TEN framework.

For example, if extension A owns a message and uses a buffer within that message, it must return ownership of both the message and the buffer to the TEN runtime when it no longer needs them.

### Copy Semantics by Default

To prevent issues such as data races caused by multiple extensions handling the same resource simultaneously, the TEN framework defaults to **copy semantics** when transmitting data across TEN runtime and extension boundaries. Copying ensures that ownership is separated, maintaining safety.

### Borrowing Semantics

The TEN framework also supports **borrowing semantics**, allowing an extension to use a resource without taking ownership. This approach enhances resource utilization while maintaining message and thread safety. Borrowing APIs must be used in conjunction with GiveBack APIs to ensure the safety and integrity of resources.

When an extension returns ownership of a message to the TEN runtime, the runtime will only accept the message if all associated resources have also been returned. For instance, if extension A borrows a data message's buffer but does not return the buffer's ownership before returning the message itself, the TEN runtime will reject the message (e.g., `SendData` will fail).

### Borrowing API

Extensions can directly access resources within a message using borrowing APIs, allowing temporary use without transferring ownership.

### GiveBack API

The GiveBack APIs are used to return the ownership of resources back to the TEN framework, ensuring that resources are safely managed and freed when no longer needed.

## Command

TEN clients and TEN extensions can send commands to other TEN extensions.

If a command is used in a situation where the result isn't needed, it's fine. TEN runtime will automatically discard any unnecessary results.

### Result

A result is the response to a command. For example, when TEN extension A sends command X to TEN extension B, after processing the command, TEN extension B will send a result back to TEN extension A. The result may include any meaningful details, allowing TEN extension A to gain more information about the execution outcome.

A typical result's JSON representation might look like this:

```json
{
  "ten": {
    "status_code": 400
  }
}
```


# Preparation (/docs/ten_framework/0.11.60/preparation)

The TEN framework supports Windows, Linux, and Mac as development environments, but it is recommended to use Linux or Mac.

## Development Environment

### Linux

Using Ubuntu as an example, you must first install the following packages:

* gcc: One of the C/C++ compilers supported by the TEN framework.
* clang: Another C/C++ compiler supported by the TEN framework.
* clang-format: A code formatting tool used by the TEN framework during runtime.
* clang-tidy: A static code analyzer.
* cmake: The build system. The version must be greater than 3.13.
* Python3: Required for the TEN framework Python binding.
* pytest: Used for the TEN framework's integration testing.

### Mac

To set up your Mac development environment, you need the following:

* XCode
* CocoaPods
* Personal Account

Follow these instructions to prepare the environment:

```shell
brew install llvm googletest doxygen ninja clang-format
brew install include-what-you-use
```

Since the version of `cmake` installed via `brew` might be outdated, if so, you need to install `cmake` from the official website and set up a symbolic link.

```shell
ln -s /Applications/CMake.app/Contents/bin/cmake /usr/local/bin/cmake
```

If `cmake` cannot find `clang-tidy`, you can resolve the issue with the following command:

```shell
ln -sf /usr/local/opt/llvm/bin/clang-tidy /usr/local/bin/clang-tidy
```

To build various language bindings for the TEN framework, you can install the required languages using the following command:

```shell
brew install python golang
```

### Windows

If you need to develop the TEN framework on Microsoft Windows, you can manually install the following packages:

* Visual Studio Community Version (up to 2022)

  Be sure to select the clang-related tools and add the clang binary path to the `PATH` environment variable.

* cmake

* python

> ⚠️ **Note:**
> When installing `cmake`/`python` on Windows, the installation path should not contain spaces.

## ten\_gn

The TEN framework uses `ten_gn` as its build system. `ten_gn` is a build system based on Google GN. The source code for `ten_gn` is located in the `core/ten_gn` directory of the TEN framework repository. Add the `ten_gn` directory path to the system `PATH` in your development environment.

## Using Docker Containers

We provide pre-written Docker files that allow you to create a container with all the necessary packages required to build the TEN framework from source.

### Ubuntu 22.04

Navigate to `tools/docker_for_building/ubuntu/22.04` and run the following commands to create and enter the build environment.

```shell
docker-compose up -d
docker-compose run ten-building-ubuntu-2204
```


# Profiling (/docs/ten_framework/0.11.60/profiling)

## TEN Framework GO Project

Use `gperftools` to profile your program's CPU usage and heap memory usage. The following steps describe how to prepare the environment, use `gperftools` to profile your program, and analyze the performance data.

### Installing `gperftools`

To install `gperftools` on Ubuntu, run the following commands:

```shell
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

### Profiling CPU Usage with `gperftools`

To profile the CPU usage of your program, run the following commands:

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/.../cpu

exec /.../program
```

When the program exits, a CPU performance data file will be generated. The data will be stored in the `/.../cpu.0001.prof` file.

If you want to generate CPU performance data periodically, you can set the `CPUPROFILE_FREQUENCY` environment variable. For example, to generate CPU performance data every 30 seconds, run the following command:

```shell
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/.../cpu CPUPROFILE_FREQUENCY=30 exec /.../program
```

#### Analyzing CPU Performance Data

To analyze the CPU performance data, run the following commands:

```shell
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### Profiling Heap Memory Usage with `gperftools`

To profile your program's heap memory usage, run the following commands:

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

It is recommended to use the `HEAP_PROFILE_TIME_INTERVAL` environment variable to periodically generate heap memory performance data. All heap memory performance data will be stored in the `HEAPPROFILE` directory.

#### Analyzing Heap Memory Performance Data

Each generated heap memory performance data file will have a `.heap` extension. To analyze the heap memory performance data, you can convert the `.heap` files to a human-readable text format. For example, to convert a `.heap` file in `/.../heap` to a text format, run the following command:

```shell
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

The performance data will look like this:

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

You can also convert the `.heap` file to other formats like PDF or SVG. For example, to convert a `.heap` file in `/.../heap` to PDF format, run the following command:

```shell
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

The performance data in PDF format will be similar to the text output but visualized graphically.

Additionally, you can compare two sets of heap memory performance data. For example, to compare the heap memory performance data in `/.../heap.0001.heap` and `/.../heap.0002.heap`, run the following command:

```shell
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

The PDF will show the differences in heap memory usage between the two data sets.

If you are generating heap memory performance data periodically, we provide a script to analyze all the heap memory performance data in a directory and generate an Excel file showing the heap memory usage trends. For example, to analyze all heap memory performance data in `/.../heap`, run the following command:

```shell
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

The script will generate raw performance data with symbols in the `/.../raw` directory, human-readable text performance data in the `/.../text` directory, and an Excel file in `/.../heap.xlsx`.

You can plot a line chart to display the heap memory usage trends.

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### Profiling with `pprof`

We recommend using `pprof` to profile programs written in Go. The easiest way to profile a Go program is by using `pprof_app_go` as your TEN application. When activated, the application reads environment variables to decide whether to activate the `pprof` server and heap profiler.

```shell
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### Analyzing `pprof` Performance Data

To convert heap performance data to a text format, use the following command:

```shell
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

To analyze all heap performance data and generate an Excel file, run:

```shell
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

You can plot a line chart to display the heap memory usage trends:

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

This will give you a visual representation of Go heap performance data trends.


# Version System (/docs/ten_framework/0.11.60/version_system)

The TEN framework follows the semver specification.

## Version Information of TEN Package

The version information of a TEN package is recorded in the `version` field of its own `manifest.json` file. For every TEN package, this `version` field is mandatory.

```json
{
  "version": "1.0.0"
}
```

## Version Requirement of Dependency

> ⚠️ **Note:**
> If no version requirement is specified for a dependency, the latest version is implied.

```text
└── ten_packages/
    ├── extension/
    ├── extension_group/
    ├── protocol/
    └── system/
```

These packages are distinguished by the value of the `type` field in their `manifest.json`.

```json
{
  "type": "system" // "extension", "extension_group", "protocol"
}
```

In a TEN package, the version requirements for the dependent TEN packages are specified as follows.

```json
{
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.1"
    },
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "1.0.3"
    },
    {
      "type": "extension",
      "name": "default_http_extension_cpp",
      "version": "1.0.0"
    },
    {
      "type": "extension_group",
      "name": "default_extension_group",
      "version": "1.0.0"
    }
  ]
}
```

## Version Requirements Specification

A version requirements string can contain multiple version requirement strings, separated by commas. Each version requirement string supports the following operators:

* `>`, `>=`, `=`

  Manually specify a version range or an exact dependency version.

* `~`

  Used to specify a minimum version with some flexibility for updates. If the major, minor, and patch versions are specified, or only the major and minor versions are specified, only patch-level changes are allowed. If only the major version is specified, both minor and patch-level changes are allowed.

* `^`

  The `^` operator can be omitted, meaning that if a version requirement string does not include any operator, it defaults to `^`. For example, the string `0.1.12` represents `^0.1.12`. Although `0.1.12` looks like a specific version, it actually specifies a version requirement that allows SemVer-compatible updates.

  Updates are allowed as long as they do not modify the leftmost non-zero number in the major, minor, or patch combination. In this case, if you upgrade the TEN package, it will update to version `0.1.13` (if it's the latest `0.1.z` version), but not to `0.2.0`. If the version string specified is `1.0`, it will update to `1.1` (if it's the latest `1.y` version), but not to `2.0`.

  > ⚠️ **Note:**
  > Versions `0.0.x` are considered incompatible with any other versions.


# Events (/docs/ten_agent_examples/api-reference/events)

import { Callout } from 'fumadocs-ui/components/callout';
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';

# TEN Agent Events

Events are the primary communication mechanism between the RTC server, MLLM extension, and your custom agent extension. TEN Agent uses a **Server→Client (S2C)** model for incoming events and **Client→Server (C2S)** primitives for outgoing commands.

## S2C Events (Server to Client)

These are events **sent from the server** that your agent extension receives and handles.

### User & Session Events

#### `UserJoinedEvent`

Triggered when a user joins the RTC session.

| Property      | Type     | Description                            |
| ------------- | -------- | -------------------------------------- |
| **user\_id**  | `string` | Unique identifier for the user joining |
| **timestamp** | `float`  | Event timestamp (Unix epoch)           |

**Example:**

```python
case UserJoinedEvent():
    self._rtc_user_count += 1
    await self._greeting_if_ready()
```

#### `UserLeftEvent`

Triggered when a user leaves the RTC session.

| Property      | Type     | Description                            |
| ------------- | -------- | -------------------------------------- |
| **user\_id**  | `string` | Unique identifier for the user leaving |
| **timestamp** | `float`  | Event timestamp                        |

**Example:**

```python
case UserLeftEvent():
    self._rtc_user_count -= 1
```

### Transcript Events

#### `InputTranscriptEvent`

Speech recognized by the ASR extension (user speech converted to text).

| Property       | Type     | Description                                        |
| -------------- | -------- | -------------------------------------------------- |
| **text**       | `string` | The recognized speech text                         |
| **final**      | `bool`   | Whether this is a final (complete) transcript      |
| **metadata**   | `dict`   | Additional context (session\_id, confidence, etc.) |
| **stream\_id** | `string` | Unique identifier for this speech segment          |

**Example:**

```python
case InputTranscriptEvent():
    self.current_metadata = {"session_id": event.metadata.get("session_id", "100")}
    self.session_ready = True
    await self._greeting_if_ready()
```

<Callout type="info">
  **Final vs Streaming**: The `final` flag indicates whether the transcript is complete or still being recognized. Non-final transcripts should not trigger processing.
</Callout>

#### `OutputTranscriptEvent`

Assistant speech generated by the MLLM (text-to-speech will convert this to audio).

| Property       | Type     | Description                               |
| -------------- | -------- | ----------------------------------------- |
| **text**       | `string` | The assistant's speech text               |
| **is\_final**  | `bool`   | Whether this completes the response       |
| **stream\_id** | `string` | Unique identifier for this speech segment |

**Example:**

```python
case OutputTranscriptEvent():
    await self._send_transcript("assistant", event.text, event.is_final, event.stream_id)
```

### Tool & Function Events

#### `ToolRegisterEvent`

A tool/function becomes available for the agent to call.

| Property     | Type   | Description                         |
| ------------ | ------ | ----------------------------------- |
| **tool**     | `str`  | Tool/function identifier            |
| **source**   | `str`  | Extension that registered this tool |
| **metadata** | `dict` | Tool schema and parameters          |

**Example:**

```python
case ToolRegisterEvent():
    await self.agent.register_tool(event.tool, event.source)
```

#### `FunctionCallEvent`

The MLLM is requesting a tool/function be called (function calling in action).

| Property           | Type     | Description                                      |
| ------------------ | -------- | ------------------------------------------------ |
| **call\_id**       | `string` | Unique identifier for this function call request |
| **function\_name** | `string` | Name of the function to call                     |
| **arguments**      | `dict`   | Function parameters as key-value pairs           |

**Example:**

```python
case FunctionCallEvent():
    await self.agent.call_tool(event.call_id, event.function_name, event.arguments)
```

<Callout type="warning">
  After processing a `FunctionCallEvent`, you must send the result back via the C2S `SendFunctionCallOutput` primitive to continue the conversation.
</Callout>

### Interrupt Event

#### `ServerInterruptEvent`

The MLLM detected the start of user speech and sent an interrupt signal.

| Property   | Type     | Description                                                 |
| ---------- | -------- | ----------------------------------------------------------- |
| **reason** | `string` | Why the interrupt occurred (e.g., "user\_speech\_detected") |

**Example:**

```python
case ServerInterruptEvent():
    await self._interrupt()
```

## C2S Primitives (Client to Server)

These are commands your agent extension **sends to the MLLM server**.

### Set Message Context

Initialize or update the conversation context with message history.

```python
async def _set_context_messages(self, messages: list[MLLMClientMessageItem]):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_SET_MESSAGE_CONTEXT,
        "v2v",
        MLLMClientSetMessageContext(messages=messages).model_dump(),
    )
```

**Parameters:**

* **messages**: List of message items (system, user, assistant)

### Send Message Item

Add a single message to the conversation.

```python
async def _send_message_item(self, message: MLLMClientMessageItem):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_SEND_MESSAGE_ITEM,
        "v2v",
        MLLMClientSendMessageItem(item=message).model_dump(),
    )
```

**Parameters:**

* **message**: A single message (text, image, or audio)

### Trigger Response

Ask the MLLM to generate a response (thinking + speaking).

```python
async def _send_create_response(self):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_CREATE_RESPONSE,
        "v2v",
        MLLMClientCreateResponse().model_dump(),
    )
```

<Callout type="tip">
  Call this after setting context or sending a message to trigger the MLLM to think and speak.
</Callout>

### Send Function Call Output

Return the result of a tool/function call back to the MLLM.

```python
async def _send_function_call_output(self, result: str, call_id: str):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_FUNCTION_CALL_OUTPUT,
        "v2v",
        MLLMClientFunctionCallOutput(
            output=result,
            call_id=call_id,
        ).model_dump(),
    )
```

**Parameters:**

* **output**: The tool result as a string
* **call\_id**: The call\_id from the original `FunctionCallEvent`

## Event Flow Diagram

<Tabs items={["Realtime V2V", "Cascade"]}>
  <Tab>
    ```
    User Speech
        ↓
    RTC → ASR Extension
        ↓
    InputTranscriptEvent (S2C)
        ↓
    MainExtension.on_data()
        ↓
    _set_context_messages() (C2S)
        ↓
    _send_create_response() (C2S)
        ↓
    MLLM processes & generates response
        ↓
    OutputTranscriptEvent (S2C)
        ↓
    TTS Extension converts to audio
        ↓
    Audio streamed to user
    ```
  </Tab>

  <Tab>
    ```
    User Message (text/voice)
        ↓
    MainExtension processes input
        ↓
    Sends to MLLM (C2S)
        ↓
    MLLM responds with text
        ↓
    OutputTranscriptEvent (S2C)
        ↓
    TTS converts & streams audio
        ↓
    User hears response
    ```
  </Tab>
</Tabs>

## Common Patterns

### Greeting When Ready

```python
async def _greeting_if_ready(self):
    if self._rtc_user_count == 1 and self.config.greeting and self.session_ready:
        await self._send_message_item(
            MLLMClientMessageItem(
                role="user",
                content=self.config.greeting,
            )
        )
        await self._send_create_response()
```

### Handling Tool Results

```python
case FunctionCallEvent():
    try:
        result = await call_tool(event.function_name, event.arguments)
        await self._send_function_call_output(json.dumps(result), event.call_id)
    except Exception as e:
        await self._send_function_call_output(json.dumps({"error": str(e)}), event.call_id)
```

### Graceful Shutdown

```python
case UserLeftEvent():
    self._rtc_user_count -= 1
    if self._rtc_user_count == 0:
        await self._cleanup()
```

## Reference

For complete event definitions and types, refer to the source code in the TEN Framework:

* [TEN Agent Events](https://github.com/TEN-framework/TEN-Agent/tree/main/ai_agents/agents/examples)
* [MLLM Extension API](https://github.com/TEN-framework/ten-framework)


# Configuration Schemas (/docs/ten_agent_examples/api-reference/schemas)

import { Callout } from 'fumadocs-ui/components/callout';
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';

# Configuration Schemas

TEN Agent configurations are defined in two key files: `property.json` (extension runtime properties) and `manifest.json` (metadata and dependencies).

## property.json

The `property.json` file defines runtime configurable properties for an extension.

### Schema Structure

```json
{
  "extension_group": "ten_agent_extensions",
  "api_version": "0.1",
  "dependencies": [],
  "nodes": [
    {
      "type": "extension",
      "name": "main_control",
      "addon": "main_control.extension:MainControlExtension",
      "property": {
        "greeting": "Hello! I'm your TEN agent.",
        "max_history": 50
      }
    }
  ]
}
```

### Common Properties by Extension Type

#### Main Control Extension

| Property              | Type     | Default | Description                                     |
| --------------------- | -------- | ------- | ----------------------------------------------- |
| **greeting**          | `string` | Empty   | Initial greeting message when first user joins  |
| **max\_history**      | `int`    | 100     | Maximum conversation history messages to retain |
| **enable\_interrupt** | `bool`   | `true`  | Allow interrupting assistant with user speech   |
| **lang**              | `string` | "en"    | Language code (en, cn, etc.)                    |

**Example:**

```json
{
  "greeting": "Hi there! How can I help you today?",
  "max_history": 50,
  "enable_interrupt": true,
  "lang": "en"
}
```

#### LLM Extension (Gemini 2.0 Live)

| Property           | Type     | Default                | Description                             |
| ------------------ | -------- | ---------------------- | --------------------------------------- |
| **api\_key**       | `string` | -                      | Google Cloud API key                    |
| **model**          | `string` | "gemini-2.0-flash-exp" | Model identifier                        |
| **system\_prompt** | `string` | -                      | System-level instructions for the model |
| **temperature**    | `float`  | 0.7                    | Response randomness (0.0-1.0)           |
| **max\_tokens**    | `int`    | 1024                   | Maximum response length                 |

**Example:**

```json
{
  "api_key": "${GOOGLE_API_KEY}",
  "model": "gemini-2.0-flash-exp",
  "system_prompt": "You are a helpful assistant.",
  "temperature": 0.7,
  "max_tokens": 1024
}
```

<Callout type="warning">
  **Security**: Never commit API keys directly. Use environment variables (`${VAR_NAME}`) and store values in `.env` files or deployment secrets.
</Callout>

#### STT Extension (Speech-to-Text)

| Property        | Type     | Default  | Description                            |
| --------------- | -------- | -------- | -------------------------------------- |
| **provider**    | `string` | "google" | STT provider (google, deepgram, azure) |
| **api\_key**    | `string` | -        | Provider API key                       |
| **language**    | `string` | "en-US"  | Language code                          |
| **model**       | `string` | -        | Provider-specific model                |
| **chunk\_size** | `int`    | 1024     | Audio chunk size in bytes              |

**Example:**

```json
{
  "provider": "google",
  "api_key": "${GOOGLE_API_KEY}",
  "language": "en-US",
  "chunk_size": 1024
}
```

#### TTS Extension (Text-to-Speech)

| Property     | Type     | Default  | Description                              |
| ------------ | -------- | -------- | ---------------------------------------- |
| **provider** | `string` | "google" | TTS provider (google, elevenlabs, azure) |
| **api\_key** | `string` | -        | Provider API key                         |
| **voice**    | `string` | -        | Voice identifier (provider-specific)     |
| **language** | `string` | "en-US"  | Language code                            |
| **speed**    | `float`  | 1.0      | Speech speed (0.5-2.0)                   |

**Example:**

```json
{
  "provider": "elevenlabs",
  "api_key": "${ELEVENLABS_API_KEY}",
  "voice": "bella",
  "language": "en-US",
  "speed": 1.0
}
```

#### RTC Extension

| Property             | Type   | Default | Description                     |
| -------------------- | ------ | ------- | ------------------------------- |
| **port**             | `int`  | 8000    | Server port for RTC connections |
| **enable\_video**    | `bool` | `false` | Enable video streaming          |
| **enable\_audio**    | `bool` | `true`  | Enable audio streaming          |
| **max\_connections** | `int`  | 100     | Maximum concurrent connections  |

**Example:**

```json
{
  "port": 8000,
  "enable_video": false,
  "enable_audio": true,
  "max_connections": 100
}
```

## manifest.json

The `manifest.json` file defines metadata, version information, and dependencies.

### Schema Structure

```json
{
  "type": "extension",
  "name": "main_control",
  "version": "0.1.0",
  "api_version": "0.1",
  "dependencies": [
    {
      "type": "extension",
      "name": "mllm_in",
      "version": ">=0.1.0"
    }
  ],
  "api": {
    "input": [
      {
        "name": "on_start_request",
        "type": "json"
      }
    ],
    "output": [
      {
        "name": "mllm_in",
        "type": "json"
      }
    ]
  }
}
```

### Core Fields

| Field            | Type     | Description                             |
| ---------------- | -------- | --------------------------------------- |
| **type**         | `string` | Extension type ("extension" or "addon") |
| **name**         | `string` | Unique extension name                   |
| **version**      | `string` | Semantic version (major.minor.patch)    |
| **api\_version** | `string` | API compatibility version               |
| **description**  | `string` | Human-readable description              |
| **author**       | `string` | Extension author/maintainer             |
| **dependencies** | `array`  | Required dependencies                   |
| **api**          | `object` | Input/output specification              |

### Dependencies Section

```json
"dependencies": [
  {
    "type": "extension",
    "name": "extension_name",
    "version": ">=0.1.0",
    "required": true
  }
]
```

| Field        | Type     | Default | Description                               |
| ------------ | -------- | ------- | ----------------------------------------- |
| **type**     | `string` | -       | "extension" or "addon"                    |
| **name**     | `string` | -       | Dependency name                           |
| **version**  | `string` | -       | Version constraint (>=0.1.0, \~1.0, etc.) |
| **required** | `bool`   | `true`  | Whether dependency is mandatory           |

### API Specification

#### Input

```json
"input": [
  {
    "name": "on_data",
    "type": "json",
    "required": true
  }
]
```

#### Output

```json
"output": [
  {
    "name": "output_topic",
    "type": "json"
  }
]
```

## Configuration Examples

<Tabs items={["Realtime V2V", "Cascade", "Minimal"]}>
  <Tab>
    ### Realtime V2V Configuration

    ```json
    {
      "extension_group": "ten_agent_extensions",
      "api_version": "0.1",
      "nodes": [
        {
          "type": "extension",
          "name": "main",
          "addon": "main_extension:MainExtension",
          "property": {
            "greeting": "Hello! I'm a voice assistant.",
            "max_history": 50,
            "lang": "en"
          }
        },
        {
          "type": "extension",
          "name": "mllm_in",
          "addon": "mllm_in_v2v:MLLMV2VExtension",
          "property": {
            "api_key": "${GOOGLE_API_KEY}",
            "model": "gemini-2.0-flash-exp",
            "system_prompt": "You are a helpful voice assistant."
          }
        },
        {
          "type": "extension",
          "name": "rtc",
          "addon": "rtc_extension:RTCExtension",
          "property": {
            "port": 8000,
            "enable_audio": true
          }
        }
      ]
    }
    ```
  </Tab>

  <Tab>
    ### Cascade Configuration

    ```json
    {
      "extension_group": "ten_agent_extensions",
      "api_version": "0.1",
      "nodes": [
        {
          "type": "extension",
          "name": "main",
          "addon": "main_extension:MainExtension",
          "property": {
            "greeting": "Hi! What can I do for you?",
            "max_history": 100
          }
        },
        {
          "type": "extension",
          "name": "asr",
          "addon": "asr_extension:ASRExtension",
          "property": {
            "provider": "google",
            "api_key": "${GOOGLE_API_KEY}",
            "language": "en-US"
          }
        },
        {
          "type": "extension",
          "name": "llm",
          "addon": "llm_extension:LLMExtension",
          "property": {
            "api_key": "${OPENAI_API_KEY}",
            "model": "gpt-4",
            "temperature": 0.7
          }
        },
        {
          "type": "extension",
          "name": "tts",
          "addon": "tts_extension:TTSExtension",
          "property": {
            "provider": "elevenlabs",
            "api_key": "${ELEVENLABS_API_KEY}",
            "voice": "bella"
          }
        }
      ]
    }
    ```
  </Tab>

  <Tab>
    ### Minimal Configuration

    ```json
    {
      "extension_group": "ten_agent_extensions",
      "api_version": "0.1",
      "nodes": [
        {
          "type": "extension",
          "name": "main",
          "addon": "main_extension:MainExtension",
          "property": {}
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Environment Variables

<Callout type="tip">
  Use environment variables for sensitive data and environment-specific settings.
</Callout>

### Common Variables

```bash
# API Keys
export GOOGLE_API_KEY="your-google-api-key"
export OPENAI_API_KEY="your-openai-api-key"
export ELEVENLABS_API_KEY="your-elevenlabs-key"

# Configuration
export TEN_AGENT_LANG="en"
export TEN_AGENT_GREETING="Hello!"
export TEN_AGENT_PORT="8000"
```

### Loading Environment Variables

```python
import os
from dotenv import load_dotenv

load_dotenv()  # Load from .env file

api_key = os.getenv("GOOGLE_API_KEY")
```

## Validation Tips

<Callout type="warning">
  Always validate configuration before starting the agent.
</Callout>

1. **Check required fields** are present
2. **Validate API keys** are non-empty
3. **Verify version constraints** are compatible
4. **Test dependency connections** exist
5. **Log configuration on startup** for debugging

## References

* [TEN Framework Configuration Guide](https://github.com/TEN-framework/ten-framework)
* [Property File Format](https://github.com/TEN-framework/ten-framework)
* [Manifest Specification](https://github.com/TEN-framework/ten-framework)


# Run Demo (/docs/ten_agent_examples/demo)

This guide will help you to run the TEN-Agent demo. The demo is a web-based interface that allows you to interact with the TEN-Agent. The orchestration of demo site is fixed and cannot be changed.

## Run demo from pre-built Docker image

The easiest way to run the demo is to use the pre-built Docker image. The image contains the latest version of the TEN-Agent and the demo. The project docker compose file already contains the pre-built image, so you can start by following the steps in [getting started guide](./getting_started).

After you successfully setup, you will need to run an additional command to switch graph folder to demo folder,

```bash
task use AGENT=agents/examples/demo
```

Once finished, you can access the demo by opening the browser and navigating to `http://localhost:3002`.

## Run demo from source code

If you want to run the demo from the source code, start a new terminal, change to the folder where you cloned TEN-Agent project, follow below steps:

```bash
# Change to the demo directory
cd demo

# Install dependencies
pnpm install

# Start the demo
pnpm dev
```

Once the demo is started, you can access it by opening the browser and navigating to `http://localhost:3001`.

> **Note:** The demo depends on the golang web server (located in `server` directory). The web server will start when you run `task run` command in the container after you follow the steps in the [getting started guide](./getting_started).


# Deploy TEN Agent Example (/docs/ten_agent_examples/deploy_ten_agent/deploy_agent_service)

Once you have customized your agent (either by using the playground or editing property.json directly), you can deploy it by creating a release docker image for your service.

## Make your agent service into a docker image

The Dockerfile has been prepared in every example folder. You can build the docker image by running the following command:

```shell
cd ai_agents
docker build -f agents/examples/<example-name>/Dockerfile -t example-app .
```

For Apple Silicon, use the following command:

```shell
docker build -f agents/examples/<example-name>/Dockerfile -t example-app . --platform linux/amd64
```

> Note: The above docker build commands need to be executed outside of Docker container.

## Run the docker image

After building the docker image, you can run it by running the following command:

```shell
docker run --rm -it --env-file .env -p 8080:8080 -p 3000:3000 example-app
```

The agent service will be running on [http://localhost:8080](http://localhost:8080), and a frontend will be ready for access on [http://localhost:3000](http://localhost:3000).

## Test the agent service

The service exposes a list of apis that you can use to interact with the agent. The api reference can be found [https://github.com/TEN-framework/TEN-Agent/tree/main/server](https://github.com/TEN-framework/TEN-Agent/tree/main/server).

You can also test the agent service by accessing [http://localhost:3000](http://localhost:3000).


# 创建 TTS 扩展 (/docs/ten_agent_examples/extension_dev/TTS_Extension_Development_Guide)

# 创建 TTS Extension 完整指南

本教程将指导你从零开始创建一个生产级别的 TTS（Text-to-Speech） Extension，涵盖从项目创建、核心开发、测试验证到发布上线的完整流程。

## 什么是 TTS Extension

TTS Extension 是 TEN Framework 生态系统中的一个**标准扩展积木**（Standard Extension），专门用于实现文本转语音（Text-to-Speech）功能。

### 核心功能

TTS Extension 的主要职责包括：

1. **接收文字**: 从上游扩展持续接收需要转换成语音的文字（通常来自于大模型）
2. **实时合成**: 将文字实时转换成对应的音频数据流
3. **发送音频**: 将合成的音频数据传递给下游扩展进行后续处理

### 在对话流中的位置

作为标准积木，TTS Extension 在 TEN Agent 对话流中扮演着**文本到音频转换**的关键角色：

```
[上游积木]  ──文字流──>  [TTS Extension]  ──音频流──>  [下游积木]
```

**典型的上游积木**：

* **LLM Extension**: 生成对话回复文字
* **Translation Extension**: 翻译后的文字结果
* **Text Processing Extension**: 经过预处理的文字内容

**典型的下游积木**：

* **RTC Extension**: 将音频推送到 RTC 频道
* **Audio Playback Extension**: 本地播放音频
* **Audio Processing Extension**: 对音频进行后处理（如混音、音效等）

## 📚 实现模式

TTS的实现基本分为[http版](#http-模式)和[websocket版](#websocket-模式)两种（目前TTS厂商主流支持这两种）。有些厂商提供了SDK，但实际底层基本也是http或websocket，根据SDK的实际行为选择http模式或http模式。
http是比较基础的实现方法，建议初次开发TTS Extension的开发者优先实现[http版](#http-模式)。然后再尝试实现进阶的[websocket版](#websocket-模式)。

### 架构介绍

目前有`AsyncTTS2BaseExtension` 和`AsyncTTS2HttpExtension`两个基类供开发者继承，基于这两个基类可以更方便的实现TTS Extension。

### 实现方式基本区分

**路径1：直接继承AsyncTTS2BaseExtension（WebSocket模式）**

```
┌─────────────────────────────────────────────────────────────┐
│                    AsyncTTS2BaseExtension                    │
│  【通用基类】提供TTS Extension的基础设施                      │
│  - 消息队列管理                                               │
│  - 生命周期管理                                               │
│  - 音频数据发送                                               │
│  - 指标上报                                                   │
│  - 错误处理                                                   │
└─────────────────────────────────────────────────────────────┘
                              ↑ 继承（路径1）
┌─────────────────────────────────────────────────────────────┐
│        VendorTTSExtension (WebSocket/SDK模式子类)             │
│  【子类实现】实现厂商特定逻辑                                   │
│  - 完整的request_tts()实现                                     │
│  - WebSocket/SDK客户端实现                                     │
│  - 配置类实现                                                  │
│  - 供应商信息                                                  │
│  - 采样率配置                                                  │
└─────────────────────────────────────────────────────────────┘
```

**路径2：继承AsyncTTS2HttpExtension（HTTP模式）**

```
┌─────────────────────────────────────────────────────────────┐
│                    AsyncTTS2BaseExtension                    │
│  【通用基类】提供TTS Extension的基础设施                      │
│  - 消息队列管理                                               │
│  - 生命周期管理                                               │
│  - 音频数据发送                                               │
│  - 指标上报                                                   │
│  - 错误处理                                                   │
└─────────────────────────────────────────────────────────────┘
                              ↑ 继承
┌─────────────────────────────────────────────────────────────┐
│               AsyncTTS2HttpExtension (HTTP模式)              │
│  【模式基类】提供HTTP模式的完整实现                             │
│  - 配置加载和验证                                              │
│  - 客户端管理                                                  │
│  - 请求处理逻辑（完整的request_tts()实现）                     │
│  - TTFB计算和上报                                              │
│  - PCMWriter管理                                              │
└─────────────────────────────────────────────────────────────┘
                              ↑ 继承（路径2）
┌─────────────────────────────────────────────────────────────┐
│              VendorTTSExtension (HTTP模式子类)                 │
│  【子类实现】实现厂商特定逻辑                                   │
│  - 配置类实现（create_config()）                               │
│  - 客户端实现（create_client()）                               │
│  - 供应商信息（vendor()）                                      │
│  - 采样率配置（synthesize_audio_sample_rate()）               │
└─────────────────────────────────────────────────────────────┘
```

详细的基类、子类职责可参考 [TTS 基类、子类各自功能](#tts-基类子类各自功能)

## 🚀 项目初始化

### 创建扩展项目

使用 TMan 的 TTS 专用模板快速创建项目骨架：

```bash title="Terminal"
# 进入扩展目录
cd ten-framework/ai_agents/agents/ten_packages/extension

# 创建TTS扩展项目
tman create extension my_tts_extension --template default_tts_python --template-data class_name_prefix=MyTts
```

创建成功后会显示：

```bash title="输出信息"
Package 'extension:my_tts_extension' created successfully in 'my_tts_extension' in 2 seconds.
```

### 安装项目依赖

#### 添加第三方库依赖

首先在 `requirements.txt` 中添加所需的第三方库依赖：

```text title="requirements.txt"
websockets~=14.0
pydantic
requests
httpx
aiofiles
```

#### 安装 TEN 依赖

进入创建的扩展目录并安装依赖：

```bash title="Terminal"
cd my_tts_extension
tman install --standalone
```

这会根据 `manifest.json` 中声明的依赖构建依赖树，并安装到 `.ten` 目录下。

## 项目结构概览

```
my_tts_extension/
├── .vscode/               # VS Code 调试配置
│   └── launch.json       # 调试启动配置
├── manifest.json          # 扩展元数据和依赖声明
├── property.json          # 默认配置参数，可参考[property.json内容](#property.json内容)
├── requirements.txt       # Python 依赖
├── config.py              # 配置管理类，可参考 [配置管理设计](#配置管理设计)
├── {vendor}_tts.py        # TTS客户端核心实现
├── extension.py           # 主要实现文件
└── tests/                 # 测试文件
    ├── bin/start          # 测试启动脚本
    ├── test_basic.py      # 单元测试
    └── configs/           # 测试配置
```

## HTTP 模式

### 模式特点

HTTP模式使用标准的HTTP流式请求，适合传统的REST API TTS服务。实现简单，维护成本低，但延迟相对较高。典型实现如Rime TTS。

### 核心架构

```
┌─────────────────┐    HTTP Request  ┌─────────────────┐
│   Extension     │─────────────────►│  TTS Provider   │
│                 │  POST请求         │                 │
│ - HTTP客户端     │                  │ - REST API      │
│ - 流式响应处理    │◄─────────────────│ - 流式音频响应    │
│ - 错误重试       │  HTTP Response    │ - 错误处理       │
└─────────────────┘                  └─────────────────┘
```

### 实现规范

HTTP模式基于`AsyncTTS2HttpExtension`基类实现，该基类已经提供了完整的请求处理逻辑、TTFB计算、音频数据处理等功能。开发者只需要实现客户端接口和配置类即可。

<div className="fd-steps">
  <div className="fd-step">
    #### HTTP客户端接口实现

    **客户端必须实现`AsyncTTS2HttpClient`接口：**

    ```python
    from ten_ai_base.tts2_http import AsyncTTS2HttpClient
    from ten_ai_base.struct import TTS2HttpResponseEventType
    from httpx import AsyncClient, Timeout, Limits

    class VendorTTSClient(AsyncTTS2HttpClient):
        """HTTP TTS Client, implements AsyncTTS2HttpClient interface"""
        
        def __init__(self, config: VendorTTSConfig, ten_env: AsyncTenEnv):

            # API endpoint configuration (using abstract method)
            self.endpoint = self._get_api_endpoint()
            
            # Request headers configuration (using abstract method)
            self.headers = self._create_headers()
            
            # HTTP client configuration
            self.client = AsyncClient(
                http2=True,  # Enable HTTP/2
                follow_redirects=True,
            )
        
        def _get_api_endpoint(self) -> str:
            """Get API endpoint - to be implemented by subclass"""
            raise NotImplementedError("Subclasses must implement _get_api_endpoint")
        
        def _create_headers(self) -> dict:
            """Create request headers - to be implemented by subclass"""
            raise NotImplementedError("Subclasses must implement _create_headers")
    ```

    **资源管理方法：**

    ```python
    async def clean(self) -> None:
        """Clean up resources - required by AsyncTTS2HttpClient interface"""
        if self.client:
            self.ten_env.log_debug("Cleaning HTTP client")
            # Note: Depending on actual implementation, may only need to set client to None
            # Or call await self.client.aclose()
            self.client = None

    async def cancel(self) -> None:
        """Cancel current request - required by AsyncTTS2HttpClient interface"""
        self.ten_env.log_debug("VendorTTS: cancel() called.")
        self._is_cancelled = True
    ```
  </div>

  <div className="fd-step">
    #### 请求处理

    **TTS请求处理（实现AsyncTTS2HttpClient接口）：**

    ```python
    from typing import AsyncIterator, Tuple
    from ten_ai_base.struct import TTS2HttpResponseEventType

    async def get(
        self, text: str, request_id: str
    ) -> AsyncIterator[Tuple[bytes | None, TTS2HttpResponseEventType]]:
        """Process single TTS request - required by AsyncTTS2HttpClient interface
        
        Note: Return type uses TTS2HttpResponseEventType enum, not integer
        """
        self._is_cancelled = False
        
        if not self.client:
            return
        
        try:
            # Build request data (using abstract method)
            request_data = self._create_request_data(text)
            
            # Send streaming request
            async with self.client.stream(
                "POST",
                self.endpoint,
                headers=self.headers,
                json=request_data,
            ) as response:
                # Handle response
                async for chunk in response.aiter_bytes(chunk_size=4096):
                    # Process response, prepare audio
                    pass
                
                # Request completed
                if not self._is_cancelled:
                    yield None, TTS2HttpResponseEventType.END
        
        except Exception as e:
            # Error handling
            pass

    def _create_request_data(self, text: str) -> dict:
        """Create request data - to be implemented by subclass"""
        raise NotImplementedError("Subclasses must implement _create_request_data")

    ```
  </div>

  <div className="fd-step">
    #### Extension实现

    **Extension必须继承`AsyncTTS2HttpExtension`：**

    ```python
    from ten_ai_base.tts2_http import AsyncTTS2HttpExtension, AsyncTTS2HttpConfig, AsyncTTS2HttpClient

    class VendorTTSExtension(AsyncTTS2HttpExtension):
        """TTS Extension implementation, inherits from AsyncTTS2HttpExtension base class"""
        
        async def create_config(self, config_json_str: str) -> AsyncTTS2HttpConfig:
            """Create config object - required by AsyncTTS2HttpExtension interface"""
            return VendorTTSConfig.model_validate_json(config_json_str)
        
        async def create_client(
            self, config: AsyncTTS2HttpConfig, ten_env: AsyncTenEnv
        ) -> AsyncTTS2HttpClient:
            """Create client object - required by AsyncTTS2HttpExtension interface"""
            return VendorTTSClient(config=config, ten_env=ten_env)
        
        def vendor(self) -> str:
            """Return vendor name"""
            return "vendor_name"
        
        def synthesize_audio_sample_rate(self) -> int:
            """Return audio sample rate"""
            return self.config.sample_rate if self.config else 16000
    ```

    **注意**：`AsyncTTS2HttpExtension`基类已经实现了完整的`request_tts()`方法，包括：

    * 请求处理逻辑
    * TTFB计算和上报
    * 音频数据处理
    * 错误处理
    * PCM文件写入

    开发者只需要实现上述抽象方法即可。
  </div>

  <div className="fd-step">
    #### 请求优化

    http请求有一些可以优化的参数，可以根据TTS厂商情况设置。可参考 [HTTP 请求优化](#http-请求优化)
  </div>
</div>

### 最佳实践

1. **使用基类**：继承`AsyncTTS2HttpExtension`基类，避免重复实现请求处理逻辑
2. **接口实现**：严格按照`AsyncTTS2HttpClient`接口实现客户端
3. **事件类型**：使用`TTS2HttpResponseEventType`枚举，确保类型安全
4. **抽象方法**：使用抽象方法（`_get_api_endpoint()`, `_create_headers()`等）分离厂商特定逻辑
5. **连接池管理**：使用HTTP连接池，减少连接建立开销
6. **超时控制**：设置合理的超时时间
7. **错误处理**：正确返回事件类型，让Extension基类统一处理错误
8. **元数据上报**：实现`get_extra_metadata()`方法，支持TTFB指标上报
9. **日志记录**：实现必要的日志记录，包括供应商错误、状态变化、请求响应等。详细要求请参考 [日志规范](#日志规范)

## websocket 模式

### 模式特点

WebSocket模式支持websocket通信，允许流式音频响应，实现最低延迟的TTS响应。典型实现如ElevenLabs TTS2。

### 核心架构

```
┌─────────────────┐    WebSocket     ┌─────────────────┐
│   Extension     │─────────────────►│  TTS Provider   │
│                 │  发送请求         │                 │
│ - 请求管理       │                  │ - 流式音频响应    │
│ - 响应处理       │◄─────────────────│ - 任务状态管理    │
│ - 连接复用       │  接收响应          │ - 错误处理       │
└─────────────────┘                  └─────────────────┘
```

### 实现规范

<div className="fd-steps">
  <div className="fd-step">
    #### 连接管理策略

    **连接生命周期管理：**

    ```python
    class VendorTTS2Synthesizer:
        def __init__(self, config, ten_env, error_callback, response_msgs):
            # Connection state management
            self._session_closing = False
            self._connect_exp_cnt = 0
            self.websocket_task = None
            self.channel_tasks = []
            self._session_started = False
            
            # Event synchronization mechanism
            self._connection_event = asyncio.Event()
            self._connection_success = False
            self._receive_ready_event = asyncio.Event()
            
            # Start WebSocket connection monitoring
            self.websocket_task = asyncio.create_task(self._process_websocket())
    ```

    **自动重连机制：**

    ```python
    async def _process_websocket(self) -> None:
        """Main WebSocket connection monitoring and reconnection logic"""
        try:
            # Use websockets.connect's automatic reconnection mechanism
            async for ws in websockets.connect(
                uri=self.uri,
                ...
            ):
                self.ws = ws
                try:
                    # Start send and receive tasks
                    self.channel_tasks = [
                        # For vendors supporting bidirectional streams, can use both send and receive loops.
                        # If vendor only supports one request at a time, don't use loop for send functionality
                        asyncio.create_task(self._send_loop(ws)),
                        asyncio.create_task(self._receive_loop(ws)),
                    ]
                    
                    # Wait for receive loop to be ready
                    await self._receive_ready_event.wait()
                    await self.start_connection()
                    
                    await self._await_channel_tasks()
                    
                except websockets.ConnectionClosed as e:
                    if not self._session_closing:
                        # Reset all event states
                        self._receive_ready_event.clear()
                        self._connection_event.clear()
                        self._connection_success = False
                        self._session_started = False
                        continue
        except Exception as e:
            self.ten_env.log_error(f"WebSocket connection error: {e}")
    ```
  </div>

  <div className="fd-step">
    #### 消息队列处理

    **文本输入队列：**

    ```python
    async def _send_loop(self, ws: ClientConnection) -> None:
        """Text sending loop"""
        try:
            # Send initialization message
            init_msg = ...
            await ws.send(json.dumps(init_msg))
            
            while not self._session_closing:
                # Send text
                pass
                    
        except asyncio.CancelledError:
            raise
        except Exception as e:
            self.ten_env.log_error(f"Exception in send_loop: {e}")
            raise
    ```

    **音频接收处理：**

    ```python
    async def _receive_loop(self, ws: ClientConnection) -> None:
        """Message receiving loop"""
        try:
            self._receive_ready_event.set()
            
            while self._session_closing == False:
                message = await ws.recv()
                # Parse response
                pass
                    
        except asyncio.CancelledError:
            raise
        except Exception as e:
            self.ten_env.log_error(f"Exception in receive_loop: {e}")
            raise
    ```
  </div>

  <div className="fd-step">
    #### 错误处理和重连机制

    **异常处理策略：**

    ```python
    def _process_ws_exception(self, exp) -> None | Exception:
        """Handle WebSocket connection exception and decide whether to reconnect"""
        self.ten_env.log_debug(f"Websocket internal error: {exp}")
        self._connect_exp_cnt += 1
        
        if self._connect_exp_cnt > 5:  # Maximum retry count
            self.ten_env.log_error(f"Max retries exceeded: {str(exp)}")
            return exp
        return None  # Continue reconnecting
    ```

    **错误回调处理：**

    ```python
    if self.error_callback:
        module_error = ModuleError(
            message=data["error"],
            module=ModuleType.TTS,
            code=error_code,
            vendor_info=error_info,
        )
        await self.error_callback("", module_error)
    ```
  </div>

  <div className="fd-step">
    #### 资源管理和清理

    **连接取消和清理：**

    ```python
    def cancel(self) -> None:
        """Cancel current connection, used for flush scenarios"""

    def _clear_queues(self) -> None:
        """Clear all queues to prevent processing old data, used after receiving flush"""

    ```
  </div>
</div>

### 最佳实践

1. **连接预热**：在初始化时建立WebSocket连接，减少首次请求延迟
2. **自动重连**：实现指数退避的重连策略
3. **资源清理**：及时清理取消的合成器，避免内存泄漏
4. **错误分类**：区分网络错误、认证错误和业务错误
5. **队列管理**：使用有界队列防止内存溢出
6. **超时处理**：设置合理的超时时间，避免长时间阻塞
7. **日志记录**：实现必要的日志记录，包括供应商错误、状态变化、请求响应等。详细要求请参考 [日志规范](#日志规范)

## Extension 结构及其他文件实现

Extension目录的结构可参考 [项目结构概览](#项目结构概览)

配置Config文件实现可参考 [配置管理设计](#配置管理设计)

必须实现的功能可参考 [必须实现的功能](#必须实现的功能-1)

不同模式的特殊要求可参考 [不同模式的特殊要求](#不同模式的特殊要求)

## 🧪 单元测试

### 测试文件结构

TTS Extension的测试目录应包含完整的测试套件，确保代码质量和功能正确性。

```
tests/
├── __init__.py                    # 测试包初始化
├── conftest.py                   # pytest配置和fixtures
├── test_basic.py                 # 基础功能测试
├── test_error_msg.py             # 错误处理测试，主要检测出现错误时error message的生成
├── test_params.py                # 参数配置测试，主要检测配置参数的检查功能
├── test_robustness.py            # 健壮性测试，主要检测对于异常情况的处理
├── test_metrics.py               # 指标测试
└── configs/                      # 测试配置文件
    ├── test_config.json          # 测试配置
    ├── invalid_config.json       # 无效配置测试
    └── mock_config.json          # Mock测试配置
```

### 测试最佳实践

1. **测试覆盖**：确保所有主要功能都有测试覆盖
2. **Mock使用**：合理使用Mock避免依赖外部服务
3. **异步测试**：正确处理异步测试和等待
4. **资源清理**：确保测试后正确清理资源
5. **错误场景**：测试各种错误和异常情况
6. **边界测试**：测试边界条件和极限情况
7. **并发测试**：测试并发请求处理能力
8. **配置测试**：测试各种配置参数组合

## 🔗 集成测试（Guarder）

### 环境变量配置

创建 `.env` 文件配置真实 API 密钥：

```bash title=".env"
# TTS Vendor Services API Key
VENDOR_TTS_API_KEY=your_api_key_here
# 例如：
ELEVENLABS_TTS_API_KEY=your_elevenlabs_api_key
```

### 测试配置

在 `tests/configs/` 目录下创建以下测试配置文件，Guarder 测试会使用这些配置：

<div className="fd-steps">
  <div className="fd-step">
    #### 基础音频设置配置

    **`property_basic_audio_setting1.json`** - 用于基础音频设置测试、边界输入测试、指标测试、无效文本处理测试：

    ```json title="tests/configs/property_basic_audio_setting1.json"
    {
      "dump": true,
      "dump_path": "./tests/keep_dump_output/",
      "params": {
        "output_format": "pcm_44100",
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```

    **`property_basic_audio_setting2.json`** - 用于基础音频设置测试（对比不同的采样率配置）：

    ```json title="tests/configs/property_basic_audio_setting2.json"
    {
      "dump": true,
      "dump_path": "./tests/keep_dump_output/",
      "params": {
        "output_format": "pcm_44100",
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```

    **注意**：根据你的 TTS 服务商，可能需要调整参数名称（如 `key`、`api_key`、`sample_rate`、`output_format` 等）。
  </div>

  <div className="fd-step">
    #### Dump 功能测试配置

    **`property_dump.json`** - 用于 dump 功能测试、flush 测试、按请求ID导出测试：

    ```json title="tests/configs/property_dump.json"
    {
      "dump": true,
      "dump_path": "./tests/dump_output/",
      "params": {
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```
  </div>

  <div className="fd-step">
    #### 错误处理测试配置

    **`property_invalid.json`** - 用于无效必需参数测试（测试无效的 API Key）：

    ```json title="tests/configs/property_invalid.json"
    {
      "params": {
        "key": "invalid"
      }
    }
    ```

    **`property_miss_required.json`** - 用于缺少必需参数测试（测试缺少 API Key）：

    ```json title="tests/configs/property_miss_required.json"
    {
      "params": {
        "key": ""
      }
    }
    ```
  </div>

  <div className="fd-step">
    #### 完整配置示例（可选）

    **`property_bechmarl.json`** - 完整配置示例，包含多个参数，可用于基准测试：

    ```json title="tests/configs/property_bechmarl.json"
    {
      "dump": true,
      "dump_path": "./tests/keep_dump_output/",
      "params": {
        "output_format": "pcm_44100",
        "language": "en",
        "voice_id": "your_voice_id",
        "model_id": "your_model_id",
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```
  </div>
</div>

### 配置文件说明

**关键配置项**：

* **`params.key`** 或 **`params.api_key`**：TTS 服务商的 API Key，使用 `${env:VENDOR_TTS_API_KEY}` 从环境变量读取
* **`dump`**：是否启用音频 dump 功能（`true` 或 `false`）
* **`dump_path`**：音频 dump 文件的保存路径
* **`params.sample_rate`** 或 **`params.output_format`**：音频采样率配置，根据服务商不同而不同

**环境变量支持**：

配置文件中可以使用 `${env:VARIABLE_NAME}` 格式从环境变量读取值，这样可以在不修改配置文件的情况下切换不同的 API Key。

### 运行 Guarder 测试

使用真实 API 密钥运行完整集成测试：

```bash title="Terminal"
cd ai_agents
task tts-guarder-test EXTENSION=your_extension_name CONFIG_DIR=tests/configs
```

### Guarder测试点说明

Guarder集成测试包含以下测试点，确保Extension符合TEN Framework的标准：

#### 1. **基础音频设置测试** (`test_basic_audio_setting.py`)

**测试目标**：验证Extension能够根据配置正确设置音频参数

**测试点**：

* 验证不同配置文件的采样率设置是否正确
* 验证Extension能够正确读取和响应不同的采样率配置
* 验证音频帧的采样率与实际配置一致

**预期结果**：

* 不同配置文件产生不同的采样率输出
* 所有音频帧的采样率保持一致
* 没有错误或异常

#### 2. **边界输入测试** (`test_corner_input.py`)

**测试目标**：验证Extension对边界输入的处理

**测试点**：

* 验证Extension能够处理边界输入
* 验证Extension能够正确上报metrics数据
* 验证metrics在`tts_audio_end`之前上报

**预期结果**：

* Extension能够处理边界输入并生成音频
* 能够接收到metrics数据
* metrics在`tts_audio_end`之前发送

#### 3. **PCM Dump功能测试** (`test_dump.py`)

**测试目标**：验证PCM音频文件导出功能

**测试点**：

* 验证启用dump功能时，能够生成PCM文件
* 验证PCM文件保存在指定路径
* 验证PCM文件不为空

**预期结果**：

* 当`dump=true`时，在`dump_path`目录下生成PCM文件
* PCM文件包含音频数据
* 文件命名符合规范

#### 4. **按请求ID导出测试** (`test_dump_each_request_id.py`)

**测试目标**：验证每个请求ID都能生成独立的dump文件

**测试点**：

* 验证多个请求能够生成多个独立的dump文件
* 验证每个请求ID对应一个dump文件
* 验证dump文件数量正确

**预期结果**：

* 发送N个请求，生成N个dump文件
* 每个dump文件对应一个request\_id
* 文件命名包含request\_id信息

#### 5. **Flush功能测试** (`test_flush.py`)

**测试目标**：验证flush请求的处理

**测试点**：

* 验证收到flush请求时，能够正确中断当前请求
* 验证flush时发送的`tts_audio_end`事件，`reason`应为`2`（INTERRUPTED）
* 验证flush完成后发送`tts_flush_end`事件
* 验证事件顺序：`tts_audio_end`在`tts_flush_end`之前

**预期结果**：

* 收到flush请求后，能够中断当前TTS合成
* 发送`tts_audio_end`事件，`reason=2`（INTERRUPTED）
* 发送`tts_flush_end`事件
* 事件顺序正确

#### 6. **指标上报测试** (`test_metrics.py`)

**测试目标**：验证指标数据的正确上报

**测试点**：

* 验证Extension能够上报metrics数据
* 验证metrics在`tts_audio_end`之前发送
* 验证metrics包含必需字段（module、vendor、metrics等）

**预期结果**：

* 能够接收到metrics数据
* metrics在`tts_audio_end`之前发送
* metrics数据格式正确，包含必需字段

#### 7. **无效必需参数测试** (`test_invalid_required_params.py`)

**测试目标**：验证对无效必需参数的错误处理

**测试点**：

* 验证当配置参数无效时（如API key格式错误），能够正确返回错误
* 验证错误代码为`FATAL_ERROR`（-1000）
* 验证错误消息清晰，便于用户定位问题

**预期结果**：

* Extension能够检测到无效的必需参数
* 返回错误，错误代码为`-1000`（FATAL\_ERROR）
* 错误消息包含问题描述

#### 8. **无效文本处理测试** (`test_invalid_text_handling.py`)

**测试目标**：验证Extension对无效文本的处理能力

**测试点**：

* 验证当文本无效时（如空字符串、纯标点、纯空格等），能够正确处理
* 验证返回错误代码为`NON_FATAL_ERROR`（1000）
* 验证错误包含`vendor_info`字段
* 验证在错误后，发送有效文本能够正常工作

**测试的无效文本类型**：

* 空字符串
* 纯空格、制表符、换行符
* 纯标点符号（中英文）
* 表情符号和特殊字符
* 数学公式、化学方程式
* 混合无效字符

**预期结果**：

* 无效文本返回错误，错误代码为`1000`（NON\_FATAL\_ERROR）
* 错误包含`vendor_info`，包含供应商名称、错误代码和消息
* 错误后，发送有效文本能够正常生成音频

#### 9. **缺少必需参数测试** (`test_miss_required_params.py`)

**测试目标**：验证对缺少必需参数的错误处理

**测试点**：

* 验证当缺少必需参数时（如缺少API key），能够正确返回错误
* 验证错误代码为`FATAL_ERROR`（-1000）
* 验证错误消息提示缺少的参数

**预期结果**：

* Extension能够检测到缺少的必需参数
* 返回错误，错误代码为`-1000`（FATAL\_ERROR）
* 错误消息明确提示缺少的参数

### 测试注意事项

1. **环境准备**：确保设置正确的环境变量（API key等）
2. **配置准备**：确保测试配置文件正确，包含有效的API凭证
3. **网络连接**：确保能够访问TTS供应商的API服务
4. **测试隔离**：每次测试运行前清理dump目录，确保测试结果准确
5. **错误分析**：如果测试失败，仔细查看日志输出，定位问题原因
6. **测试顺序**：某些测试可能有依赖关系，建议按顺序运行
7. **资源清理**：测试完成后清理临时文件和资源

## 🌐 端到端测试

完成开发后，可以使用 TMan Designer 快速替换 TEN Agent 对话图中的 TTS 扩展，验证在实际对话场景下的效果。

### 使用 TMan Designer 替换 TTS 扩展

```bash title="Terminal"
# 在 TEN Agent 项目目录下启动
cd /path/to/your/ten-agent-project
tman designer
```

TMan Designer 会打开可视化界面，你可以：

1. **选择 TTS 节点**: 点击现有的 TTS 扩展积木
2. **替换为你的扩展**: 选择 `my_tts_extension`
3. **配置参数**: 设置 API Key、语音ID等参数
4. **一键应用**: 完成替换并启动测试

替换完成后，通过真实对话验证扩展的音频质量、响应速度和稳定性。

## 提Pull Request前检查清单

在提交Pull Request之前，请确保已完成以下所有检查项：

<div className="fd-steps">
  <div className="fd-step">
    ### 功能实现 ✅

    **要求**：完成所有核心功能的实现

    **检查项**：

    * [ ] 实现`request_tts()`方法，能够正确处理文本输入并生成音频
    * [ ] 实现`vendor()`方法，返回正确的供应商名称
    * [ ] 实现`synthesize_audio_sample_rate()`方法，返回正确的采样率
    * [ ] 如果使用HTTP模式，实现`create_config()`和`create_client()`方法
    * [ ] 如果使用HTTP模式，实现配置类和客户端类的所有必需方法
    * [ ] 如果是websocket模式，要有自动重连机制
    * [ ] 正确实现错误处理，区分`FATAL_ERROR`和`NON_FATAL_ERROR`
    * [ ] 正确处理flush请求，实现`cancel_tts()`方法
    * [ ] 正确发送`tts_audio_start`和`tts_audio_end`事件
    * [ ] 正确计算和上报TTFB指标
    * [ ] 正确计算和上报音频时长
    * [ ] 正确发送metrics数据
    * [ ] 日志记录符合要求
  </div>

  <div className="fd-step">
    ### 提交的文件 ✅

    * [ ] **厂商交互功代码**：通常为xxx\_tts.py
    * [ ] **主控制代码**：通常为extension.py
    * [ ] **单元测试代码**：通常在tests文件夹下
    * [ ] **Guarder测试配置文件**：通常在tests/configs文件夹下
    * [ ] **最小启动参数文件**：property.json
    * [ ] **版本、依赖、接口定义文件**：manifest.json
  </div>

  <div className="fd-step">
    ### 单元测试（UT）✅

    **要求**：完成所有单元测试，确保代码质量

    **检查项**：

    * [ ] 所有单元测试通过
    * [ ] 测试所有主要功能路径
    * [ ] 测试错误处理逻辑
    * [ ] 测试边界条件
    * [ ] 测试参数验证逻辑
    * [ ] 测试配置加载和验证

    **运行命令**：

    ```bash
    # 方式1：使用task命令（推荐）
    # 在项目根目录（ten-framework/ai_agents）下运行
    task test-extension EXTENSION=agents/ten_packages/extension/your_extension_name

    # 方式2：手动运行
    cd agents/ten_packages/extension/your_extension_name
    tman -y install --standalone
    ./tests/bin/start

    # 方式3：运行所有Extension的测试
    task test-agent-extensions
    ```

    **注意**：

    * 确保在项目根目录（`ten-framework/ai_agents`）下运行task命令
    * 如果Extension没有安装依赖，task命令会自动安装
    * 测试配置文件应在`tests/configs/`目录下
  </div>

  <div className="fd-step">
    ### Ten Agent自测 ✅

    **要求**：在Ten Agent中完成自测，确保Extension能够正常工作

    **检查项**：

    * [ ] 在Ten Agent中成功加载Extension
    * [ ] 能够听到Agent声音
    * [ ] 能够正常多轮对话
    * [ ] 能够正常打断对话
  </div>

  <div className="fd-step">
    ### Guarder集成测试 ✅

    **要求**：通过所有Guarder集成测试，并将测试结果贴在PR评论中

    **测试位置**：`ten-framework/ai_agents/agents/integration_tests/tts_guarder`

    **运行命令**：

    ```bash
    # 方式1：使用task命令（推荐）
    # 在项目根目录（ten-framework/ai_agents）下运行
    task tts-guarder-test EXTENSION=your_extension_name CONFIG_DIR=tests/configs

    # 方式2：手动运行
    cd agents/integration_tests/tts_guarder
    ./scripts/install_deps_and_build.sh linux x64
    ./tests/bin/start --extension_name your_extension_name --config_dir agents/ten_packages/extension/your_extension_name/tests/configs

    # 运行单个测试文件
    ./tests/bin/start --extension_name your_extension_name --config_dir agents/ten_packages/extension/your_extension_name/tests/configs tests/test_basic_audio_setting.py
    ```

    **环境变量设置**：

    ```bash
    # 在项目根目录创建.env文件，或设置环境变量
    # 根据实际供应商设置API Key
    export VENDOR_TTS_API_KEY=your_api_key_here
    # 例如：
    export ELEVENLABS_TTS_API_KEY=your_elevenlabs_api_key
    # 或在.env文件中：
    # ELEVENLABS_TTS_API_KEY=your_elevenlabs_api_key
    ```

    **注意**：

    * 确保在项目根目录（`ten-framework/ai_agents`）下运行task命令
    * 测试配置文件应在Extension目录下的`tests/configs/`目录中
    * 如果使用task命令，会自动从`.env`文件读取环境变量
    * 确保Extension已正确安装依赖

    **测试结果要求**：

    * [ ] 所有Guarder测试通过
    * [ ] 在PR评论中粘贴guarder 测试结果
    * [ ] 如有测试失败，说明原因并提供解决方案
  </div>
</div>

### PR提交检查清单总结

在提交PR之前，请确认以下所有项目已完成：

* [ ] **功能实现**：所有核心功能已实现并通过自测
* [ ] **所有必备文件都已提交**：代码运行的必备文件
* [ ] **单元测试**：所有UT通过，覆盖率达标
* [ ] **Ten Agent自测**：在Ten Agent中验证功能正常
* [ ] **Guarder测试**：所有Guarder测试通过
* [ ] **测试结果**：在PR评论中粘贴Guarder测试结果
* [ ] **代码审查**：代码已自检，符合编码规范
* [ ] **提交信息**：PR描述清晰，包含变更说明和测试结果

## Spec

### HTTP 请求优化

**连接复用：**

```python
class VendorTTSClient:
    def __init__(self, config, ten_env):
        # Use connection pool
        self.client = AsyncClient(
            timeout=Timeout(
                connect=10.0,  # Connection timeout
                read=30.0,     # Read timeout
                write=10.0,    # Write timeout
                pool=5.0       # Connection pool timeout
            ),
            limits=Limits(
                max_connections=100,
                max_keepalive_connections=20,
                keepalive_expiry=600.0,
            ),
            http2=True,
            follow_redirects=True,
        )
```

**请求压缩：**

```python
def _get_headers(self) -> dict:
    """Get optimized request headers"""
    return {
        "Authorization": f"Bearer {self.api_key}",
        "Content-Type": "application/json",
        "Accept": "audio/pcm",
        "Accept-Encoding": "gzip, deflate",  # Enable compression
        "User-Agent": "TEN-Framework-TTS/1.0",
    }
```

### TTS 基类、子类各自功能

#### 基类 `AsyncTTS2BaseExtension` 核心功能

TTS Extension基于`AsyncTTS2BaseExtension`基类构建，该基类继承自`AsyncExtension`，提供了TTS Extension的完整基础设施。

```python
class AsyncTTS2BaseExtension(AsyncExtension, ABC):
    """TTS Extension基类"""
    
    # 抽象方法 - 子类必须实现
    @abstractmethod
    async def request_tts(self, t: TTSTextInput) -> None:
        """处理TTS请求 - 子类必须实现
        - 从队列中接收文本输入
        - 调用TTS服务生成音频
        - 使用send_tts_audio_data()发送音频数据
        """
        
    @abstractmethod
    def vendor(self) -> str:
        """返回供应商名称 - 子类必须实现
        - 用于指标上报和错误追踪
        """
        
    @abstractmethod
    def synthesize_audio_sample_rate(self) -> int:
        """返回音频采样率 - 子类必须实现
        - 用于音频帧格式化和时长计算
        """
    
    # 可选重写方法
    async def cancel_tts(self) -> None:
        """取消TTS请求 - 子类可选重写
        - 收到flush请求时调用
        - 用于实现TTS特定的取消逻辑
        - 应该快速执行，避免阻塞主线程
        """
    
    # 已实现的完整功能
    
    # 生命周期管理
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        """Extension初始化 - 基类已实现
        - 初始化消息队列
        - 初始化指标计数器
        """
        
    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        """Extension启动 - 基类已实现
        - 启动消息队列处理任务
        """
        
    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        """Extension停止 - 基类已实现
        - 发送最后一批指标
        - 清空消息队列
        - 取消处理任务
        """
        
    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        """Extension销毁 - 基类已实现"""
    
    # 消息队列处理
    async def on_data(self, ten_env: AsyncTenEnv, data: Data) -> None:
        """处理接收到的数据 - 基类已实现
        - 处理tts_text_input：放入消息队列
        - 处理tts_flush：清空队列并取消当前任务
        - 使用锁机制防止并发问题
        """
        
    async def _process_input_queue(self, ten_env: AsyncTenEnv) -> None:
        """异步处理队列中的消息 - 基类已实现
        - 从队列中逐个取出消息
        - 调用request_tts()处理每个请求
        """
        
    async def _flush_input_items(self) -> None:
        """清空消息队列 - 基类已实现
        - 清空输入队列
        - 取消当前处理任务
        - 调用cancel_tts()执行TTS特定取消逻辑
        """
    
    # 音频发送方法
    async def send_tts_audio_data(self, audio_data: bytes, timestamp: int = 0) -> None:
        """发送音频数据 - 基类已实现
        - 自动处理不完整的音频帧（leftover_bytes）
        - 格式化音频帧（采样率、声道数、采样宽度）
        - 发送到下游Extension
        """
        
    async def send_tts_audio_start(self, request_id: str, turn_id: int = -1, 
                                   extra_metadata: dict | None = None) -> None:
        """发送音频开始事件 - 基类已实现
        - 通知下游Extension音频开始
        - 支持元数据传递
        """
        
    async def send_tts_audio_end(self, request_id: str, request_event_interval_ms: int, 
                                  request_total_audio_duration_ms: int, turn_id: int = -1, 
                                  reason: TTSAudioEndReason = TTSAudioEndReason.REQUEST_END,
                                  extra_metadata: dict | None = None) -> None:
        """发送音频结束事件 - 基类已实现
        - 通知下游Extension音频结束
        - 包含事件间隔时长和音频总时长
        - 支持不同的结束原因（REQUEST_END, INTERRUPTED等）
        - 清理请求元数据
        """
    
    # 指标上报方法
    async def send_tts_ttfb_metrics(self, request_id: str, ttfb_ms: int, 
                                    turn_id: int = -1, extra_metadata: dict | None = None) -> None:
        """发送TTFB指标 - 基类已实现
        - TTFB（Time To First Byte）指标上报
        - 支持额外元数据
        """
        
    async def send_usage_metrics(self, request_id: str = "", 
                                 extra_metadata: dict | None = None) -> None:
        """发送使用量指标 - 基类已实现
        - 输入字符数、输出字符数
        - 接收的音频时长
        - 总使用量统计
        """
        
    async def send_metrics(self, metrics: ModuleMetrics, request_id: str = "") -> None:
        """发送通用指标 - 基类已实现
        - 发送ModuleMetrics对象
        """
    
    # 错误处理
    async def send_tts_error(self, request_id: str | None, error: ModuleError,
                             turn_id: int = -1, extra_metadata: dict | None = None) -> None:
        """发送错误信息 - 基类已实现
        - 标准化的错误报告格式
        - 包含供应商信息、错误代码、错误消息
        - 支持元数据传递
        """
    
    # 辅助方法
    def synthesize_audio_channels(self) -> int:
        """返回音频声道数 - 基类已实现（默认1）"""
        return 1
    
    def synthesize_audio_sample_width(self) -> int:
        """返回音频采样宽度（字节） - 基类已实现（默认2，16-bit）"""
        return 2
    
    def get_uuid(self) -> str:
        """生成唯一标识符 - 基类已实现"""
    
    def update_metadata(self, request_id: str | None, metadata: dict | None) -> dict:
        """更新元数据 - 基类已实现
        - 合并请求元数据和额外元数据
        """
    
    # 指标辅助方法
    def metrics_add_output_characters(self, characters: int) -> None:
        """添加输出字符数 - 基类已实现"""
    
    def metrics_add_input_characters(self, characters: int) -> None:
        """添加输入字符数 - 基类已实现"""
    
    def metrics_add_recv_audio_chunks(self, chunks: bytes) -> None:
        """添加接收的音频块 - 基类已实现"""
    
    async def metrics_calculate_duration(self) -> None:
        """计算音频时长 - 基类已实现"""
    
    def metrics_reset(self) -> None:
        """重置指标计数器 - 基类已实现"""
```

##### 基类提供的完整功能

`AsyncTTS2BaseExtension`基类已经完整实现了以下功能，开发者无需重复实现：

1. **生命周期管理**
   * `on_init()`: 初始化消息队列和指标计数器
   * `on_start()`: 启动消息队列处理任务
   * `on_stop()`: 发送最后一批指标、清空队列、取消任务
   * `on_deinit()`: 销毁处理

2. **异步消息队列处理**
   * `on_data()`: 处理上游发送的`tts_text_input`和`tts_flush`数据
   * `_process_input_queue()`: 异步处理队列中的消息，调用`request_tts()`
   * `_flush_input_items()`: 收到flush后清空队列并取消当前任务
   * 使用锁机制防止并发问题，确保flush时不会继续接收新请求

3. **音频数据管理**
   * `send_tts_audio_data()`: 自动处理不完整音频帧，格式化并发送
   * `send_tts_audio_start()`: 发送音频开始事件
   * `send_tts_audio_end()`: 发送音频结束事件，包含时长和原因

4. **指标上报**
   * `send_tts_ttfb_metrics()`: TTFB指标上报
   * `send_usage_metrics()`: 使用量指标上报（字符数、音频时长等）
   * `send_metrics()`: 通用指标上报
   * 自动计算音频时长和统计使用量

5. **错误处理**
   * `send_tts_error()`: 标准化的错误报告机制
   * 包含供应商信息、错误代码、错误消息

6. **辅助功能**
   * 音频参数获取（声道数、采样宽度）
   * 元数据管理
   * UUID生成
   * 指标统计和重置

##### 子类需要实现的方法

开发者只需要实现以下3个抽象方法（以及可选的`cancel_tts()`）：

1. **`async def request_tts(t: TTSTextInput) -> None`**
   * 处理TTS请求的核心逻辑
   * 从配置或服务获取音频数据
   * 使用`send_tts_audio_data()`发送音频
   * 使用`send_tts_audio_start()`和`send_tts_audio_end()`发送事件
   * 使用`send_tts_ttfb_metrics()`上报TTFB指标
   * 使用`send_tts_error()`报告错误

2. **`def vendor() -> str`**
   * 返回供应商名称，用于指标和错误追踪

3. **`def synthesize_audio_sample_rate() -> int`**
   * 返回音频采样率，用于音频帧格式化

4. **`async def cancel_tts() -> None`**（可选）
   * 实现TTS特定的取消逻辑
   * 例如：关闭连接、停止请求等
   * 应该快速执行，避免阻塞

#### 基类 `AsyncTTS2HttpExtension` 核心功能

HTTP模式基于`AsyncTTS2HttpExtension`基类实现，该基类继承自`AsyncTTS2BaseExtension`，并提供了HTTP模式的完整实现。

```python
class AsyncTTS2HttpExtension(AsyncTTS2BaseExtension):
    """HTTP模式TTS Extension基类"""
    
    # 抽象方法 - 子类必须实现
    @abstractmethod
    async def create_config(self, config_json_str: str) -> AsyncTTS2HttpConfig:
        """从JSON字符串创建配置对象 - 子类必须实现"""
        
    @abstractmethod
    async def create_client(self, config: AsyncTTS2HttpConfig, ten_env: AsyncTenEnv) -> AsyncTTS2HttpClient:
        """创建客户端对象 - 子类必须实现"""
    
    @abstractmethod
    def vendor(self) -> str:
        """返回供应商名称 - 子类必须实现（继承自基类）"""
        
    @abstractmethod
    def synthesize_audio_sample_rate(self) -> int:
        """返回音频采样率 - 子类必须实现（继承自基类）"""

    async def request_tts(self, t: TTSTextInput) -> None:
        """处理TTS请求 - 基类已完整实现
        包含完整的请求处理逻辑：
        - 请求状态管理（request_id、turn_id等）
        - 客户端自动重连机制
        - PCMWriter管理（创建、清理）
        - 调用client.get()获取音频流
        - 音频数据处理和发送
        - TTFB计算和上报（首次音频块时）
        - 音频时长计算
        - 音频开始/结束事件发送
        - 错误处理和上报
        """
        
    def _calculate_audio_duration_ms(self) -> int:
        """计算音频时长（毫秒）- 基类已实现
        根据音频字节数、采样率、声道数计算
        """
```

##### 基类提供的完整功能

`AsyncTTS2HttpExtension`基类已经完整实现了以下功能，开发者无需重复实现：

1. **生命周期管理**
   * `on_init()`: 配置加载、验证和客户端创建
   * `on_stop()`: 资源清理
   * `on_deinit()`: 销毁处理

2. **请求处理逻辑**（在`request_tts()`中实现）
   * 请求状态跟踪（request\_id、turn\_id、first\_chunk等）
   * 客户端自动重连（client为None时自动重新创建）
   * PCMWriter生命周期管理（创建、清理）
   * 调用客户端获取音频流
   * 音频数据流式处理
   * TTFB指标计算和上报（首次音频块时）
   * 音频时长计算
   * 音频开始/结束事件发送
   * 错误处理和上报

3. **资源管理**
   * PCMWriter自动管理（按request\_id）
   * 客户端资源清理
   * 状态重置

##### 子类需要实现的方法

开发者只需要实现以下4个抽象方法：

1. **`create_config(config_json_str: str) -> AsyncTTS2HttpConfig`**
   * 从JSON字符串创建配置对象

2. **`create_client(config: AsyncTTS2HttpConfig, ten_env: AsyncTenEnv) -> AsyncTTS2HttpClient`**
   * 创建客户端对象

3. **`vendor() -> str`**
   * 返回供应商名称

4. **`synthesize_audio_sample_rate() -> int`**
   * 返回音频采样率

##### 配置接口 `AsyncTTS2HttpConfig`

配置类必须继承`AsyncTTS2HttpConfig`并实现以下抽象方法：

```python
class AsyncTTS2HttpConfig(BaseModel):
    """HTTP模式配置基类"""
    
    dump: bool = False
    dump_path: str = "/tmp"
    
    @abstractmethod
    def update_params(self) -> None:
        """更新配置参数 - 子类必须实现
        - 从params字典中提取配置项
        - 处理参数映射和转换
        - 删除黑名单参数
        """
        
    @abstractmethod
    def to_str(self, sensitive_handling: bool = True) -> str:
        """转换为字符串 - 子类必须实现
        - 支持敏感信息加密
        - 用于日志记录
        """
        
    @abstractmethod
    def validate(self) -> None:
        """验证配置 - 子类必须实现
        - 检查必需参数
        - 验证参数范围和格式
        """
```

##### 客户端接口 `AsyncTTS2HttpClient`

客户端类必须实现`AsyncTTS2HttpClient`接口：

```python
class AsyncTTS2HttpClient:
    """HTTP模式客户端接口"""
    
    @abstractmethod
    async def clean(self) -> None:
        """清理资源 - 子类必须实现
        - 清理HTTP客户端连接
        - 释放相关资源
        """
        
    @abstractmethod
    async def cancel(self) -> None:
        """取消当前请求 - 子类必须实现
        - 设置取消标志
        - 中断正在进行的请求
        """
        
    @abstractmethod
    async def get(
        self, text: str, request_id: str
    ) -> AsyncIterator[Tuple[bytes | None, TTS2HttpResponseEventType]]:
        """获取音频流 - 子类必须实现
        - 发送HTTP POST请求
        - 处理流式响应
        - 返回音频块和事件类型
        - 使用TTS2HttpResponseEventType枚举
        """
        
    @abstractmethod
    def get_extra_metadata(self) -> dict[str, Any]:
        """获取额外元数据 - 子类必须实现
        - 返回除透传metadata之外额外需要传递的内容
        - 用于TTFB指标上报
        - 如voice_id、model_id等
        """
```

#### 基类与子类职责

##### 详细说明

**1. AsyncTTS2BaseExtension（通用基类）**

**职责**：提供TTS Extension的基础设施，所有TTS Extension共享的功能。

**负责**：

* ✅ 消息队列的完整管理（接收、处理、flush）
* ✅ Extension生命周期管理（init、start、stop、deinit）
* ✅ 音频数据发送和格式化
* ✅ 指标上报（TTFB、使用量等）
* ✅ 错误处理和上报
* ✅ 元数据管理

**不负责**：

* ❌ 具体的TTS服务调用
* ❌ 配置加载（由模式基类或子类负责）
* ❌ 客户端管理（由模式基类负责）
* ❌ 厂商特定的逻辑

**2. AsyncTTS2HttpExtension（HTTP模式基类）**

**职责**：提供HTTP模式的完整实现，所有HTTP模式TTS Extension共享的逻辑。

**负责**：

* ✅ 配置加载和验证（调用子类的`create_config()`）
* ✅ 客户端创建和管理（调用子类的`create_client()`）
* ✅ HTTP请求处理逻辑（完整的`request_tts()`实现）
* ✅ TTFB计算和上报
* ✅ **PCMWriter生命周期管理（如启用dump功能）**
  * **初始化**：基类在`request_tts()`中自动检测新请求ID时初始化
  * **写入**：基类在处理音频流时自动写入
  * **销毁/刷新**：基类在请求完成时自动刷新
* ✅ 请求状态跟踪（request\_id、turn\_id等）
* ✅ 音频流处理（调用子类的`client.get()`）
* ✅ 错误处理和事件发送

**不负责**：

* ❌ 厂商特定的配置结构（由子类配置类实现）
* ❌ 厂商特定的HTTP通信逻辑（由子类客户端类实现）
* ❌ 供应商信息（由子类实现）

**3. VendorTTSExtension（子类实现）**

**职责**：实现厂商特定的逻辑，与具体TTS服务提供商交互。

根据继承路径的不同，子类的职责也不同：

**路径1：直接继承AsyncTTS2BaseExtension（WebSocket）**

**负责**：

* ✅ **完整的`request_tts()`实现**
  * 请求状态管理（request\_id、turn\_id等）
  * WebSocket连接管理或SDK调用
  * 音频流处理和发送
  * TTFB计算和上报
  * 音频开始/结束事件发送
  * 错误处理和上报
  * **PCMWriter管理（如启用dump功能）**
    * **初始化**：在`request_tts()`中检测到新的`request_id`时，创建`PCMWriter`实例并保存到字典中（按`request_id`索引），同时清理旧的PCMWriter
    * **写入**：每次收到音频数据块时，在调用`send_tts_audio_data()`之前，使用`PCMWriter.write()`写入音频数据
    * **销毁/刷新**：请求完成时（包括正常完成和中断），在发送`audio_end`事件之前，使用`PCMWriter.flush()`刷新缓冲区
* ✅ **客户端类实现**
  * WebSocket连接管理
  * 或SDK客户端封装
  * 资源清理
  * 请求取消
* ✅ **配置类实现**
  * 参数提取和映射
  * 日志输出格式
  * 参数验证
* ✅ **Extension类实现**
  * `vendor()`: 返回供应商名称
  * `synthesize_audio_sample_rate()`: 返回采样率
  * `cancel_tts()`: 可选实现取消逻辑

**不负责**：

* ❌ 消息队列管理（基类已实现）
* ❌ 音频数据发送（基类已实现）
* ❌ 指标上报（基类已实现）
* ❌ 错误处理格式（基类已实现）

**路径2：继承AsyncTTS2HttpExtension（HTTP模式）**

**负责**：

* ✅ **配置类实现（`VendorTTSConfig`）**
  * `update_params()`: 参数提取和映射
  * `to_str()`: 日志输出格式
  * `validate()`: 参数验证
* ✅ **客户端类实现（`VendorTTSClient`）**
  * `get()`: HTTP请求发送和响应处理
  * `clean()`: 资源清理
  * `cancel()`: 请求取消
  * `get_extra_metadata()`: 元数据获取
* ✅ **Extension类实现**
  * `create_config()`: 创建配置对象
  * `create_client()`: 创建客户端对象
  * `vendor()`: 返回供应商名称
  * `synthesize_audio_sample_rate()`: 返回采样率

**不负责**：

* ❌ 消息队列管理（基类已实现）
* ❌ 音频数据发送（基类已实现）
* ❌ 指标上报（基类已实现）
* ❌ 错误处理格式（基类已实现）
* ❌ **请求处理流程（HTTP模式基类已实现）**
* ❌ **PCMWriter管理（HTTP模式基类已实现）**

**注意**：两种路径都需要在配置类中支持`dump`和`dump_path`参数来启用PCM功能。

### TTS Extension 接口

#### 输入输出数据格式

TTS 标准接口（`tts-interface.json`）中除了属性配置外，还定义了输入和输出的数据格式规范：

**输入数据**：

* **文本输入** (`tts_text_input`): 从上游接收的文本数据流
* **刷新请求** (`tts_flush`): 取消当前请求并清空队列

**输出数据**：

* **文本结果** (`tts_text_result`): 包含时间戳的文本结果（当`enable_words=true`时）
* **音频开始** (`tts_audio_start`): 音频开始事件
* **音频结束** (`tts_audio_end`): 音频结束事件，包含时长统计
* **刷新完成** (`tts_flush_end`): 刷新处理完成的通知
* **错误信息** (`error`): 发生错误时的错误详情
* **性能指标** (`metrics`): TTFB、音频时长等性能数据
* **PCM音频帧** (`pcm_frame`): 向下游发送的音频数据流

详细的数据结构定义和字段说明请参考 `tts-interface.json` 文件。

#### 上游输入接口（data\_in）

<div className="fd-steps">
  <div className="fd-step">
    ##### tts\_text\_input - TTS文本输入

    **用途**：接收来自上游Extension（通常是LLM Extension）的文本输入，用于生成语音。

    **必需字段**：

    * `request_id` (string): 请求唯一标识符，用于追踪和关联请求
    * `text` (string): 需要转换为语音的文本内容

    **可选字段**：

    * `text_input_end` (bool): 标识当前文本输入是否结束
      * `true`: 表示当前轮次文本输入已完成。
      * `false`或不设置: 表示还有后续文本输入
    * `metadata` (object): 元数据信息
      * `session_id` (string): 会话ID，用于关联同一会话的多个请求
      * `turn_id` (int64): 轮次ID，用于标识对话轮次
      * 其他需要TTS Extension 透传给下游积木的内容

    **处理说明**：

    * Extension需要将接收到的文本放入请求队列
    * 支持流式文本输入，可以接收多个`tts_text_input`消息
    * 当`text_input_end=true`时，开始处理完整的文本输入进行TTS合成
  </div>

  <div className="fd-step">
    ##### tts\_flush - 刷新请求

    **用途**：取消当前正在处理的TTS请求，清空请求队列。

    **字段**：

    * `flush_id` (string): 刷新请求的唯一标识符
    * `metadata` (object): 元数据信息
      * `session_id` (string): 会话ID
      * 其他需要TTS Extension 在处理完flush之后透传给下游的内容

    **处理说明**：

    * 收到flush请求时，Extension必须：
      1. 清空输入队列中的所有待处理请求
      2. 取消当前正在进行的TTS合成请求
      3. 调用客户端的`cancel()`方法中断TTS服务调用
      4. 发送`tts_flush_end`响应，通知下游Extension刷新完成
  </div>
</div>

#### 下游输出接口（data\_out）

<div className="fd-steps">
  <div className="fd-step">
    ##### tts\_text\_result - TTS文本结果（字幕对齐用）

    **用途**：当`enable_words=true`时，输出包含时间戳的文本结果。

    **必需字段**：

    * `request_id` (string): 请求ID，与输入的`tts_text_input.request_id`对应
    * `text` (string): 合成音频对应的文本内容
    * `start_ms` (int64): 文本开始时间戳（毫秒）
    * `duration_ms` (int64): 文本持续时长（毫秒）
    * `words` (array): 含时间戳的文本数组
      * `word` (string): 词语内容
      * `start_ms` (int64): 词语开始时间戳（毫秒）
      * `duration_ms` (int64): 词语持续时长（毫秒）

    **可选字段**：

    * `text_result_end` (bool): 标识文本结果是否结束
    * `metadata` (object): 元数据信息
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他上游传入需要透传的内容

    **发送说明**：

    * 仅在`enable_words=true`时需要发送此消息
    * 如果供应商TTS服务支持词级别时间戳，需要在收到供应商响应时提取并发送
    * 如果不支持词级别时间戳，可以不发送此消息
  </div>

  <div className="fd-step">
    ##### tts\_flush\_end - 刷新结束

    **用途**：响应`tts_flush`请求，通知下游Extension刷新操作已完成。

    **字段**：

    * `flush_id` (string): 与输入的`tts_flush.flush_id`对应
    * `metadata` (object): 元数据信息, tts\_flush 带入的内容透传出去

    **发送说明**：

    * 在完成flush操作后必须发送，确保下游Extension知道刷新已完成
  </div>

  <div className="fd-step">
    ##### tts\_audio\_start - 音频开始事件

    **用途**：通知下游Extension音频数据开始发送。

    **字段**：

    * `request_id` (string): 请求ID，与输入的`tts_text_input.request_id`对应
    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他透传数据

    **发送说明**：

    * 在开始发送第一个音频数据块之前发送
    * 用于下游Extension进行音频播放准备
  </div>

  <div className="fd-step">
    ##### tts\_audio\_end - 音频结束事件

    **用途**：通知下游Extension音频数据发送完成。

    **必需字段**：

    * `request_id` (string): 请求ID，与输入的`tts_text_input.request_id`对应
    * `request_event_interval_ms` (int64): 请求事件间隔时长（毫秒）
      * 从接收第一个`tts_text_input`到发送第一个音频数据块的时间间隔
    * `request_total_audio_duration_ms` (int64): 请求总音频时长（毫秒）
      * 整个请求生成的音频总时长
    * `reason` (int64): 结束原因
      * `1`: 正常结束（REQUEST\_END）
      * `2`: 中断（INTERRUPTED）
      * 其他值根据实际需求定义

    **可选字段**：

    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID

    **发送说明**：

    * 在所有音频数据块发送完成后发送
    * 必须准确计算`request_event_interval_ms`和`request_total_audio_duration_ms`
    * 如果请求被中断（收到flush），`reason`应设置为中断状态
  </div>

  <div className="fd-step">
    ##### error - 错误信息

    **用途**：向上游或下游Extension报告错误信息。

    **必需字段**：

    * `module` (string): 模块名称，必须为`"tts"`
    * `code` (int64): 错误代码
      * `-1000`: 致命错误（FATAL\_ERROR）
      * `1000`: 非致命错误（NON\_FATAL\_ERROR）
    * `message` (string): 错误描述信息

    **可选字段**：

    * `id` (string): 错误唯一标识符
    * `vendor_info` (object): 供应商特定的错误信息
      * `vendor` (string): 供应商名称
      * `code` (string): 供应商错误代码
      * `message` (string): 供应商错误消息
    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他需要透传的参数

    **发送说明**：

    * 当TTS合成过程中出现错误时必须发送
    * `NON_FATAL_ERROR` 是指无法恢复的错误，通常是基本参数问题。
    * `FATAL_ERROR` 是指可以恢复的错误，通常表示当前请求有问题，但不影响后续请求
  </div>

  <div className="fd-step">
    ##### metrics - 指标数据

    **用途**：上报TTS Extension的性能和使用量指标。

    **必需字段**：

    * `module` (string): 模块名称，通常为`"tts"`
    * `vendor` (string): 供应商名称
    * `metrics` (object): 指标数据对象
      * 可以包含TTFB、音频时长、字符数、请求数等指标

    **可选字段**：

    * `id` (string): 指标唯一标识符
    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他需要透传的内容

    **发送说明**：

    * 用于监控和统计TTS Extension的使用情况
    * 通常在请求完成时上报
  </div>
</div>

#### 音频输出接口（audio\_frame\_out）

##### pcm\_frame - PCM音频帧

**用途**：向下游Extension发送PCM格式的音频数据。

**字段**：

* `metadata` (object): 元数据信息
  * `session_id` (string): 会话ID
  * `turn_id` (int64): 轮次ID

**音频格式要求**：

* 格式：PCM（脉冲编码调制）
* 采样率：由`synthesize_audio_sample_rate()`方法返回（常见值：16000、24000、44100、48000 Hz）
* 声道数：由`synthesize_audio_channels()`方法返回（默认1，单声道）
* 采样宽度：由`synthesize_audio_sample_width()`方法返回（默认2字节，16-bit）
* 字节序：小端序（Little-Endian）

**发送说明**：

* 音频数据以帧为单位流式发送
* 基类会自动处理不完整的音频帧（leftover\_bytes）
* 如果启用了`dump`功能，音频数据会同时保存到文件

#### 接口调用流程示例

##### 正常请求流程

```
上游Extension                     TTS Extension                    下游Extension                         TTS厂商服务器
     |                                  |                                 |                                 |
     |---- tts_text_input ------------->|                                 |                                 |
     |    (text="Hello")                |                                 |                                 |
     |                                  |-------------------------------调用TTS服务 ------------------------->|
     |                                  |                                 |                                 |
     |                                  |<------------------------------接收音频数据--------------------------|
     |                                  |                                 |                                 |
     |                                  |-----tts_ttfb(metrics)---------->|                                 |
     |                                  |                                 |                                 |
     |                                  |---- tts_audio_start  ---------->|                                 |
     |                                  |                                 |                                 |
     |                                  |---- pcm_frame ----------------->|                                 |
     |                                  |---- pcm_frame ----------------->|                                 |
     |                                  |---- pcm_frame ----------------->|                                 |
     |                                  |    ...                          |                                 |
     |                                  |---- tts_audio_end ------------->|                                 |
     |                                  |                                 |                                 |
     |                                  |---- input,output metrics ------>|                                 |
```

##### Flush请求流程

```
上游Extension                     TTS Extension                    下游Extension
     |                                  |                                 |
     |---- tts_flush ------------------>|                                 |
     |                                  | [内部处理]                       |
     |                                  |   1. 取消当前请求                 |
     |                                  |   2. 清空队列                    |
     |                                  |   3. 调用客户端cancel()           |
     |                                  |                                 |
     |                                  |---- tts_flush_end ------------->|
```

### manifest.json 内容

`manifest.json` 文件定义了 TTS Extension 的元数据、依赖关系、API 接口和属性声明。

#### 基本结构

```json title="manifest.json"
{
  "type": "extension",
  "name": "vendor_tts_python",
  "version": "0.1.0"
}
```

**关键字段说明**：

* `type`: 必须是 `"extension"`
* `name`: Extension 的唯一标识符，建议使用 `{vendor}_tts_python` 格式
* `version`: Extension 版本号，遵循语义化版本规范，每次更新代码都需要更新版本号，这样才能发不到TEN Extension Store

#### 依赖声明

```json
"dependencies": [
  {
    "type": "system",
    "name": "ten_runtime_python",
    "version": "0.11"
  },
  {
    "type": "system",
    "name": "ten_ai_base",
    "version": "0.7"
  }
]
```

**必需依赖**：

* `ten_runtime_python`: TEN Framework Python 运行时环境
* `ten_ai_base`: TEN AI Base 系统包，提供 TTS 基类和接口定义

#### 文件包含配置

```json
"package": {
  "include": [
    "manifest.json",
    "property.json",
    "BUILD.gn",
    "**.tent",
    "**.py",
    "README.md",
    "requirements.txt"
  ]
}
```

定义 Extension 打包时需要包含的文件，使用 glob 模式匹配。

#### API 接口配置

**1. Interface 继承**

```json
"api": {
  "interface": [
    {
      "import_uri": "../../system/ten_ai_base/api/tts-interface.json"
    }
  ]
}
```

**说明**：

* 必须继承 `ten_ai_base` 系统包下的标准 TTS 接口
* 该接口定义了所有 TTS Extension 必须遵循的标准属性：
  * `dump`: 布尔值，配置是否开启音频 dump
  * `dump_path`: 字符串，音频 dump 的存储路径

**2. Property 声明**

```json
"api": {
  "property": {
    "properties": {
      "dump": {
          "type": "bool"
      },
      "dump_path": {
          "type": "string"
      },
      "params": {
        "type": "object",
        "properties": {
             ...
        }
      }
    }
  }
}
```

**说明**：

* 在 `api.property.properties` 中声明 Extension 特有的配置属性
* `params` TTS 厂商使用的参数，罗列TTS工作的最小集参数，不需要全部罗列：

**Elvenlabs TTS完整示例**：

```json title="manifest.json"
{
  "type": "extension",
  "name": "vendor_tts_python",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "0.11"
    },
    {
      "type": "system",
      "name": "ten_ai_base",
      "version": "0.7"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "BUILD.gn",
      "**.tent",
      "**.py",
      "README.md",
      "requirements.txt"
    ]
  },
  "api": {
    "interface": [
      {
        "import_uri": "../../system/ten_ai_base/api/tts-interface.json"
      }
    ],
    "property": {
      "properties": {
        "params": {
          "type": "object",
          "properties": {
            "key": {
              "type": "string"
            },
            "model_id": {
              "type": "string"
            },
            "voice_id": {
              "type": "string"
            },
            "output_format": {
              "type": "string"
            }
          }
        },
        "dump": {
          "type": "bool"
        },
        "dump_path": {
          "type": "string"
        }
      }
    }
  }
}
```

### property.json内容

在 `property.json` 中提供可以使TTS工作的最小集默认配置：

```json title="property.json"
{
  "params": {
    "api_key": "your_tts_api_key_here",
    "voice_id": "default_voice",
    "model": "default_model",
    "sample_rate": "24000"
  },
  "extra_params": {
    "extra_key": "extra_value"
  },
  "dump": false,
  "dump_path": "/tmp/tts_audio_dump"
}
```

其中dump是指开启音频dump功能，TTS生成的所有音频都讲保存在文件里，同一个request id 的文本将按顺序保存在同一个文件。不同的request id文本保存在不同的文件。dump\_path是文件保存的位置。
params里存放的是可以透传给TTS厂商的参数。
具体内容和参数命名需要参考TTS厂商官网。
如果有其他一些定制化参数但又不是TTS厂商官方定义的，可以统一放在extra\_params里面。extra\_params的值跟params的值一样是json对象

### 配置管理设计

#### 设计配置类

创建灵活的配置类，支持必填参数和可选透传参数：

```python title="config.py"
from pydantic import BaseModel
from typing import Dict, Optional

class MyTTSConfig(BaseModel):
    # All TTS Vendor parameters are in params, including required and optional parameters
    params: Dict[str, Optional[str]] = {}

    #Not TTS Vendor parameters, but used in this TTS Extension
    extra_params: Dict[str, Optional[str]] = {}
    
    # Audio dump related configuration - standard implementation for all TTS extensions
    dump: bool = False
    dump_path: Optional[str] = None
```

#### 读取扩展配置

在 `on_init` 阶段读取和初始化配置：

```python title="extension.py"
from ten_ai_base.const import LOG_CATEGORY_KEY_POINT, LOG_CATEGORY_VENDOR
from ten_ai_base.message import ModuleError, ModuleErrorCode

@override
async def on_init(self, ten_env: AsyncTenEnv) -> None:
    await super().on_init(ten_env)
    
    # Read complete extension configuration
    config_json, _ = await ten_env.get_property_to_json("")
    
    try:
        # Deserialize configuration to config class instance
        self.config = MyTTSConfig.model_validate_json(config_json)
        
        # Print configuration information (sensitive information masked)
        ten_env.log_info(
            f"config: {self.config.to_json(sensitive_handling=True)}",
            category=LOG_CATEGORY_KEY_POINT,
        )
            
    except Exception as e:
        ten_env.log_error(
            f"invalid property: {e}",
            category=LOG_CATEGORY_KEY_POINT
        )
        # Use default configuration when configuration error occurs
        self.config = MyTTSConfig.model_validate_json("{}")
        # Send fatal error
        await self.send_tts_error(
            ModuleError(
                module=MODULE_NAME_TTS,
                code=ModuleErrorCode.FATAL_ERROR.value,
                message=str(e),
            ),
        )
```

#### 配置敏感信息脱敏

为配置类添加脱敏方法，保护敏感信息：

```python title="config.py"
from ten_ai_base.utils import encrypt

class MyTTSConfig(BaseModel):
    params: Dict[str, Optional[str]] = {}
    dump: bool = False
    dump_path: Optional[str] = None
    
    def to_json(self, sensitive_handling: bool = False) -> str:
        """
        Serialize configuration to JSON, support sensitive information masking
        
        Args:
            sensitive_handling: Whether to mask sensitive information
        """
        if not sensitive_handling:
            return self.model_dump_json()
        
        # Deep copy configuration object
        config = self.model_copy(deep=True)
        
        # Mask sensitive fields in params
        if config.params:
            encrypted_params = {}
            for key, value in config.params.items():
                # Encrypt fields containing sensitive words like 'key', 'token', 'secret'
                if (key in ['api_key', 'key', 'token', 'secret', 'password'] 
                    and isinstance(value, str) and value):
                    encrypted_params[key] = encrypt(value)
                else:
                    encrypted_params[key] = value
            config.params = encrypted_params
            
        return config.model_dump_json()
```

#### 配置功能最佳实践

1. **类型安全**：使用pydantic的类型验证，确保参数类型正确
2. **必需参数检查**：在`validate()`方法中检查必需参数
3. **参数范围验证**：验证采样率、声道数等参数的有效范围
4. **敏感信息保护**：使用`to_str()`方法的安全模式处理敏感信息
5. **默认值设置**：为所有最小集参数提供合理的默认值
6. **错误信息清晰**：提供清晰的错误信息，便于调试

### 必须实现的功能

**注意**：除了以下功能实现外，还需要实现必要的日志记录。详细要求请参考 [日志规范](#日志规范)。

<div className="fd-steps">
  <div className="fd-step">
    #### 初始化方法

    * **`__init__(config, ten_env, error_callback)`**：初始化客户端，接收配置对象、环境对象和错误回调函数
    * **`_initialize_client()`**：初始化具体的客户端连接（WebSocket连接、HTTP客户端、SDK客户端等）
  </div>

  <div className="fd-step">
    #### 核心请求处理

    **WebSocket模式：**

    * **`async def get(text, request_id) -> AsyncIterator[tuple[bytes | None, int, int | None]]`**：处理TTS请求，返回异步迭代器，每次yield音频数据块、事件类型和TTFB时间（首次音频块时）
      * 返回值：`(audio_data, event_type, ttfb_ms)`
      * 事件类型：`EVENT_TTS_RESPONSE`（音频数据）、`EVENT_TTS_REQUEST_END`（请求结束）、`EVENT_TTS_ERROR`（错误）、`EVENT_TTS_INVALID_KEY_ERROR`（认证错误）

    **HTTP模式：**

    * **`async def get(text, request_id) -> AsyncIterator[Tuple[bytes | None, TTS2HttpResponseEventType]]`**：处理TTS请求，返回异步迭代器，使用`TTS2HttpResponseEventType`枚举
      * 返回值：`(audio_data, event_type)`
      * 事件类型：`TTS2HttpResponseEventType.RESPONSE`、`TTS2HttpResponseEventType.END`、`TTS2HttpResponseEventType.ERROR`、`TTS2HttpResponseEventType.INVALID_KEY_ERROR`、`TTS2HttpResponseEventType.FLUSH`
  </div>

  <div className="fd-step">
    #### 抽象方法（用于解耦厂商特定逻辑）

    **WebSocket模式：**

    * **`_get_websocket_uri() -> str`**：获取WebSocket连接URI
    * **`_create_request_data(text) -> dict`**：创建请求数据格式
    * **`_parse_response(data) -> tuple`**：解析响应数据，返回音频数据和事件类型
    * **`_receive_responses() -> AsyncIterator`**：接收WebSocket响应流

    **HTTP模式（继承`AsyncTTS2HttpClient`时）：**

    * **`_get_api_endpoint() -> str`**：获取API端点URL
    * **`_create_headers() -> dict`**：创建HTTP请求头（包含认证信息）
    * **`_create_request_data(text) -> dict`**：创建HTTP请求体数据
    * **`_is_authentication_error(error_message) -> bool`**：判断是否为认证错误

    **SDK模式：**

    * **`_parse_credentials()`**：解析认证信息（证书文件、服务账号等）
    * **`_create_streaming_config()`**：创建流式配置对象
    * **`_create_request_generator(text)`**：创建请求生成器（第一个请求包含配置，后续发送文本）
    * **`_call_streaming_api(request_generator)`**：调用SDK的流式API
    * **`_extract_audio_content(response)`**：从SDK响应中提取音频数据
  </div>

  <div className="fd-step">
    #### 资源管理方法

    * **`async def stop()`**：停止客户端，关闭连接和清理资源
    * **`def cancel()`**：取消当前请求，设置取消标志，清理队列和关闭连接
    * **`async def clean()`**（HTTP模式）：清理HTTP客户端资源
    * **`async def reset()`**（WebSocket/SDK模式，可选）：重置客户端连接
  </div>

  <div className="fd-step">
    #### 辅助方法

    **HTTP模式：**

    * **`def get_extra_metadata() -> dict[str, Any]`**：返回供应商特定的元数据（如voice\_id、model\_id），用于TTFB指标上报

    **WebSocket/SDK模式（可选）：**

    * **`async def start()`**：启动客户端连接（WebSocket单向流模式中，连接已在`__init__`中启动）
    * **`_process_ws_exception(exception)`**：处理WebSocket异常
    * **`_await_connection_tasks()`**：等待连接任务完成
  </div>
</div>

### 日志规范

TTS Extension 必须实现以下日志，用于调试、监控和问题排查。所有日志应使用 `ten_env` 提供的日志方法，并指定正确的日志级别和分类。

#### 日志分类

TTS Extension 使用以下日志分类：

* **`LOG_CATEGORY_KEY_POINT`**：Extension 本身的关键事件，如配置加载、文本过滤等
* **`LOG_CATEGORY_VENDOR`**：与供应商相关的日志，如供应商错误、状态变化、请求响应等

#### 基类已实现的日志

以下日志已在基类 `AsyncTTS2BaseExtension` 中实现，子类**不需要**重复实现：

<div className="fd-steps">
  <div className="fd-step">
    ##### 文本输入相关日志

    **收到 TTS 文本输入** (`get_tts_text_input`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`get tts_text_input: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"get tts_text_input: {tts_text_input_object}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### Flush 相关日志

    **收到 flush** (`receive tts flush`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`receive tts flush`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        "receive tts flush ",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **发送 flush\_end** (`send tts_flush_end`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`send tts_flush_end`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        "send tts_flush_end",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### 错误信息日志

    **发送错误信息** (`tts_error`)

    **日志级别**：`error`

    **日志分类**：`key_point`

    **日志格式**：`tts_error: {}`

    **基类实现**：

    ```python
    self.ten_env.log_error(
        f"tts_error: msg: {error_msg}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### 音频事件日志

    **音频开始** (`tts_audio_start`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_audio_start: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_audio_start: {tts_audio_start} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **音频结束** (`tts_audio_end`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_audio_end: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_audio_end: {tts_audio_end} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### 指标日志

    **TTFB 指标** (`tts_ttfb`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_ttfb: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_ttfb: {ttfb} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **Metrics 指标** (`tts_metrics`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_metrics: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_metrics: {tts_metrics} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **注意**：以上日志均由基类 `AsyncTTS2BaseExtension` 自动处理，子类无需重复实现。开发者只需关注下面"必须实现的日志"部分。
  </div>
</div>

#### 必须实现的日志

<div className="fd-steps">
  <div className="fd-step">
    ##### 配置参数日志

    **类型**：配置参数

    **打印时机及内容**：在 `on_init` 回调里读取 `property` 并反序列化成 `config` 结构体后，加密打印 `config`

    **日志级别**：`Info`

    **日志分类**：`key_point`

    **日志格式**：`config: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_KEY_POINT

    ten_env.log_info(
        f"config: {self.config.to_str(sensitive_handling=True)}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **说明**：

    * 必须使用 `sensitive_handling=True` 对敏感信息进行脱敏处理
    * 配置信息应在 Extension 初始化时记录，便于问题排查
  </div>

  <div className="fd-step">
    ##### 供应商相关日志

    ###### 2.1 vendor\_error - 供应商错误日志

    **类型**：供应商相关

    **打印时机及内容**：当收到供应商返回的错误结果时，打印原始错误内容

    **日志级别**：`error`

    **日志分类**：`vendor`

    **日志格式**：`vendor_error: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_error(
        f"vendor_error: code: {code} reason: {cancellation_details.reason}, error_details: {cancellation_details.error_details}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 当供应商返回错误时，必须记录详细的错误信息
    * 包括错误代码、原因和详细错误信息

    ###### 2.2 vendor\_status - 供应商状态变化日志

    **类型**：供应商相关

    **打印时机及内容**：客户端与供应商服务器之间的状态变化

    **日志级别**：`debug`

    **日志分类**：`vendor`

    **日志格式**：`vendor_status: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_debug(
        f"vendor_status: connected to: {url}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 记录与供应商服务器的连接状态变化
    * 包括连接建立、断开、重连等状态

    ###### 2.3 send\_text\_to\_tts\_server - 发送文本到TTS服务器日志

    **类型**：供应商相关

    **打印时机及内容**：发送到 TTS 服务器的文本

    **日志级别**：`debug`

    **日志分类**：`vendor`

    **日志格式**：`send_text_to_tts_server`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_debug(
        f"send_text_to_tts_server: {text} of request_id: {request_id}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 记录发送到 TTS 服务器的文本内容和对应的 `request_id`
    * 用于追踪请求流程

    ###### 2.4 receive\_audio - 接收音频日志

    **类型**：供应商相关

    **打印时机及内容**：接收到的音频

    **日志级别**：`debug`

    **日志分类**：`vendor`

    **日志格式**：`receive_audio: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_debug(
        f"receive_audio: duration: {ms} of request id: {request_id}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 记录接收到的音频信息，包括音频时长和对应的 `request_id`
    * 用于监控音频接收情况
  </div>

  <div className="fd-step">
    ##### Extension 本身的关键日志

    ###### 3.1 skip\_tts\_text\_input - 跳过文本输入日志

    **类型**：过滤不发送的内容

    **打印时机及内容**：当某些文本输入被过滤不发送时

    **日志级别**：`debug`

    **日志分类**：`key_point`

    **日志格式**：`skip_tts_text_input`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_KEY_POINT

    self.ten_env.log_debug(
        f"skip_tts_text_input: {text} of request id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **说明**：

    * 当 TTS 厂商接收某些内容会有特殊行为需要避免发送时，应记录此日志
    * 记录被跳过的文本内容和对应的 `request_id`
  </div>
</div>

#### 日志最佳实践

1. **敏感信息保护**：配置参数等包含敏感信息的日志必须使用 `sensitive_handling=True` 进行脱敏
2. **日志级别选择**：
   * `error`：用于错误信息，如供应商返回的错误
   * `info`：用于关键信息，如配置加载
   * `debug`：用于调试信息，如状态变化、请求响应等
3. **日志分类使用**：
   * `LOG_CATEGORY_KEY_POINT`：Extension 本身的关键事件
   * `LOG_CATEGORY_VENDOR`：与供应商相关的所有日志
4. **日志格式统一**：使用统一的日志格式，便于日志分析和问题排查
5. **包含关键信息**：日志中应包含 `request_id` 等关键信息，便于追踪请求流程


# Create a Hello World Extension (/docs/ten_agent_examples/extension_dev/create_a_hello_world_extension)

In this chapter, we are going to create a `Hello World` extension step by step, available in Python, Go, and C++. Feel free to choose whichever language you prefer. So buckle up.

## Prerequisites

Before diving into this chapter, you’ll need to be familiar with the [basics covered earlier](/docs/ten_agent/getting_started). Specifically, ensure you understand how to use `docker compose up` and are aware of the services running in the background.

<div className="fd-steps">
  <div className="fd-step">
    ## Compose up the servers

    First things first, let’s start by composing the servers. Run the following command:

    <Callout>
      If the caption says `Terminal`, it means you are running the command locally.
      If the caption says `Bash`, it means you are running the command in the Docker
      container.
    </Callout>

    ```bash title="Terminal"
    docker compose up -d
    ```

    Once the command is entered, you should see output similar to this:

    ```bash title="Terminal"
    Attaching to ten_agent_dev, ten_agent_playground
    ten_agent_dev         | cd agents && tman designer
    ten_agent_dev         | :-)  Starting server at http://0.0.0.0:49483
    ten_agent_playground  |   ▲ Next.js 14.2.4
    ten_agent_playground  |   - Local:        http://localhost:3000
    ten_agent_playground  |   - Network:      http://0.0.0.0:3000
    ten_agent_playground  |
    ten_agent_playground  |  ✓ Starting...
    ten_agent_playground  |  ✓ Ready in 429ms
    ...
    ```

    Now, we’ve got the following services running:

    • `ten_agent_dev` at `http://0.0.0.0:49483` (TMAN Designer)

    • `ten_agent_playground` at `http://localhost:3000` (TEN Agent playground)
  </div>

  <div className="fd-step">
    ## Enter the docker container

    To work within the isolated environment, run the following command:

    ```bash title="Terminal"
    docker exec -it ten_agent_dev bash
    ```
  </div>

  <div className="fd-step">
    ## Create the hello world extension

    By running the following commands, an extension called `hello_world` will be created in Python, Go, or C++.

    import { Tab, Tabs } from 'fumadocs-ui/components/tabs'

    <Tabs items={['Python', 'Go', 'C++']} defaultIndex={0}>
      <Tab>
        ```bash title="Bash"
        cd /app/agents/ten_packages/extension
        tman create extension hello_world --template=default_async_extension_python@0.10 --template-data class_name_prefix=HelloWorld
        ```
      </Tab>

      <Tab>
        ```bash title="Bash"
        cd /app/agents/ten_packages/extension
        tman create extension hello_world --template=default_extension_go@0.10 --template-data class_name_prefix=HelloWorld
        ```
      </Tab>

      <Tab>
        ```bash title="Bash"
        cd /app/agents/ten_packages/extension
        tman create extension hello_world --template=default_extension_cpp@0.10 --template-data class_name_prefix=HelloWorld
        ```
      </Tab>
    </Tabs>

    After running the command, the log will display something like this:

    ```bash title="Bash"
    Package 'extension:hello_world' created successfully in '/app' in 3 seconds.
    ```
  </div>

  <div className="fd-step">
    ## Adding Extension Node to a graph

    Now launch TMAN Designer and open the graph you want to modify.

    1. Right-click anywhere on the canvas to open the context menu, then select "Add Node".
    2. In the dialog that appears, with the "Addon" field, select the `hello_world` extension you created earlier.
    3. For the name field give it a unique name `hello_world` in the graph.
    4. Click "Add Node" to place the extension node on the canvas.
    5. You should see the `hello_world` node appear on the canvas.
  </div>

  <div className="fd-step">
    ## Try your first message connection in graph

    An extension can send/receive messages to/from other nodes in the graph. Now let's create a simple message connection to give it a shot.
    Imagine we are going to let `hello_world` send a "Hello, World!" with message name `output_text` as output, and receive any message named `input_text` as input.

    ### 1). Update `manifest.json` to declare message protocols

    In the `manifest.json` file of your extension, you need to declare the message protocols that your extension will use. Add the following to the `api` section:

    ```json
    ...
      "api": {
        "data_in": [
          {
            "name": "input_text",
            "property": {
              "properties": {
                "text": {
                  "type": "string"
                }
              }
            }
          }
        ],
        "data_out": [
          {
            "name": "output_text",
            "property": {
              "properties": {
                "text": {
                  "type": "string"
                }
              }
            }
          }
        ]
      },
    ...
    ```

    In the `extension.py` file of your extension, we will create a simple async loop to send "Hello, World!" every 5 seconds.

    ```python
        async def on_start(self, ten_env: AsyncTenEnv) -> None:
            ten_env.log(LogLevel.DEBUG, "on_start")

            # IMPLEMENT: read properties, initialize resources
            asyncio.create_task(self._send_hello(ten_env))


        async def _send_hello(self, ten_env: AsyncTenEnv) -> None:
            while True:
                await asyncio.sleep(5)  # Send every 5 seconds
                data = Data.create("output_text")
                # set empty path means root of the data
                data.set_property_string("text", "Hello, World!")
                await ten_env.send_data(data)
    ```

    Save the file, then Choose "App->Reload All Apps" to reload extension settings.

    <Callout title="Note">
      This reload step is mandatory when you want changes in `manifest.json` to take effect after you change the file without TMAN Designer.
    </Callout>

    ### 2). Add connection from `hello_world` to `main`

    In TMAN Designer, right click on `hello_world` node, and choose "Add Connection from hello\_world". In the popup dialog, choose "data" as message type, and "output\_text" as message name. Then choose "main" as destination node, and click "Add Connection".
    The connection should be successfully created! You should see an edge connecting from `hello_world` to `main`.

    ### 3). Add code in `main` to process the input

    Now it's time to write some code to process incoming messages. Open the `main.py` file and add the following code:

    ```python
        # hello_wolrd/extension.py
        async def on_data(self, ten_env: AsyncTenEnv, data: Data):
            name = data.get_name()
            if name == "output_text":
                # process input_text directly here
                text, _ = data.get_property_string("text")
                if text:
                    ten_env.log_info(f"[MainControlExtension] Received input_text: {text}")
                    return

            await self.agent.on_data(data)
    ```

    ### 4). Reply a message from `main` to `hello_world`

    As the `main` file will talk to lots of other extensions, we will suggest you to send an outbound message by using `setDests` API.
    Using of this API is typically recommended in `main` only, as it is expected to know the context of the conversation and the intended recipients. Using `setDests` in other extensions may make the extension hard to reuse.

    ```python
        # main/extension.py
        async def on_data(self, ten_env: AsyncTenEnv, data: Data):
            name = data.get_name()
            if name == "output_text":
                # process input_text directly here
                text, _ = data.get_property_string("text")
                if text:
                    source_name = data.get_source().extension_name
                    ten_env.log_info(f"[MainControlExtension] Received input_text: {text} from: {source_name}")

                    # create a reply
                    reply = Data.create("input_text")
                    reply.set_property_string("text", f"Echo: {text}")
                    # reply to where it came from
                    reply.set_dests([Loc(
                        app_uri="",
                        graph_id="",
                        extension_name=source_name
                    )])
                    await ten_env.send_data(reply)
                    return

            await self.agent.on_data(data)
    ```

    And modify `hello_world` implementation to receive echo reply and print it in ten\_env logs.

    ```python
        async def on_data(self, ten_env: AsyncTenEnv, data: Data) -> None:
            data_name = data.get_name()
            ten_env.log(LogLevel.INFO, "on_data name {}".format(data_name))

            # IMPLEMENT: process data
            name = data.get_name()
            if name == "input_text":
                text, _ = data.get_property_string("text")
                if text:
                    ten_env.log_info(f"[HelloWorldExtension] Received input_text: {text}")
                    return
    ```
  </div>

  <div className="fd-step">
    ## Verify the extension

    Start the app using TMAN Designer. Check if following message is properly printed in the log console:

    ```
    [MainControlExtension] Received input_text: Hello, World!
    [HelloWorldExtension] Received input_text: Echo: Hello, World!
    ```

    Congratulations! You’ve successfully created your first `hello_world` extension.
  </div>
</div>


# Create an ASR Extension - Complete Guide (/docs/ten_agent_examples/extension_dev/create_asr_extension)

# Create an ASR Extension - Complete Guide

This guide covers the complete process of creating, developing, testing, and publishing an ASR Extension.

## Usage Guide

* **Basic Part**: Implement basic functionality to make the ASR Extension work and pass end-to-end verification.
* **Advanced Part**: Implement production-grade features (auto-reconnection, finalize mechanism, standardized logging, audio dump, etc.), perfect testing, and publish to the store.

Choose the appropriate section to read based on your needs.

## Prerequisites

* Understand the development and testing process of TEN Extensions. Refer to [How to Develop with Extensions](/docs/ten_framework/development/how_to_develop_with_ext).
* Master Python asynchronous programming (`asyncio`, `async/await`).
* Have the `tman` command-line tool installed and be familiar with its basic usage.
* Have an API key from an ASR vendor ready (for testing).

## Table of Contents

### Part 1: Basic - Implement Basic Functionality

1. [Introduction to ASR Extension](#1-introduction-to-asr-extension)
2. [Create Project](#2-create-project)
3. [Understand Interface Specifications](#3-understand-interface-specifications)
4. [Implement Core Functionality](#4-implement-core-functionality)
5. [Testing](#5-testing)
6. [Basic Development Checklist](#6-basic-development-checklist)

### Part 2: Advanced - Production-Grade Quality

7. [Auto Reconnection Mechanism](#7-auto-reconnection-mechanism)
8. [Optimize Finalize Mechanism](#8-optimize-finalize-mechanism)
9. [Standardized Logging](#9-standardized-logging)
10. [Error Reporting Specifications](#10-error-reporting-specifications)
11. [Audio Dump Functionality](#11-audio-dump-functionality)
12. [Audio Buffer Strategy](#12-audio-buffer-strategy)
13. [Perfect Testing](#13-perfect-testing)
14. [Advanced Development Checklist](#14-advanced-development-checklist)

### Part 3: Publish and Contribute

15. [Publish to TEN Store](#15-publish-to-ten-store)
16. [Adapt Other ASR Services](#16-adapt-other-asr-services)

### Appendix

* [A. ASR Interface Specifications](#a-asr-interface-specifications)
* [B. Base Class Methods Reference](#b-base-class-methods-reference)
* [C. Full Code Example](#c-full-code-example)
* [D. Debugging Tools](#d-debugging-tools)

***

# Part 1: Basic - Implement Basic Functionality

<div className="fd-steps">
  <div className="fd-step">
    ## Introduction to ASR Extension

    The ASR Extension is one of the standard building blocks of the TEN Framework, responsible for **transcribing audio streams into text** in corresponding languages in **real-time**.

    ### Position in the Conversation Flow

    ```
    [RTC Extension] ──audio stream──> [ASR Extension] ──text stream──> [LLM Extension]
    ```
  </div>

  <div className="fd-step">
    ## Create Project

    ### 2.0 Preparation

    <Callout type="info">
      If you haven't cloned the TEN Framework repository yet, please clone it first:

      ```bash
      git clone https://github.com/TEN-framework/ten-framework.git
      cd ten-framework
      ```

      This tutorial assumes you are developing in the root directory of the TEN Framework repository.
    </Callout>

    ### 2.1 Create with Template

    Use the `tman` command-line tool to create a project from the ASR extension template:

    ```bash
    cd ai_agents/agents/ten_packages/extension
    tman create extension my_asr_extension --template default_asr_python --template-data class_name_prefix=MyAsr
    ```

    **Command Parameter Explanation**:

    * `extension my_asr_extension`: Create an extension with the directory name and plugin name `my_asr_extension`.
    * `--template default_asr_python`: Use the ASR Python extension template.
    * `--template-data class_name_prefix=MyAsr`: Set the class name prefix for the Python Extension class to `MyAsr` (the generated class name will be `MyAsrExtension`).

    ### 2.2 Install TEN Package Dependencies

    ```bash
    cd my_asr_extension
    tman install --standalone
    ```

    <Callout type="tip">
      This command uses the dependency calculation and download capabilities of the `tman` tool to calculate the dependency tree based on the dependencies declared in `manifest.json`. Since these are development dependencies, the `--standalone` parameter is required. Dependencies will be installed in the `.ten` directory, including Python modules for development and system packages for standalone testing.
    </Callout>

    ### 2.3 Project Structure

    ```
    my_asr_extension/
    ├── manifest.json       # Extension metadata
    ├── property.json       # Default configuration
    ├── requirements.txt    # Python dependencies
    ├── extension.py        # Main implementation file
    ├── addon.py            # Extension entry point
    ├── __init__.py         # Python package initialization
    ├── docs/               # Documentation directory
    ├── .vscode/            # VS Code debug configuration
    └── tests/              # Test files
    ```

    #### File Descriptions

    **manifest.json and property.json** are standard metadata files for TEN Extensions:

    * `manifest.json`: Contains the extension's **name, version, description, dependencies, and schema definitions**.
    * `property.json`: Defines the extension's **default property values**.

    <Callout type="info">
      For a detailed understanding of the TEN Framework metadata system, please refer to the [Metadata System Documentation](/docs/ten_framework/metadata_system/metadata_system).
    </Callout>

    **requirements.txt** is unique to Python extensions and is used to declare the extension's **dependencies on third-party pip packages**.

    **extension.py** is the **core source code** of the extension, containing all business logic implementation.

    **tests/** folder is used for **standalone extension testing**, including unit tests and test configurations.
  </div>

  <div className="fd-step">
    ## Understand Interface Specifications

    The ASR Extension inherits from `AsyncASRBaseExtension` and needs to implement the following methods:

    ### Required Methods

    ```python
    vendor() -> str                                          
        # Return vendor name (e.g., "deepgram", "azure")

    start_connection() -> None                               
        # Establish connection with the vendor

    stop_connection() -> None                                
        # Stop connection

    send_audio(frame: AudioFrame, session_id: str | None) -> bool  
        # Send audio frame to the vendor
        # session_id: Audio source identifier (can be ignored for single-person scenarios)
        # Returns True if sent successfully

    finalize(session_id: str | None) -> None                 
        # Called when VAD detects end of speech to trigger final results

    is_connected() -> bool                                   
        # Return current connection status

    input_audio_sample_rate() -> int                         
        # Return expected audio sample rate (e.g., 16000)
    ```

    ### Base Class Automatic Handling

    The base class `AsyncASRBaseExtension` already handles:

    * Audio frame reception and queue management
    * Performance metric calculation (TTFW, TTLW)
    * Session management and metadata passing

    You only need to focus on integrating with the specific ASR vendor.
  </div>

  <div className="fd-step">
    ## Implement Core Functionality

    ### 4.1 Configuration Management

    #### Config Model Design

    ```python
    from pydantic import BaseModel
    from typing import Dict, Optional

    class MyAsrConfig(BaseModel):
        # Vendor parameters (pass-through design)
        params: Dict[str, Optional[str]] = {}
        
        # Audio dump functionality
        dump: bool = False
        dump_path: Optional[str] = None
    ```

    #### Advantages of Params Pass-Through Design

    `params` is a dictionary used to store **all vendor-related parameters**. These parameters will be **passed directly to the vendor SDK** without needing to be enumerated in the Extension code.

    **Why design it this way?**

    ✅ **Flexibility**: Users can configure any parameter supported by the vendor via `property.json`, without being limited by the Extension implementation.

    ✅ **Extensibility**: When vendors add new parameters, the Extension code does not need modification.

    ✅ **Simplicity**: Avoids defining a configuration field for every single parameter.

    **Example**:

    ```json title="property.json"
    {
      "params": {
        "api_key": "your_api_key_here",
        "language": "zh-CN",
        "model": "nova-2",
        "punctuate": "true",
        "custom_param": "any_value"  // Any parameter supported by the vendor
      },
      "dump": false
    }
    ```

    Usage - directly pass through to the vendor:

    ```python
    # Read from params and pass to vendor
    api_key = self.config.params.get("api_key")
    language = self.config.params.get("language", "en-US")  # With default value
    ```

    <Callout type="warning">
      **Note**: The `property.json` file generated by the template is empty `{}`. You need to manually add configurations.
    </Callout>

    ### 4.2 Read Configuration

    ```python
    @override
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        await super().on_init(ten_env)
        
        # Read configuration
        config_json, _ = await ten_env.get_property_to_json("")
        self.config = MyAsrConfig.model_validate_json(config_json)
        
        ten_env.log_info(f"Config loaded: {self.config.model_dump_json()}")
    ```

    ### 4.3 Implement Basic Methods

    ```python
    class MyAsrExtension(AsyncASRBaseExtension):
        def __init__(self, name: str):
            super().__init__(name)
            self.config: MyAsrConfig = MyAsrConfig()
            self.client = None  # Vendor SDK client
            self.is_connected_flag = False
            
        @override
        def vendor(self) -> str:
            return "my_vendor"  # Change to your vendor name
        
        @override
        def input_audio_sample_rate(self) -> int:
            # Read from params with default value
            return int(self.config.params.get("sample_rate", "16000"))
        
        @override
        def is_connected(self) -> bool:
            return self.is_connected_flag
    ```

    ### 4.4 Implement Connection Management

    <Callout type="warning">
      **Important**: The following code is an example only. The actual implementation depends on your vendor.

      Different vendors have very different connection methods:

      * Some provide ready-made **SDKs** (e.g., Azure, Deepgram).
      * Some require direct **WebSocket** connections.
      * Some use **HTTP streaming APIs**.

      Please refer to your vendor's documentation and use the corresponding connection method.
    </Callout>

    #### Establish Connection

    ```python
    @override
    async def start_connection(self) -> None:
        """Establish connection with the vendor"""
        try:
            await self.stop_connection()  # Stop existing connection first
            
            # 1. Initialize vendor client
            # Pass parameters from params to vendor SDK
            # Example (adjust according to actual vendor):
            self.client = VendorClient(
                api_key=self.config.params.get("api_key"),
                language=self.config.params.get("language", "en-US"),
                # ... Read other parameters directly from params and pass through
            )
            
            # 2. Register event handlers
            # Different vendors have different event mechanisms; this is just an example
            self.client.on("connected", self._on_open)
            self.client.on("result", self._on_transcript)
            self.client.on("error", self._on_error)
            
            # 3. Start connection
            await self.client.connect()
            
        except Exception as e:
            self.ten_env.log_error(f"Failed to connect: {e}")

    @override
    async def stop_connection(self) -> None:
        """Stop connection"""
        if self.client:
            await self.client.disconnect()
            self.client = None
            self.is_connected_flag = False
    ```

    <Callout type="tip">
      **Value of Params Pass-Through**: All parameters are read from the `params` dictionary and passed through to the vendor SDK. This allows users to flexibly configure any parameter supported by the vendor without modifying the code.

      **Reference Existing Implementations**:

      * `azure_asr_python` - Uses Azure SDK
      * `deepgram_asr_python` - Uses Deepgram SDK
    </Callout>

    ### 4.5 Send Audio

    ```python
    @override
    async def send_audio(self, audio_frame: AudioFrame, session_id: str | None) -> bool:
        """Send audio data
        
        Args:
            audio_frame: Audio frame
            session_id: Session ID, used to identify different conversation turns
            
        Returns:
            bool: Returns True if sent successfully, otherwise False
        """
        if not self.is_connected() or not self.client:
            return False
            
        try:
            audio_buf = audio_frame.get_buf()
            if audio_buf:
                await self.client.send(bytes(audio_buf))
                return True
        except Exception as e:
            self.ten_env.log_error(f"Failed to send audio: {e}")
        
        return False
    ```

    ### 4.6 Handle Recognition Results

    The vendor will return recognition results via callback functions. You need to convert them to the standard format and send them:

    ```python
    async def _on_transcript(self, result):
        """Handle recognition results"""
        # 1. Extract text
        text = result.text.strip()
        if not text:
            return
        
        # 2. Convert to standard ASR result
        asr_result = ASRResult(
            text=text,
            final=result.is_final,           # Whether it is the final result
            start_ms=result.start_time_ms,   # Start time
            duration_ms=result.duration_ms   # Duration
        )
        
        # 3. Send to downstream
        await self.send_asr_result(asr_result)
    ```

    Other required event handlers:

    ```python
    async def _on_open(self):
        """Connection established"""
        self.is_connected_flag = True
        self.ten_env.log_info("Connection opened")

    async def _on_error(self, error):
        """Handle error"""
        self.ten_env.log_error(f"Vendor error: {error}")
    ```

    ### 4.7 Implement Finalize

    When VAD detects that the user has finished speaking, it calls the `finalize()` method to notify the vendor to return the final result as soon as possible:

    ```python
    @override
    async def finalize(self, session_id: str | None) -> None:
        """Trigger final result"""
        if self.client:
            # Different vendors have different finalize methods:
            # 1. Call finalize API (Recommended)
            await self.client.finalize()
            
            # 2. Or send silence packets
            # await self.client.send_silence()
            
            # 3. Or disconnect and reconnect
            # await self.stop_connection()
            # await self.start_connection()
            
            # Notify completion
            await self.send_asr_finalize_end()
    ```

    <Callout type="info">
      Different vendors have different finalize mechanisms. Choose the one that best suits your vendor. For detailed explanation, see the [Advanced Part](#8-optimize-finalize-mechanism).
    </Callout>

    Basic functionality is now implemented! Next, let's test it.
  </div>

  <div className="fd-step">
    ## Testing

    ### 5.1 Unit Testing

    Unit tests are used to verify that the basic functionality of the ASR Extension works correctly.

    #### Test Objectives

    Basic functionality needs to verify at least:

    1. **Config Loading**: Can correctly read configurations from `property.json`.
    2. **Connection Establishment**: Can successfully connect to the vendor.
    3. **Audio Processing**: Can receive and send audio frames.
    4. **Result Output**: Can output results in the standard `ASRResult` format.

    #### Test Flow Example

    A typical unit test will:

    1. **Prepare Test Audio**: Load PCM audio data from a file.
    2. **Send Frame by Frame**: Send audio to the ASR Extension frame by frame.
    3. **Verify Results**: Check if `ASRResult` in the standard format is received.

    **ASRResult Standard Format**:

    ```python
    ASRResult(
        text="Recognized text",           # Required: Recognized text
        final=True,                       # Required: Whether it is the final result
        start_ms=0,                       # Optional: Start time (ms)
        duration_ms=1000,                 # Optional: Duration (ms)
        language="zh-CN",                 # Optional: Language
        words=[]                          # Optional: Word-level information
    )
    ```

    #### Run Tests

    ```bash
    cd my_asr_extension
    ./tests/bin/start
    ```

    <Callout type="tip">
      The template already contains basic test cases. You can add more tests in `tests/test_basic.py`.

      Reference tests from existing implementations:

      * `azure_asr_python/tests/test_asr_result.py` - Test result output
      * `deepgram_asr_python/tests/test_basic.py` - Basic functionality tests
    </Callout>

    ### 5.2 End-to-End Testing

    Use TMan Designer in the TEN Agent project to replace the ASR extension:

    ```bash
    cd /path/to/your/ten-agent-project
    tman designer
    ```

    Via the visual interface:

    1. Select the existing ASR node.
    2. Replace it with your `my_asr_extension`.
    3. Configure the API Key and other parameters.
    4. Start and conduct real conversation testing.
  </div>

  <div className="fd-step">
    ## Basic Development Checklist

    Complete the following checks to ensure basic functionality works:

    * [ ] **Project Creation**: Successfully created project using template, installed dependencies.
    * [ ] **Config Management**: Can correctly read config from property and properly pass through to vendor SDK.
    * [ ] **Connection Establishment**: `start_connection` can successfully connect to the vendor.
    * [ ] **Audio Sending**: Can send audio frames to the vendor via `send_audio`.
    * [ ] **Result Reception**: Can correctly receive recognition results from the vendor and convert to standard format.
    * [ ] **Result Sending**: Can send standardized results via `send_asr_result`.
    * [ ] **Finalize**: Implemented `finalize` method (even if simple implementation).
    * [ ] **Connection Cleanup**: `stop_connection` can correctly close the connection.
    * [ ] **Unit Testing**: Basic test cases pass.
    * [ ] **End-to-End Testing**: Can complete basic conversations in TEN Agent.

    <Callout type="success">
      After completing the above checks, your ASR Extension has basic functionality and can be used in real scenarios.
    </Callout>

    ***
  </div>
</div>

# Part 2: Advanced - Production-Grade Quality

The advanced part will help you implement a production-grade ASR Extension, covering stability, maintainability, and debuggability.

<div className="fd-steps">
  <div className="fd-step">
    ## Auto Reconnection Mechanism

    ### 7.1 Why Reconnection is Needed

    Network environments are complex, and ASR services may experience:

    * Temporary network jitter
    * Server-side active disconnection
    * Timeout errors

    Implementing auto-reconnection ensures service stability.

    ### 7.2 Use ReconnectManager

    Refer to `reconnect_manager.py` in existing ASR extensions (e.g., `azure_asr_python`).

    ```python
    from .reconnect_manager import ReconnectManager

    class MyAsrExtension(AsyncASRBaseExtension):
        def __init__(self, name: str):
            super().__init__(name)
            # Retry up to 5 times, base delay 0.5 seconds (exponential backoff)
            self.reconnect_manager = ReconnectManager(max_attempts=5, base_delay=0.5)
    ```

    ### 7.3 Reset on Connection Success

    ```python
    async def _on_open(self, *args, **kwargs) -> None:
        self.is_connected_flag = True
        self.ten_env.log_info("Connection opened", category=LOG_CATEGORY_VENDOR)
        
        if self.reconnect_manager:
            self.reconnect_manager.mark_connection_successful()
    ```

    ### 7.4 Handle Disconnection and Errors

    ```python
    async def _on_close(self, *args, **kwargs) -> None:
        self.is_connected_flag = False
        self.ten_env.log_warn("Connection closed", category=LOG_CATEGORY_VENDOR)
        
        if self.client:  # Unexpected disconnection
            await self._handle_reconnect()

    async def _on_error(self, *args, **kwargs) -> None:
        error = args[1] if len(args) > 1 else None
        self.ten_env.log_error(f"Vendor error: {error}", category=LOG_CATEGORY_VENDOR)
        
        await self.send_asr_error(
            ModuleError(module=MODULE_NAME_ASR, code=ModuleErrorCode.NON_FATAL_ERROR.value, message=str(error))
        )
        await self._handle_reconnect()
    ```

    ### 7.5 Implement Reconnection Logic

    ```python
    async def _handle_reconnect(self) -> None:
        if not self.reconnect_manager or not self.reconnect_manager.can_retry():
            self.ten_env.log_error("Max reconnection attempts reached", category=LOG_CATEGORY_VENDOR)
            await self.send_asr_error(
                ModuleError(module=MODULE_NAME_ASR, code=ModuleErrorCode.FATAL_ERROR.value, message="Reconnection failed")
            )
            return
        
        self.ten_env.log_info(
            f"Attempting reconnection {self.reconnect_manager.current_attempts + 1}/{self.reconnect_manager.max_attempts}",
            category=LOG_CATEGORY_VENDOR
        )
        
        success = await self.reconnect_manager.handle_reconnect(connect_func=self.start_connection)
        
        if success:
            self.ten_env.log_info("Reconnection successful", category=LOG_CATEGORY_VENDOR)
        else:
            self.ten_env.log_error("Reconnection failed", category=LOG_CATEGORY_VENDOR)
    ```
  </div>

  <div className="fd-step">
    ## Optimize Finalize Mechanism

    ### 8.1 Role of Finalize

    Trigger ASR to return the final result immediately when VAD detects the end of human speech to reduce conversation latency.

    ### 8.2 Three Implementation Methods

    Different vendors support different methods:

    1. **Call API** (Recommended, e.g., Deepgram)
    2. **Disconnect and Reconnect** (Need to handle reconnection logic well)
    3. **Send Silence Packets** (Need to pay attention to timestamp calculation)

    ### 8.3 Notify Completion

    ```python
    @override
    async def finalize(self, session_id: str | None) -> None:
        """Trigger final result
        
        Args:
            session_id: Session ID, used to identify different conversation turns
        """
        if self.client:
            # Record timestamp for latency calculation
            self.last_finalize_timestamp = asyncio.get_event_loop().time() * 1000
            
            # Method 1: Call API
            await self.client.finalize()
            
            # Notify completion
            await self.send_asr_finalize_end()
    ```
  </div>

  <div className="fd-step">
    ## Standardized Logging

    ### 9.1 Log Categories

    Use the `category` parameter to clarify logs:

    * `LOG_CATEGORY_KEY_POINT`: Key points (configuration, initialization)
    * `LOG_CATEGORY_VENDOR`: Vendor-related (connection, results, errors)

    ```python
    from ten_ai_base.const import LOG_CATEGORY_KEY_POINT, LOG_CATEGORY_VENDOR

    self.ten_env.log_info("Config loaded", category=LOG_CATEGORY_KEY_POINT)
    self.ten_env.log_error("Connection failed", category=LOG_CATEGORY_VENDOR)
    ```

    ### 9.2 Key Log Points

    Logs that must be printed:

    ```python
    # Configuration Loading
    self.ten_env.log_info(f"Config: {self.config.to_json(sensitive_handling=True)}", category=LOG_CATEGORY_KEY_POINT)

    # Connection Status Changes
    self.ten_env.log_info("Connection opened", category=LOG_CATEGORY_VENDOR)
    self.ten_env.log_warn("Connection closed", category=LOG_CATEGORY_VENDOR)

    # Error Information
    self.ten_env.log_error(f"Vendor error: {error}", category=LOG_CATEGORY_VENDOR)

    # Reconnection Attempts
    self.ten_env.log_info(f"Reconnecting {attempt}/{max_attempts}", category=LOG_CATEGORY_VENDOR)
    ```

    ### 9.3 Sensitive Information Masking

    ```python
    from ten_ai_base.utils import encrypt

    class MyAsrConfig(BaseModel):
        params: Dict[str, Optional[str]] = {}
        
        def to_json(self, sensitive_handling: bool = False) -> str:
            if not sensitive_handling:
                return self.model_dump_json()
            
            config = self.model_copy(deep=True)
            if config.params:
                for key in ['api_key', 'key', 'token', 'secret']:
                    if key in config.params and config.params[key]:
                        config.params[key] = encrypt(config.params[key])
            return config.model_dump_json()
    ```
  </div>

  <div className="fd-step">
    ## Error Reporting Specifications

    ### 10.1 Error Classification

    **Fatal Error (FATAL\_ERROR)**:

    * Configuration parsing failure
    * Invalid API Key
    * Initial connection failure
    * Reached maximum reconnection attempts

    **Non-Fatal Error (NON\_FATAL\_ERROR)**:

    * Temporary network issues
    * Service temporarily unavailable
    * Audio processing errors

    ### 10.2 Include Vendor Information

    ```python
    from ten_ai_base.message import ModuleError, ModuleErrorCode, ModuleErrorVendorInfo

    await self.send_asr_error(
        ModuleError(
            module=MODULE_NAME_ASR,
            code=ModuleErrorCode.NON_FATAL_ERROR.value,
            message=f"Vendor error: {str(error)}"
        ),
        ModuleErrorVendorInfo(
            vendor="deepgram",
            code=getattr(error, 'code', 'unknown'),
            message=str(error)
        )
    )
    ```
  </div>

  <div className="fd-step">
    ## Audio Dump Functionality

    ### 11.1 Why Dump is Needed

    Save original audio when recognition issues occur for:

    * Reproducing issues
    * Analyzing audio quality
    * Comparing different vendors

    ### 11.2 Implement Dump

    ```python
    import os
    from ten_ai_base.dumper import Dumper

    DUMP_FILE_NAME = "my_asr_in.pcm"

    class MyAsrExtension(AsyncASRBaseExtension):
        def __init__(self, name: str):
            super().__init__(name)
            self.audio_dumper: Optional[Dumper] = None
        
        @override
        async def on_init(self, ten_env: AsyncTenEnv) -> None:
            await super().on_init(ten_env)
            
            if self.config.dump:
                dump_file_path = os.path.join(self.config.dump_path, DUMP_FILE_NAME)
                self.audio_dumper = Dumper(dump_file_path)
                await self.audio_dumper.start()
        
        @override
        async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
            await super().on_deinit(ten_env)
            if self.audio_dumper:
                await self.audio_dumper.stop()
                self.audio_dumper = None
        
        @override
        async def send_audio(self, audio_frame: AudioFrame) -> bool:
            buf = audio_frame.get_buf()
            
            # Dump audio
            if self.audio_dumper and buf:
                await self.audio_dumper.push_bytes(bytes(buf))
            
            # Send audio
            if self.is_connected() and self.client:
                await self.client.send(bytes(buf))
                return True
            
            return False
    ```
  </div>

  <div className="fd-step">
    ## Audio Buffer Strategy

    ### 12.1 Why Recommend Keep Mode

    Default uses **Drop Mode**: Drop audio frames when disconnected.

    Recommend using **Keep Mode**: Cache audio frames and send them after the connection is restored.

    **Reason**: Ensures timestamp accuracy.

    If audio is dropped:

    * Time: 0-10s (Sent) → 10-15s (Disconnected & Dropped) → 15-20s (Sent)
    * ASR only receives 15s of audio, but the actual time span is 20s.
    * Timestamp deviation of 5s affects conversation synchronization and interruption detection.

    ### 12.2 Configure Keep Mode

    ```python
    from ten_ai_base.asr import ASRBufferConfig, ASRBufferConfigModeKeep

    @override
    def buffer_strategy(self) -> ASRBufferConfig:
        return ASRBufferConfig(
            mode=ASRBufferConfigModeKeep(byte_limit=10 * 1024 * 1024)  # 10MB cache limit
        )
    ```
  </div>

  <div className="fd-step">
    ## Perfect Testing

    ### 13.1 Unit Test Coverage

    Advanced unit tests need to cover production-grade features. Use Mock to avoid real API calls, ensuring tests are fast, stable, and repeatable.

    #### Test Case Design

    Referring to the implementation in `azure_asr_python/tests`, advanced tests should cover:

    **1. Reconnection Capability Test** (`test_reconnect.py`)

    **Test Objective**: Verify that the Extension can automatically reconnect and restore service.

    **Case Design**:

    ```python
    # Simulate vendor disconnection scenario
    def test_reconnect():
        # 1. Mock vendor: The first 3 connections will disconnect, the 4th succeeds
        # 2. Verify Extension automatically retries
        # 3. Verify normal operation after successful reconnection
        # 4. Check error report count (should have 3 NON_FATAL_ERROR)
    ```

    **Verification Points**:

    * ✅ Automatically trigger reconnection after disconnection
    * ✅ Use exponential backoff strategy
    * ✅ Report FATAL\_ERROR when max reconnection attempts reached
    * ✅ Reset counter after successful reconnection

    ***

    **2. Invalid Parameters Test** (`test_invalid_params.py`)

    **Test Objective**: Verify error handling when configuration is incorrect.

    **Case Design**:

    ```python
    # Start Extension with invalid parameters
    def test_invalid_params():
        # 1. Provide empty or invalid params (e.g., missing api_key)
        # 2. Start Extension
        # 3. Verify FATAL_ERROR received
        # 4. Check if error message contains useful information
    ```

    **Verification Points**:

    * ✅ Report FATAL\_ERROR when configuration validation fails
    * ✅ Error message is clear and easy to troubleshoot
    * ✅ Extension doesn't crash

    ***

    **3. Audio Dump Test** (`test_dump.py`)

    **Test Objective**: Verify the completeness of audio dump functionality.

    **Case Design**:

    ```python
    # Send audio after enabling dump
    def test_dump():
        # 1. Configure dump=True and dump_path
        # 2. Send N frames of audio (each frame has specific byte pattern)
        # 3. Check dump file after test ends
        # 4. Verify file size = N * frame size
        # 5. Verify each frame content is identical
    ```

    **Verification Points**:

    * ✅ Dump file is created
    * ✅ All sent audio is completely dumped
    * ✅ Dump content is identical to sent content (byte-by-byte verification)
    * ✅ Frame order is correct

    ***

    **4. Finalize Latency Test** (`test_finalize.py`)

    **Test Objective**: Verify Extension can quickly output final results.

    **Case Design**:

    ```python
    # Test finalize response speed
    def test_finalize():
        # 1. Continuously send audio
        # 2. Send asr_finalize event after 1.5 seconds
        # 3. Mock vendor quickly returns final result after receiving finalize
        # 4. Verify asr_finalize_end event received
        # 5. Check if finalize_id and metadata are correctly passed
    ```

    **Verification Points**:

    * ✅ Trigger vendor's finalize after receiving `asr_finalize`
    * ✅ Quickly receive final result (\< 300ms typical)
    * ✅ Send `asr_finalize_end` to notify downstream
    * ✅ `finalize_id` and `session_id` correctly passed

    ***

    **5. Result Format Test** (`test_asr_result.py`)

    **Test Objective**: Verify standard format of ASR results.

    **Case Design**:

    ```python
    # Verify output result data structure
    def test_asr_result():
        # 1. Mock vendor returns recognition results
        # 2. Verify ASRResult contains all required fields
        # 3. Verify both interim and final results are correct
        # 4. Verify metadata is correctly passed (e.g., session_id)
    ```

    **Verification Points**:

    * ✅ Contains required fields: `text`, `final`, `start_ms`, `duration_ms`, `language`
    * ✅ Optional fields correctly filled: `words`, `metadata`
    * ✅ `session_id` correctly passed from input to output

    ***

    **6. Error Reporting Test** (`test_vendor_error.py`)

    **Test Objective**: Verify error classification and vendor information reporting.

    **Case Design**:

    ```python
    # Simulate vendor returning errors
    def test_vendor_error():
        # 1. Mock vendor returns different types of errors
        # 2. Verify Extension reports correct error types
        # 3. Check if ModuleErrorVendorInfo is included
    ```

    **Verification Points**:

    * ✅ Temporary errors report NON\_FATAL\_ERROR
    * ✅ Serious errors report FATAL\_ERROR
    * ✅ Contains vendor error code and message
    * ✅ Error information is helpful for debugging

    ***

    **7. Performance Metrics Test** (`test_metrics.py`)

    **Test Objective**: Verify performance metrics are correctly calculated and reported.

    **Case Design**:

    ```python
    # Test TTFW, TTLW metrics
    def test_metrics():
        # 1. Send audio
        # 2. Mock vendor returns first and last word at specific times
        # 3. Verify calculated TTFW and TTLW metrics
    ```

    **Verification Points**:

    * ✅ TTFW (first word latency) correctly calculated
    * ✅ TTLW (last word latency) correctly calculated
    * ✅ Metrics reported through `metrics` message

    ***

    #### Run Tests

    ```bash
    cd my_asr_extension
    ./tests/bin/start
    ```

    <Callout type="info">
      **Importance of Mock**: Reasons for using Mock instead of real API:

      * 🚀 **Fast**: Tests complete in seconds
      * 💰 **Zero Cost**: Doesn't consume API quota
      * 🎯 **Controllable**: Can precisely simulate various scenarios (disconnection, errors, latency)
      * 🔁 **Repeatable**: Stable results, suitable for CI/CD

      Refer to `azure_asr_python/tests/mock.py` to learn how to implement Mock.
    </Callout>

    ### 13.2 Integration Testing (Guarder)

    Run with real API Key:

    ```bash
    cd ai_agents
    task asr-guarder-test EXTENSION=my_asr_extension
    ```

    Test content:

    * ASR recognition accuracy
    * Finalize latency
    * Multi-language support
    * Error handling
    * Performance metrics

    ### 13.3 VS Code Debugging

    Use the preset `.vscode/launch.json`:

    1. Set breakpoints in the code.
    2. Press `F5` to start debugging.
    3. Select "Python: Test Extension".
  </div>

  <div className="fd-step">
    ## Advanced Development Checklist

    Complete the following checks to ensure production-grade quality:

    **Stability**:

    * [ ] Implement `ReconnectManager` auto-reconnection
    * [ ] Exponential backoff strategy (avoid frequent reconnection)
    * [ ] Maximum reconnection attempts limit
    * [ ] Reset reconnection counter after connection success

    **Finalize Mechanism**:

    * [ ] Implement rapid finalize triggered by VAD
    * [ ] Call `send_asr_finalize_end()` to notify completion
    * [ ] Handle reconnection logic if using disconnect method
    * [ ] Correctly calculate timestamps if using silence packets

    **Logging Specifications**:

    * [ ] Use `LOG_CATEGORY_KEY_POINT` and `LOG_CATEGORY_VENDOR` categories
    * [ ] Log connection status changes
    * [ ] Log all error information
    * [ ] Mask sensitive information (API Key, etc.)

    **Error Reporting**:

    * [ ] Correctly distinguish FATAL\_ERROR and NON\_FATAL\_ERROR
    * [ ] Include `ModuleErrorVendorInfo` vendor information
    * [ ] Report FATAL\_ERROR for configuration errors
    * [ ] Report NON\_FATAL\_ERROR for temporary network errors

    **Audio Dump**:

    * [ ] Implement `Dumper` integration
    * [ ] Support `dump` and `dump_path` configuration
    * [ ] Initialize in `on_init`, clean up in `on_deinit`

    **Audio Buffer**:

    * [ ] Implement `buffer_strategy()` to return Keep Mode
    * [ ] Set reasonable cache limit (e.g., 10MB)

    **Test Coverage**:

    * [ ] Unit tests cover all core functionality
    * [ ] Use Mock to avoid real API calls
    * [ ] Guarder tests pass (real API)
    * [ ] End-to-end tests verify stability

    **Code Quality**:

    * [ ] Follow project code style
    * [ ] Add necessary comments
    * [ ] Provide clear README

    <Callout type="success">
      After completing the above checks, your ASR Extension has reached production-grade quality and can be contributed to the community and published to the store.
    </Callout>

    ***
  </div>
</div>

# Part 3: Publish and Contribute

<div className="fd-steps">
  <div className="fd-step">
    ## Publish to TEN Store

    ### 15.1 Submit to Main Repository

    ```bash
    # Fork TEN Framework repository
    git clone https://github.com/your-username/ten-framework.git
    cd ten-framework

    # Copy extension to correct location
    cp -r /path/to/your/my_asr_extension ai_agents/agents/ten_packages/extension/

    # Create branch
    git checkout -b feat/add-my-asr-extension

    # Commit
    git add ai_agents/agents/ten_packages/extension/my_asr_extension/
    git commit -m "feat: add my_asr_extension for [Vendor Name] ASR service"
    git push origin feat/add-my-asr-extension
    ```

    ### 15.2 Create Pull Request

    1. Visit your fork repository.
    2. Click "Compare & pull request".
    3. Fill in the title and description (functionality, features, test status).
    4. Submit and wait for review.

    ### 15.3 Auto Publish

    After PR merge:

    * ✅ Automatically uploaded to TEN Store
    * ✅ Automatically handle version numbers
    * ✅ Available for global developers to download and use
  </div>

  <div className="fd-step">
    ## Adapt Other ASR Services

    Refer to other finished ASR extensions in the TEN Framework:

    ```bash
    ten-framework/ai_agents/agents/ten_packages/extension/
    ├── azure_asr_python/          # Azure Speech Services
    ├── deepgram_asr_python/       # Deepgram ASR
    ├── google_asr_python/         # Google Cloud Speech
    ├── xfyun_asr_python/          # iFlytek (XFYun)
    └── ...                        # More extensions
    ```

    All extensions follow the same architecture and can serve as references for adapting new services.

    ***
  </div>
</div>

# Appendix

## A. ASR Interface Specifications

### A.1 Interface Inheritance

Declare in `manifest.json`:

```json
{
  "api": {
    "interface": [
      {
        "import_uri": "../../system/ten_ai_base/api/asr-interface.json"
      }
    ]
  }
}
```

### A.2 Standard Properties

Standard properties defined in `asr-interface.json`:

* `dump`: Boolean, whether to enable audio dump.
* `dump_path`: String, audio dump storage path.

### A.3 Extended Properties

Declare unique properties in `api.property`:

```json
{
  "api": {
    "property": {
      "properties": {
        "params": {
          "type": "object",
          "properties": {
            "key": { "type": "string" },
            "language": { "type": "string" }
          }
        }
      }
    }
  }
}
```

### A.4 Input/Output Data Formats

**Input**:

* `pcm_frame`: PCM audio frame
* `asr_finalize`: VAD detected end of speech event

**Output**:

* `asr_result`: Recognition result
* `asr_finalize_end`: Finalize completion notification
* `error`: Error information
* `metrics`: Performance metrics

See `asr-interface.json` file for details.

## B. Base Class Methods Reference

### B.1 AsyncASRBaseExtension

**Required Implementation**:

* `vendor() -> str`
* `start_connection() -> None`
* `stop_connection() -> None`
* `send_audio(frame: AudioFrame, session_id: str | None) -> bool`
* `finalize(session_id: str | None) -> None`
* `is_connected() -> bool`
* `input_audio_sample_rate() -> int`

**Optional Implementation**:

* `input_audio_channels() -> int`
* `input_audio_sample_width() -> int`
* `buffer_strategy() -> ASRBufferConfig`
* `audio_actual_send_metrics_interval() -> int`

**Utility Methods**:

* `send_asr_result(asr_result: ASRResult)`
* `send_asr_error(error: ModuleError, vendor_info: ModuleErrorVendorInfo | None)`
* `send_asr_finalize_end()`
* `send_connect_delay_metrics(connect_delay: int)`
* `send_vendor_metrics(vendor_metrics: dict)`

## C. Full Code Example

Refer to the full implementation of `deepgram_asr_python` or `azure_asr_python` extension:

```
ten-framework/ai_agents/agents/ten_packages/extension/
├── deepgram_asr_python/
│   ├── extension.py           # Full implementation
│   ├── reconnect_manager.py   # Reconnection manager
│   ├── manifest.json
│   ├── property.json
│   └── tests/                 # Full test cases
```

## D. Debugging Tools

### D.1 VS Code Debug Configuration

`.vscode/launch.json`:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Test Extension",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/tests/bin/start",
      "console": "integratedTerminal",
      "cwd": "${workspaceFolder}",
      "env": {
        "PYTHONPATH": "${workspaceFolder}:..."
      }
    }
  ]
}
```

### D.2 Debug Specific Test

Modify `args` parameter:

```json
{
  "args": [
    "tests/test_basic.py::test_asr_basic_functionality",
    "-v"
  ]
}
```

***

## Summary

This tutorial is divided into Basic and Advanced parts:

**Basic Part**:

* Quickly create a project
* Implement core functionality
* Pass basic tests
* Complete end-to-end verification

**Advanced Part**:

* Auto-reconnection mechanism
* Optimize finalize
* Standardized logging
* Error reporting
* Audio dump
* Perfect testing

By completing the checklist items one by one, you can develop a production-grade ASR Extension.

<Callout type="success">
  **Happy hacking!** If you encounter any issues, feel free to open an Issue on [TEN Framework GitHub](https://github.com/TEN-framework/ten-framework).
</Callout>


# Change Language (/docs/ten_agent_examples/playground/change_language)

When we talk about the language of the agent, we are referring to following aspects,

* The language in which agent understands the user's audio input. i.e. STT language
* The language in which agent responds to the user. i.e. TTS language

You will need to configure different properties of above extensions depending on your requirements.

## Change STT language for STT + LLM + TTS agents

To change the language in which agent understands the user's audio input, you will need to change the properties of the STT extension. You can do this by following the steps below,

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select "voice\_assistant" graph type.
3. Click on the Button to the right of the graph selection to open the properties configuration.
4. Choose the "STT" extension from the dropdown list.
5. Change the language property to the desired language.
6. Click on `Save Change` to apply the language to the STT extension.

> Note: The language property of the STT extension differs based on the STT service provider. Please refer to the documentation of the STT service provider for the list of supported options.

## Change STT language for STT + LLM + TTS agents (Use RTC Integrated Azure STT)

To change the language in which agent understands the user's audio input, you will need to change the properties of the RTC extension, as STT extension is integrated within it. You can do this by following the steps below,

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select "voice\_assistant\_stt\_integrated" graph type.
3. Click on the Button to the right of the graph selection to open the properties configuration.
4. Choose the "RTC" extension from the dropdown list.
5. Change the language property to the desired language. For Azure, 'en-US' stands for English. You can find more language options in the Azure documentation.
6. Click on `Save Change` to apply the language to the RTC extension.

## Change language that agent speaks

> Note: The language TTS support usually are determined by the voice of the TTS. Some TTS support multilingual voices, while some support only a single language.

To change the language in which agent responds to the user, you will need to change the properties of the TTS extension. You can do this by following the steps below,

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select "voice\_assistant" graph type.
3. Click on the Button to the right of the graph selection to open the properties configuration.
4. Choose the "TTS" extension from the dropdown list.
5. Change the voice property to the desired voice of the language.
6. Click on `Save Change` to apply the language to the TTS extension.

> Note: The voice property of the TTS extension differs based on the TTS service provider. Please refer to the documentation of the TTS service provider for the list of supported options.


# Configure Modules (/docs/ten_agent_examples/playground/configure_modules)

This guide will help you to configure modules in the TEN-Agent Playground.

## Configure Modules

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select a graph type (e.g. Voice Agent, Realtime Agent).
3. Click on the Button to the right of the graph selection to open the module selection.
4. Depending on the graph type, you can select available modules from the dropdown list.
5. Click on `Save Change` to apply the module to the graph.
6. If you see the success toast, the module is successfully applied to the graph.

## Available Modules

The following module types are available for the TEN-Agent Playground:

### Speech Recognition (STT)

The Speech Recognition module converts spoken language into text.

### Text-to-Speech (TTS)

The Text-to-Speech module converts text into spoken language.

### Large Language Model (LLM)

The Large Language Model module generates text based on the input text with influence.

### Voide to Voice Model (V2V)

The Voice to Voice Model module generates voice based on the input voice with influence.

### Tool (TOOL)

The Tool module provides a set of tools for the agent to use. The tools can be binded to `LLM` module or `V2V` module.

## Bind Tool Modules

You can bind tool modules to `LLM` or `V2V` modules in the TEN-Agent Playground.
Tool can provide additional capabilities to the agent, such as weather check, news update, etc. You can also write your own tool extension if needed.

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select a graph type (e.g. Voice Agent, Realtime Agent).
3. Click on the Button to the right of the graph selection to open the module selection.
4. Depending on the graph type, you will see `LLM` or `V2V` module in the Module Picker.
5. Click on the button to the right of the module to open the tool selection.
6. Select a tool available to bind to the module.
7. Click on `Save Change` to apply the tool to the module.
8. If you see the success toast, the tool is successfully applied to the module.


# Configure Properties (/docs/ten_agent_examples/playground/configure_properties)

This guide will help you to configure properties in the TEN-Agent Playground.

## Configure Properties

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select a graph type (e.g. Voice Agent, Realtime Agent).
3. Click on the Button to the right of the graph selection to open the property configuration.
4. From the dropdown list, you can find all extension nodes used in the graph.
5. Choose an extension node to configure its properties. e.g. stt / llm / tts / v2v / tool.
6. You will see a list of properties that can be configured for the selected extension node.
7. You can change the value of the property by clicking on the input field or switch (if it's a boolean)
8. Click on `Save Change` to apply the property to the extension node.
9. If you see the success toast, the property is successfully applied to the extension node.

## Add more properties

Some properties are defined in the extension node, while does not currently has a value. These properties will not be shown in the property configuration. You can add more properties by following the steps below:

1. In the propety list drawer, click on the `Add Property` button.
2. A new drawer will be opened with a dropdown list of available properties.
3. Select a property from the dropdown list.
4. Click on `Add` to add the property to the extension node.
5. Configure the value of the property.
6. Click on `Save Change` to apply the property to the extension node.
7. If you see the success toast, the property is successfully applied to the extension node.


# Run Playground (/docs/ten_agent_examples/playground)

This guide will help you to run the TEN-Agent Playground. The playground is a web-based interface that allows you to interact with the TEN-Agent. You can replace modules, change configurations, and test the agent's behavior in a controlled environment.

## Run playground from pre-built Docker image

The easiest way to run the playground is to use the pre-built Docker image. The image contains the latest version of the TEN-Agent and the playground. The project docker compose file already contains the pre-built image, so you can start by following the steps in [getting started guide](./getting_started).

Once finished, you can access the playground by opening the browser and navigating to `http://localhost:3000`.

## Run playground from source code

If you want to run the playground from the source code, start a new terminal, change to the folder where you cloned TEN-Agent project, follow below steps:

```bash
# Change to the playground directory
cd playground

# Install dependencies
pnpm install

# Start the playground
pnpm dev
```

Once the playground is started, you can access it by opening the browser and navigating to `http://localhost:3001`.

> **Note:** The playground depends on the golang web server (located in `server` directory) and ten-dev-server (provided by ten framework cli). The ten-dev-server is by default provided when you start with `docker compose up -d` command. The web server will start when you run `task run` command in the container after you follow the steps in the [getting started guide](./getting_started).


# Understand extension folder (/docs/ten_agent_examples/project_structure/extension)

The `ten_packages/extension` folder contains the extension modules. Each extension module is a separate Python/Golang/C++ package.

The extension folder name is often the same as the extension module name, while you can also find the extension module name in the `manifest.json` file under the extension folder. The module name shall be used in the `addon` property of the `property.json` file.

Below is a sample structure of the extension folder:

![Extension Folder Structure](https://ten-framework-assets.s3.amazonaws.com/doc-assets/extension_folder_struct.png?raw=true)

## Extension Common Files

* manifest.json: This file contains the metadata of the extension. It includes the extension name, version, properties, and apis (data, audio\_frame, video\_frame).
* property.json: This file contains the default properties of the extension. It is used to define the default configuration of the extension.

## Python Extension

* extension.py: This file contains the main logic of the extension. It usually contains the core implementation of the extension.
* requirements.txt: This file contains the Python dependencies required by the extension. Dependencies specified in this file will be installed automatically when you run `task use`.

## Golang Extension

* \<xxx>\_extension.go: This file contains the main logic of the extension. It usually contains the core implementation of the extension.
* go.mod: This file contains the Go module definition. It specifies the module name and the dependencies of the extension.

## C++ Extension

* src/main.cc: This file contains the main logic of the extension. It usually contains the core implementation of the extension.
* BUILD.gn: This file contains the build configuration of the extension. It specifies the target name and dependencies of the extension.


# Project Structure (/docs/ten_agent_examples/project_structure)

import { File, Folder, Files } from 'fumadocs-ui/components/files'
import { Callout } from 'fumadocs-ui/components/callout'

The TEN Agent project is built on top of the TEN Framework. For basic concepts, see the [TEN Framework Overview](/docs/ten_framework/concept_overview).

## Directory Structure

<Files>
  <Folder name="TEN-Agent" defaultOpen>
    <Folder name="agents" defaultOpen>
      <Folder name="examples" defaultOpen>
        <Folder name="voice-assistant">
          <File name="property.json" />

          <File name="manifest.json" />
        </Folder>

        <Folder name="realtime-agent" />

        <Folder name="story-teller" />
      </Folder>

      <Folder name="ten_packages" defaultOpen>
        <Folder name="extension" defaultOpen>
          <Folder name="openai_chatgpt" />

          <Folder name="elevenlabs_tts" />

          <Folder name="deepgram_asr" />

          <Folder name="agora_rtc" />
        </Folder>
      </Folder>
    </Folder>

    <Folder name="server">
      <File name="main.go" />
    </Folder>

    <Folder name="playground">
      <File name="package.json" />
    </Folder>

    <Folder name="demo" />

    <File name=".env" />

    <File name="docker-compose.yml" />
  </Folder>
</Files>

## Key Components

### Configuration Files

<Callout type="info" title="property.json">
  The main runtime configuration file that defines how extensions are orchestrated. This is where you configure the graph of extensions and their connections.
</Callout>

Located in each agent example (e.g., `agents/examples/voice-assistant/property.json`), this file:

* Defines which extensions to load
* Specifies connections between extensions
* Sets extension-specific properties and API keys

See [property.json Reference](/docs/ten_agent_examples/project_structure/property_json) for detailed documentation.

### Extensions Directory

```
agents/ten_packages/extension/
```

Contains all available extension modules. Each extension is a separate package that can be written in:

* **Python** - For AI/ML integrations (LLMs, TTS, ASR)
* **Go** - For high-performance networking
* **C++** - For low-latency audio/video processing

Extensions are modular and reusable across different agent configurations.

See [Extension Structure](/docs/ten_agent_examples/project_structure/extension) for details on creating extensions.

### Server

```
server/
```

The backend web server responsible for:

* Handling HTTP/WebSocket requests
* Managing agent process lifecycle (start/stop)
* Routing requests to the appropriate agent instance

See [Server Architecture](/docs/ten_agent_examples/project_structure/server) for implementation details.

### Playground

```
playground/
```

A web-based UI for interacting with your agent:

* Real-time voice/video communication
* Extension configuration
* Debug and monitoring tools

Access at [http://localhost:3000](http://localhost:3000) after starting the agent.

### Demo

```
demo/
```

Production-ready deployment configurations showcasing:

* Example agent setups
* Deployment best practices
* Integration patterns

## Agent Examples

The `agents/examples/` directory contains ready-to-use agent configurations:

| Example           | Description                                     |
| ----------------- | ----------------------------------------------- |
| `voice-assistant` | Basic voice agent with STT → LLM → TTS pipeline |
| `realtime-agent`  | OpenAI Realtime API integration                 |
| `story-teller`    | Interactive storytelling agent                  |

To use an example:

```bash title="Terminal"
cd agents/examples/voice-assistant
task install
task run
```

## Next Steps

* [property.json Reference](/docs/ten_agent_examples/project_structure/property_json) - Understand the configuration format
* [Extension Structure](/docs/ten_agent_examples/project_structure/extension) - Learn how extensions are organized
* [Server Architecture](/docs/ten_agent_examples/project_structure/server) - Dive into the backend implementation


# Understand property.json (/docs/ten_agent_examples/project_structure/property_json)

This file contains the orchestration of extensions. It is the main runtime configuration file.

![Graph at a glance](https://ten-framework-assets.s3.amazonaws.com/doc-assets/graph_at_a_glance.png?raw=true)

The `property.json` file contains the following orchestration info,

* Graphs
* Nodes
* Connections

## Graphs

The `predefined_graphs` property contains a list of available graphs. Each graph defines how agent should behave in a specific scenario. Each graph has a `name` property which is used to identify a graph.

Each graph contains `nodes` and `connections` properties.

## Nodes

The `nodes` section contains the list of extensions that are part of the graph. Each node has a `name` property which is used to identify a node, and a `addon` property which is used to specify the extension module it will use. You can have multiple nodes of the same extension type in a graph. e.g. you can have multiple `chatgpt_openai_python` extensions with same `addon` property, while each node will have a different `name` property. It's working like multiple instances of the same extension.

#### Node Property

The `property` section of a node contains the configuration of the extension. It is extension specific and available properties are defined in the `manifest.json` file under extension folder. You can use define runtime properties of an extension to customize its behavior. Below is an example of how to configure `chatgpt_openai_python` extension,

```json
{
  "name": "chatgpt_openai_python",
  "addon": "chatgpt_openai_python",
  "property": {
    "api_key": "${env:OPENAI_API_KEY|}",
    "model": "gpt-3.5-turbo",
    "temperature": 0.5,
    "max_tokens": 100,
    "prompt": "You are a helpful assistant"
  }
}
```

![Property JSON nodes](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_json_connections.png?raw=true)

#### Read environment variables

It's quite common that an extension will require `api_key` to work. If you don't want to hardcode the `api_key` in the `property.json` file, you can use environment variables. You can use `${env:<env_var_name>|<default_value>}` syntax to read environment variables. Below is an example of how to read `OPENAI_API_KEY` environment variable,

```json
{
  "name": "chatgpt_openai_python",
  "addon": "chatgpt_openai_python",
  "property": {
    "api_key": "${env:OPENAI_API_KEY|}"
  }
}
```

## Connections

The `connections` section contains the list of connections between nodes. Each connection specifies the source and destination nodes respectively. Every connection has an `extension_group` and `extension` property which is used to specify the source node, and it defines a set of multimodal data protocols(audio\_frame, video\_frame, data, cmd) supported by TEN Framwork. For each data protocols, it has a list of destination definitions, and each destination definition has a `name` property which is the key of the property data, and `dest` property which defines a list of destination nodes. Below is an example of how to connect two nodes,

```json
{
  "extension_group": "default",
  "extension": "agora_rtc",
  "audio_frame": [
    {
      "name": "pcm_frame",
      "dest": [
        {
          "extension_group": "default",
          "extension": "deepgram_asr"
        }
      ]
    }
  ]
}
```

![Property JSON Connections](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_json_nodes.png?raw=true)

In the above example, we are connecting `agora_rtc` extension to `deepgram_asr` extension. The `agora_rtc` extension is sending `pcm_frame` data to `deepgram_asr` extension.


# Understand web server (/docs/ten_agent_examples/project_structure/server)

The web server is a simple Golang web server that serves the http requests. It is responsible for handling the incoming requests and starting/stopping agent processes.

The playground/demo UI has been built to interact with the web server. You can also interact with the web server using curl or any other http client.

## Request & Response Examples

The server provides a simple layer for managing agent processes. The API resources are described below.

### API Resources

* [POST /start](#post-start)
* [POST /stop](#post-stop)
* [POST /ping](#post-ping)

### POST /start

This api starts an agent with given graph and override properties. The started agent will join into the specified channel, and subscribe to the uid which your browser/device's rtc use to join.

| Param         | Description                                                                                                                                                                                                                                                                                               |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| request\_id   | any uuid for tracing purpose                                                                                                                                                                                                                                                                              |
| channel\_name | channel name, it needs to be the same with the one your browser/device joins, agent needs to stay with your browser/device in the same channel to communicate                                                                                                                                             |
| user\_uid     | the uid which your browser/device's rtc use to join, agent needs to know your rtc uid to subscribe your audio                                                                                                                                                                                             |
| bot\_uid      | optional, the uid bot used to join rtc                                                                                                                                                                                                                                                                    |
| graph\_name   | the graph to be used when starting agent, will find in property.json                                                                                                                                                                                                                                      |
| properties    | additional properties to override in property.json, the override will not change original property.json, only the one agent used to start                                                                                                                                                                 |
| timeout       | determines how long the agent will remain active without receiving any pings. If the timeout is set to `-1`, the agent will not terminate due to inactivity. By default, the timeout is set to 60 seconds, but this can be adjusted using the `WORKER_QUIT_TIMEOUT_SECONDS` variable in your `.env` file. |

Example:

```bash
curl 'http://localhost:8080/start' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "request_id": "c1912182-924c-4d15-a8bb-85063343077c",
    "channel_name": "test",
    "user_uid": 176573,
    "graph_name": "camera_va_openai_azure",
    "properties": {
      "openai_chatgpt": {
        "model": "gpt-4o"
      }
    }
  }'
```

### POST /stop

This api stops the agent you started

| Param         | Description                                       |
| ------------- | ------------------------------------------------- |
| request\_id   | any uuid for tracing purpose                      |
| channel\_name | channel name, the one you used to start the agent |

Example:

```bash
curl 'http://localhost:8080/stop' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "request_id": "c1912182-924c-4d15-a8bb-85063343077c",
    "channel_name": "test"
  }'
```

### POST /ping

This api sends a ping to the server to indicate connection is still alive. This is not needed if you specify `timeout:-1` when starting the agent, otherwise the agent will quit if not receiving ping after timeout in seconds.

| Param         | Description                                       |
| ------------- | ------------------------------------------------- |
| request\_id   | any uuid for tracing purpose                      |
| channel\_name | channel name, the one you used to start the agent |

Example:

```bash
curl 'http://localhost:8080/ping' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "request_id": "c1912182-924c-4d15-a8bb-85063343077c",
    "channel_name": "test"
  }'
```


# Setup Development Environment (/docs/ten_agent_examples/setup_development_env)

The easiest way to get started with TEN Agent is to use the provided Docker container. This container includes all the necessary dependencies and configurations to run the agent. You can also customize the agent by creating your own extensions or modifying the existing ones.

While if you want to develop the agent, using suitable development tools can save you lots of time. This guide will contain following tutorials,

* Setup local development environment with VSCode to connect with your Docker container.
* Setup cloud development environment with Github Codespace.


# Cloud Development in Codespace (/docs/ten_agent_examples/setup_development_env/setting_up_development_inside_codespace)

Github Codespace is a cloud-based development environment that allows you to develop your code directly in the cloud. It's a great way to develop your agent without having to set up a local development environment. You don't need to have a great network or powerful machine. This guide will show you how to set up a development environment for TEN Agent using Github CodeSpace.

## Step 1: Create a Codespace with TEN-Agent repository

Navigate to TEN-Agent github repository, and click on the `Code` button. Then, select `Open with Codespaces`. Github will then create a Codespace for you in a new browser tab. The first time you create a Codespace, it may take a few minutes to set up.

![Open with Codespaces](https://ten-framework-assets.s3.amazonaws.com/doc-assets/start_codespace.png?raw=true)

## Step 2: Start working with Codespace

Once the Codespace is ready, you will see the VSCode editor in the browser. You can start working with the TEN-Agent repository in the Codespace. You can open the terminal, create new files, and run commands just like you would in a local development environment.

The steps are similar to the process outlined in the [Quick Start](./getting_started) guide. However, there's no need to run `docker compose up` command with Codespace. You can directly start building agent using,

```bash
# Prepare .env file from .env.example and set your api keys inside it
cp ./.env.example .env
# Build the agent after finish key configurations
task use
```

And once successfully built, you can run following command to start server,

```bash
task run
```

## Step 3: Start Front-end UI

As we are not using docker compose with Codespace, you will need to start the front-end UI separately. You can do this by running the following command in the terminal,

```bash
cd playground
pnpm install
pnpm dev
```

This will start the front-end UI at `localhost:3000`.

## Step 4: Access your remote setup

This is a very special step which applies to Codespace only. Codespace will automatically generate a URL for every port you exposed in your Codespace. You can find the URL by clicking on the `Ports` tab in the bottom left corner of the Codespace window.

![Ports](https://ten-framework-assets.s3.amazonaws.com/doc-assets/codespace_ports.png?raw=true)

All ports visibility are by default set to `private`, you can change it to `public` by right click and choose `port visibility`. Once you make it public, you can access the URL in your browser to see the front-end UI. Here we will right click on the visibility column of 3000 port, and change its visibility to `public`.

![Change Visibility](https://ten-framework-assets.s3.amazonaws.com/doc-assets/codespace_visibility.png?raw=true)

Now you can access the front-end UI at the URL generated by Codespace. Find the `Forwarded Address` column for port 3000, click on that and browser should start a new tab for you with your front-end UI.

![Forwarded Address](https://ten-framework-assets.s3.amazonaws.com/doc-assets/codespace_forwarded_addr.png?raw=true)

And that's it! You have successfully set up a development environment for TEN Agent using Github Codespace. You can now start developing your agent directly in the cloud. For debugging and development experience it's basically the same as your local VSCode experience.


# VSCode Dev Container (/docs/ten_agent_examples/setup_development_env/setting_up_vscode_for_development_inside_container)

When developing with TEN, it's generally recommended to perform compilation and development within a container. However, if you're using VSCode outside the container, you may encounter issues where symbols cannot be resolved. This is because some environment dependencies are installed within the container, and VSCode can't recognize the container's environment, leading to unresolved header files.

To solve this, you can mount VSCode within the container so that it recognizes the container’s environment and resolves the header files accordingly. This guide will walk you through using VSCode's Dev Containers and Docker extensions to achieve this.

## Step 1: Install the Docker Extension

First, install the [Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker) in VSCode. This extension allows you to manage Docker containers directly within VSCode.

## Step 2: Install the Dev Containers Extension

Next, install the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). This extension enables VSCode to connect to Docker containers for development.

## Step 3: Start the Development Environment Using Docker Compose

This step is similar to the process outlined in the [Quick Start](./getting_started) guide. However, instead of running:

\{% code title=">\_ Terminal" %}

```shell
docker compose up
```

\{% endcode %}

Using the `docker compose up -d` command, start the container in detached mode:

\{% code title=">\_ Terminal" %}

```shell
docker compose up -d
```

\{% endcode %}

After executing this command, the container should start. Open VSCode, switch to the Docker extension, and you should see the running container.

![Docker Containers](https://ten-framework-assets.s3.amazonaws.com/doc-assets/docker_containers.png)

## Step 4: Connect to the Container

In the Docker extension within VSCode, find the `ten_agent_dev` container in the list and click `Attach Visual Studio Code` to connect to the container. VSCode will then open a new window that is connected to the container, where you can proceed with development.

In the Dev Container environment connected to the container, your local extensions and settings will not be applied, as this environment is within the container. Therefore, you will need to install extensions and configure settings inside the container. To install extensions within the container, open the newly launched VSCode window, click on `Extensions` in the left sidebar, search for the required extension, and follow the prompts to install it inside the container.

## Step 5: Setup breakpoint for debugging

Setting breakpoints in the code is a common practice when debugging. To set a breakpoint in the code, click on the left margin of the line number where you want to set the breakpoint. A red dot will appear, indicating that a breakpoint has been set.

![Setting Breakpoint](https://ten-framework-assets.s3.amazonaws.com/doc-assets/setting_breakpoint.png)

if you cannot set the breakpoint, it usually means you have not installed the language extension in the container. You can install the language extension by clicking on the `Extensions` icon in the left sidebar, searching for the required extension, and following the prompts to install it.

Once you have set the breakpoint, you can start debugging by clicking on the `Run and Debug` icon in the left sidebar, selecting the `debug python` configuration, and clicking on the green play button to start debugging.

![Debug Configuration](https://ten-framework-assets.s3.amazonaws.com/doc-assets/debug_config.png?raw=true)

In this way, VSCode is directly starting the agent application, which means Golang web server is not paticipating in the run. Therefore, you will need to pay attention to below points:

### Which graph is being used in this mode?

Web server will help you manipulate `property.json` when starting agent to help you select the graph you want to use. However, in this mode, you will need to manually modify `property.json` to select the graph you want to use. Ten will by default select **the first graph with `auto_start` property set to true** to start.

### RTC properties

RTC feature is mandatory for your client to talk to agent server. Web server will help you generate RTC tokens and channels and update these into `property.json` for you. However, in this mode, you will need to manually modify `property.json` to update RTC tokens and channels yourself. The easiest way is to run with playground/demo first, and a temp property file will be generated, you can copy the RTC tokens and channels from there.

You shall find the temp property file path in the ping request logs like below,

```shell
2024/11/23 08:39:19 INFO handlerPing start channelName=agora_74np6e requestId=218f8e36-f4d0-4c83-a6e8-d3b4dec2a187 service=HTTP_SERVER
2024/11/23 08:39:19 INFO handlerPing end worker="&{ChannelName:agora_74np6e HttpServerPort:10002 LogFile:/tmp/astra/app-agora_74np6e-20241123_083855_000.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-agora_74np6e-20241123_083855_000.json Pid:5330 QuitTimeoutSeconds:60 CreateTs:1732351135 UpdateTs:1732351159}" requestId=218f8e36-f4d0-4c83-a6e8-d3b4dec2a187 service=HTTP_SERVER
```

The temp property file path is shown as `PropertyJsonFile:/tmp/astra/property-agora_74np6e-20241123_083855_000.json`.


# How does Interruption Work (/docs/ten_agent_examples/tutorials/how_does_interrupt_work)

## Overview

The interrupt mechanism in
[TEN-Agent](https://github.com/TEN-framework/TEN-Agent) consists of two main
parts: **Interrupt Detection** and **Interrupt Response**. This document details
both parts and explains how the interrupt command propagates through the AI
agent graph.

## Part 1: Interrupt Detection

<div className="fd-steps">
  <div className="fd-step">
    ### Current Interrupt Detection Implementation

    The current
    ([interrupt\_detector\_python](https://github.com/TEN-framework/TEN-Agent/tree/main/agents/ten_packages/extension/interrupt_detector_python))
    extension implements a text-based interrupt detection mechanism:

    ```python
    def on_data(self, ten: TenEnv, data: Data) -> None:
        text = data.get_property_string(TEXT_DATA_TEXT_FIELD)
        final = data.get_property_bool(TEXT_DATA_FINAL_FIELD)

        # Trigger interrupt when text is final or reaches threshold length
        if final or len(text) >= 2:
            self.send_flush_cmd(ten)
    ```

    The interrupt detector triggers in the following cases:

    1. When receiving final text (`is_final = true`)
    2. When text length reaches a threshold (≥ 2 characters)
  </div>

  <div className="fd-step">
    ### Customize Interrupt Detection

    To implement your own interrupt detection logic, you can refer to the
    implementation of
    [interrupt\_detector\_python](https://github.com/TEN-framework/TEN-Agent/tree/main/agents/ten_packages/extension/interrupt_detector_python)
    as an example and customize the interrupt conditions based on your specific
    needs.
  </div>
</div>

## Part 2: Interrupt Response

### Chain Processing in AI Agent Graph

In a typical AI agent graph, the interrupt command (`flush`) follows a chain
processing pattern:

```text
Interrupt Detector
       ↓
    LLM/ChatGPT
       ↓
      TTS
       ↓
   agora_rtc
```

Each extension in the chain follows two key steps when receiving a `flush`
command:

1. Clean up its own resources and internal state
2. Forward the `flush` command to downstream extensions

This ensures that:

* Extensions are cleaned up in the correct order
* No residual data flows through the system
* Each extension returns to a clean state before the next operation

## Conclusion

[TEN-Agent](https://github.com/TEN-framework/TEN-Agent)'s interrupt mechanism
uses a chain processing pattern to ensure orderly cleanup of all extensions in
the AI agent graph. When an interrupt occurs, each extension first cleans up its
own state and then forwards the `flush` command to downstream extensions,
ensuring a clean system state for subsequent operations.


# How to Use a Local LLM with Ollama (/docs/ten_agent_examples/tutorials/how_to_run_local_model_in_ollama)

Ten Agent enables integration with various LLMs that support the OpenAI request-response format. This guide outlines how to use Ollama models locally with Ten Agent by leveraging OpenAI-compatible endpoints.

## Step 1: Verify Hardware Requirements

Ensure your hardware meets the necessary requirements before running an AI model locally. Key components to check include:

* **CPU/GPU**: Verify the processing power required by the model.
* **Memory**: Confirm sufficient memory is available to load and run the model.

This ensures smooth operation of the model on your system.

## Step 2: Install Required Software and Dependencies

Prepare your environment by installing the necessary software and dependencies:

1. **Install Ollama**: Download and install Ollama on your local machine. Refer to [Ollama](https://ollama.com/).
2. **Download the Model**: Use the command `ollama pull {model name}` to download the desired LLM model. For example, `ollama pull deepseek-r1`
3. **Set Up Ten Agent**: Ensure Ten Agent Dev and Ten Agent Playground are properly configured. Refer to [Ten Agent: Getting Started](../getting_started)

## Step 3: Start the Ollama Server

After setting up Ollama and downloading the required models, start the Ollama server by running the following command in your terminal:

```bash
OLLAMA_HOST=0.0.0.0 ollama serve
```

This launches the Ollama server on your local machine. Since the Ten Agent runs in a Docker container, you’ll need your private IP address to access the Ollama server. Retrieve your private IP for the LLM endpoint.

## Step 4: Configure Ten Agent

Integrate Ollama with Ten Agent using its OpenAI API compatibility:

1. Open Ten Agent Playground and navigate to the **Voice Assistance Graph (voice\_assistant)**.
2. Select **OpenAI** as the LLM.
3. Set the Ollama API endpoint as the LLM `base_url`: `http://<your-local-machine-private-ip>:11434/v1/`.

Replace `<your-local-machine-private-ip>` with the private IP address of your local machine.

4. Provide a `prompt`, e.g., "You're a helpful assistant."
5. Specify the `model` name used in Ollama (e.g., `deepseek-r1`).

## Step 5: Test the Integration

Once configured, Ten Agent will interact with Ollama using the same flow as OpenAI’s API. Test queries to ensure proper functionality.

## Additional Resources

For more information on Ollama’s OpenAI compatibility, refer to their official blog post: [Ollama OpenAI Compatibility](https://ollama.com/blog/openai-compatibility).

## Summary

Integrating a local model with Ten Agent is straightforward, thanks to its support for the OpenAI API format. The same process can be applied to connect with cloud-hosted LLMs or any LLMs supporting OpenAI API compatibility.


# How to run local AI model in Python extension (/docs/ten_framework/ai_model/how_to_run_local_model_in_python_extensions)

In the TEN framework, extensions can utilize third-party AI services or run AI models locally to improve performance and reduce costs. This tutorial explains how to run a local AI model in a Python extension and how to interact with it within the extension.

## Step 1: Check Hardware Requirements

Before running an AI model locally, ensure that your hardware meets the necessary requirements. Key components to verify include:

* **CPU/GPU**: Check if the model requires specific processing power.
* **Memory**: Ensure sufficient memory to load and run the model.

Verify that your system can support the model’s demands to ensure smooth operation.

## Step 2: Install Necessary Software and Dependencies

Once your hardware is ready, install the required software and dependencies. Follow these steps:

1. **Operating System**: Ensure compatibility with your model. Most AI frameworks support Windows, macOS, and Linux, though specific versions may be required.

2. **Python Version**: Ensure compatibility with the TEN Python runtime and the model.

3. **Required Libraries**: Install necessary libraries such as:

   * TensorFlow
   * PyTorch
   * Numpy
   * vllm

   You can list the required dependencies in a `requirements.txt` file for easy installation.

4. **Download the Model**: Obtain the local version of the AI model you plan to run.

## Step 3: Implement Your Python Extension

Below is an example of how to implement a basic text generation feature using the `vllm` inference engine in a Python extension.

First, initialize the local model within the extension:

```python
from ten_runtime import (
    Extension,
    TenEnv,
    Cmd,
    CmdResult,
)
from vllm import LLM

class TextGenerationExtension(Extension):
    def on_init(self, ten_env: TenEnv) -> None:
        self.llm = LLM(model="<model_path>")
        ten_env.on_init_done()
```

Next, implement the `on_cmd` method to handle text generation based on the provided input:

```python
    def on_cmd(self, ten_env: TenEnv, cmd: Cmd) -> None:
        prompt = cmd.get_property_string("prompt")

        outputs = self.llm.generate(prompt)
        generated_text = outputs[0].outputs[0].text

        cmd_result = CmdResult.create(StatusCode.OK)
        cmd_result.set_property_string("result", generated_text)
        ten_env.return_result(cmd_result, cmd)
```

In this code, the `on_cmd` method retrieves the `prompt`, generates text using the model, and returns the generated text as the command result.

You can adapt this approach to implement other functionalities such as image recognition or speech-to-text by processing the relevant input types.

## Step 4: Unload the Model

It’s important to unload the model during extension cleanup to free resources:

```python
import gc
import torch

class TextGenerationExtension(Extension):
    ...

    def on_deinit(self, ten_env: TenEnv) -> None:
        del self.llm
        gc.collect()
        torch.cuda.empty_cache()
        torch.distributed.destroy_process_group()
        print("Successfully deleted the LLM pipeline and freed GPU memory!")
        ten_env.on_deinit_done()
```

This ensures efficient memory management, especially when working with GPU resources.

## Summary

Running a local model in a TEN Python extension is similar to native Python development. By loading and unloading the model in the appropriate extension lifecycle methods, you can easily integrate local AI models and interact with them effectively.


# API (/docs/ten_framework/api/api)

In the TEN Framework, the API definition of an extension is contained in the `manifest.json` file. When defining APIs, the TEN Framework does not use JSON Schema, but instead uses a JSON-like syntax to define API Schema.

## Why Not Use JSON Schema

The TEN Framework chooses not to use standard JSON Schema for the following reasons:

* **Type System Differences**: The TEN Framework's type system includes many types that JSON Schema does not have, such as `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float32`, `float64`, `buf`, `ptr`, etc.

* **Reduce Redundancy**: In the TEN Framework's API definitions, many places do not need flexible settings. For example, a message's schema must be an object type and cannot be other types. Requiring developers to explicitly write `"type": "object"` would be redundant and confusing.

Based on these reasons, the TEN Framework adopts a more natural definition approach, using JSON-like syntax to define API Schema.

## Top-level API Definition

The API Schema definition is placed in the `api` field of the `manifest.json` file:

```json
{
  "api": {
    "property": {
      "properties": {...},
      "required": [...]
    },
    "cmd_in": [{
      "name": "foo",
      "property": {
        "properties": {...},
        "required": [...]
      },
      "result": {
        "property": {
          "properties": {...},
          "required": [...]
        }
      }
    }]
  }
}
```

## API Schema Definition Patterns

### Expressing Object Type Fields

Use the `properties` field to describe the type of each property within an object type field:

```json
{
  "my_object": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "age": {
        "type": "int32"
      }
    }
  }
}
```

### Expressing Required and Optional Fields

Use a `required` field at the same level as the object type field to describe required and optional fields:

```json
{
  "some_concept": {
    "properties": {
      "foo": {},
      "bar": {}
    },
    "required": ["foo"]
  }
}
```

### Expressing Array Type Fields

Use the `items` field to describe the type of each element within an array type field:

```json
{
  "my_array": {
    "type": "array",
    "items": {
      "type": "string"
    }
  }
}
```

### Usage Principles for type Field

If a field can only be a specific type (constrained by the TEN Framework itself), there is no need to explicitly write the `type` field. If a field can be multiple types, then the `type` field needs to be explicitly written.

For example, if the `some_concept` field below can only be an object type, there is no need to explicitly write the `type` field. However, the `foo` field within it can be multiple types, so the `type` field needs to be explicitly written:

```json
{
  "some_concept": {
    "properties": {
      "foo": {
        "type": "string"
      },
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "string"
          }
        },
        "required": ["baz"]
      }
    },
    "required": ["foo"]
  }
}
```

Use the `required` field appropriately to ensure the existence of critical data:

```json
{
  "name": "critical_cmd",
  "property": {
    "properties": {
      "essential_param": {
        "type": "string"
      },
      "optional_param": {
        "type": "int32"
      }
    },
    "required": ["essential_param"]
  }
}
```

## Expressing Enum Type Fields

Use the `enum` field to describe all possible values of an enumeration type:

```json
{
  "api": {
    "property": {
      "properties": {
        "status": {
          "type": "string",
          "enum": ["CREATING", "ACTIVE", "DELETING", "FAILED"]
        }
      }
    }
  }
}
```

## Extension Property Schema

Complete example of defining extension properties:

```json
{
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        },
        "config": {
          "type": "object",
          "properties": {
            "enabled": {
              "type": "bool"
            },
            "settings": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          },
          "required": ["enabled"]
        }
      },
      "required": ["api_key", "max_tokens", "temperature", "config"]
    }
  }
}
```

## Command-like Message Schema

Commands support input (`cmd_in`) and output (`cmd_out`):

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "process_text",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            },
            "options": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "max_length": {
                  "type": "int32"
                }
              }
            }
          },
          "required": ["text"]
        },
        "result": {
          "property": {
            "properties": {
              "processed_text": {
                "type": "string"
              },
              "confidence": {
                "type": "float32"
              }
            },
            "required": ["processed_text"]
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "notify_completion",
        "property": {
          "properties": {
            "task_id": {
              "type": "string"
            },
            "status": {
              "type": "string"
            }
          },
          "required": ["task_id", "status"]
        }
      }
    ]
  }
}
```

## Data-like Message Schema

Schema definitions for data-type messages:

```json
{
  "api": {
    "data_in": [
      {
        "name": "input_data",
        "property": {
          "properties": {
            "content": {
              "type": "string"
            },
            "metadata": {
              "type": "object",
              "properties": {
                "timestamp": {
                  "type": "int64"
                }
              }
            }
          },
          "required": ["content"]
        }
      }
    ],
    "data_out": [
      {
        "name": "output_data",
        "property": {
          "properties": {
            "result": {
              "type": "string"
            }
          }
        }
      }
    ],
    "audio_frame_in": [
      {
        "name": "audio_input"
      }
    ],
    "audio_frame_out": [
      {
        "name": "audio_output"
      }
    ],
    "video_frame_in": [
      {
        "name": "video_input"
      }
    ],
    "video_frame_out": [
      {
        "name": "video_output"
      }
    ]
  }
}
```

## Common Errors and Solutions

<div className="fd-steps">
  <div className="fd-step">
    ### Property Names Not Following Conventions

    **Error Example**: Property names containing hyphens or starting with numbers

    ```json
    {
      "api-key": {  // Error: contains hyphens
        "type": "string"
      },
      "2nd_param": {  // Error: starts with a number
        "type": "int32"
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "api_key": {
        "type": "string"
      },
      "second_param": {
        "type": "int32"
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### Array Type Missing items Definition

    **Error Example**:

    ```json
    {
      "my_array": {
        "type": "array"
        // Missing items definition
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "my_array": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### Object Type Missing properties Definition

    **Error Example**:

    ```json
    {
      "my_object": {
        "type": "object"
        // Missing properties definition
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "my_object": {
        "type": "object",
        "properties": {
          "field1": {
            "type": "string"
          }
        }
      }
    }
    ```
  </div>
</div>

## Summary

The TEN Framework's Schema system provides a powerful and structured way to define and validate data structures, ensuring consistency and type safety between components. By following design principles and best practices, developers can create robust and maintainable TEN applications and extensions.

The system supports rich type definitions, complex API descriptions, and flexible message transformation capabilities, providing a solid foundation for building large-scale real-time multimodal applications.


# Log (/docs/ten_framework/api/log)

The TEN framework allows extensions developed in different languages to run within the same process. This creates a need to view logs from all these extensions in a unified log with a consistent format and information, making debugging easier.

To address this, the `ten_env` provides a logging API. Within each callback of an extension, an instance of `ten_env` can be accessed. Using this instance, you can call the logging API to output logs from extensions developed in various languages to a unified log output.

## Log Levels

The TEN framework supports the following log levels in increasing order of severity:

* **DEBUG** (1): Debug information
* **INFO** (2): General information
* **WARN** (3): Warning information
* **ERROR** (4): Error information

## API Interface

### C++ Language

```cpp
#include "ten_runtime/binding/cpp/ten.h"

class Extension {
public:
    void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr<ten::cmd_t> cmd) {
        // Using convenience macros
        TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
        TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
        TEN_ENV_LOG_WARN(ten_env, "Warning message");
        TEN_ENV_LOG_ERROR(ten_env, "Error occurred");

        // Direct method call with category and fields support
        ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
                   "Direct log call", nullptr, nullptr);
    }
};
```

C++ provides the following convenience macros:

* `TEN_ENV_LOG_DEBUG(ten_env, msg)`
* `TEN_ENV_LOG_INFO(ten_env, msg)`
* `TEN_ENV_LOG_WARN(ten_env, msg)`
* `TEN_ENV_LOG_ERROR(ten_env, msg)`

### Python Language

```python
from ten_runtime import Extension, TenEnv

class MyExtension(Extension):
    def on_cmd(self, ten_env: TenEnv, cmd):
        # Using different log level methods
        ten_env.log_debug("Debug information")
        ten_env.log_info("General information")
        ten_env.log_warn("Warning information")
        ten_env.log_error("Error information")

        # Supports formatted strings and additional parameters
        test_value = "example"
        ten_env.log_info(f"Processing command, test value: {test_value}")

        # Supports category and fields parameters
        ten_env.log_info("Log with category", category="my_extension")
```

Python provides the following methods:

* `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`

### Go Language

```go
package main

import "ten_runtime/ten"

type MyExtension struct {
    ten.DefaultExtension
}

func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    // Using different log level methods
    tenEnv.LogDebug("Debug information")
    tenEnv.LogInfo("General information")
    tenEnv.LogWarn("Warning information")
    tenEnv.LogError("Error information")

    // Supports formatting
    cmdName := cmd.GetName()
    tenEnv.LogInfo("Processing command: " + cmdName)

    // Using the full Log method for additional parameters
    category := "my_extension"
    tenEnv.Log(ten.LogLevelInfo, "Log with category", &category, nil, nil)
}
```

Go provides the following methods:

* `LogDebug(msg string) error`
* `LogInfo(msg string) error`
* `LogWarn(msg string) error`
* `LogError(msg string) error`
* `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`

### Node.js/TypeScript Language

```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";

class MyExtension extends Extension {
    async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
        // Using different log level methods
        tenEnv.logDebug("Debug information");
        tenEnv.logInfo("General information");
        tenEnv.logWarn("Warning information");
        tenEnv.logError("Error information");

        // Supports string concatenation
        const cmdName = cmd.getName();
        tenEnv.logInfo("Processing command: " + cmdName);

        // Supports category and fields parameters
        tenEnv.logInfo("Log with category", "my_extension");
    }
}
```

Node.js provides the following methods:

* `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
* `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
* `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
* `logError(message: string, category?: string, fields?: Value): TenError | undefined`

## Usage Examples

### Basic Usage

```python
# Python example
def on_start(self, ten_env: TenEnv):
    ten_env.log_info("Extension starting")
    ten_env.on_start_done()

def on_cmd(self, ten_env: TenEnv, cmd):
    cmd_name = cmd.get_name()
    ten_env.log_debug(f"Received command: {cmd_name}")

    # Command processing logic
    if cmd_name == "hello":
        ten_env.log_info("Processing hello command", category="command_handler")
        # ... processing logic
    else:
        ten_env.log_warn(f"Unknown command: {cmd_name}")
```

### Using in Threads

```python
import threading
import time

def log_routine(self, ten_env: TenEnv):
    """Log messages in a separate thread"""
    while not self.stop_flag:
        self.log_count += 1
        ten_env.log_info(f"Log message {self.log_count}")
        time.sleep(0.05)

def on_start(self, ten_env: TenEnv):
    # Start logging thread
    self.log_thread = threading.Thread(
        target=self.log_routine, args=(ten_env,)
    )
    self.log_thread.start()
    ten_env.on_start_done()
```

### Error Handling

```python
def on_cmd(self, ten_env: TenEnv, cmd):
    try:
        # Business logic
        result = self.process_command(cmd)
        ten_env.log_info("Command processed successfully")
    except ValueError as e:
        ten_env.log_error(f"Parameter error: {str(e)}", category="error_handling")
    except Exception as e:
        ten_env.log_error(f"Fatal error: {str(e)}", category="error_handling")
        raise
```

## Log Format and Output

The TEN framework's logging system automatically includes the following information:

* **Timestamp**: Time when the log was recorded
* **Log Level**: DEBUG, INFO, WARN, ERROR
* **Function Name**: Name of the function that called the log (if available)
* **File Name**: Source file name where the log was called (if available)
* **Line Number**: Code line number where the log was called (if available)
* **Category**: Optional log category information
* **Message Content**: User-provided log message

The output format is similar to:

```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] User log message
```

## Advanced Log Configuration

The TEN framework supports advanced logging functionality through the `property.json` file configuration. This allows you to customize the log output format, destination, and filtering rules.

### Configuration Structure

In the `property.json` file, you can define advanced log configuration in the `ten.log` section:

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "my_extension"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

### Configuration Components

<div className="fd-steps">
  <div className="fd-step">
    #### Log Handlers

    Each handler contains three main components:

    * **matchers**: Define which log messages will be processed by this handler
    * **formatter**: Define the log output format
    * **emitter**: Define the log output destination
  </div>

  <div className="fd-step">
    #### Matchers

    Matchers are used to filter log messages:

    ```json
    {
      "level": "info",      // Log level: "off", "debug", "info", "warn", "error"
      "category": "auth"    // Optional: log category (string)
    }
    ```

    Supported log levels:

    * `"off"`: Disable logging
    * `"debug"`: Debug level
    * `"info"`: Info level
    * `"warn"`: Warning level
    * `"error"`: Error level
  </div>

  <div className="fd-step">
    #### Formatter

    Controls the log output format:

    ```json
    {
      "type": "plain",      // Format type: "plain" or "json"
      "colored": true       // Optional: enable colored output (only for plain format)
    }
    ```

    * **plain**: Human-readable plain text format
    * **json**: JSON structured format, convenient for log analysis tools
  </div>

  <div className="fd-step">
    #### Emitter

    Defines the log output destination:

    ##### Console Output

    ```json
    {
      "type": "console",
      "config": {
        "stream": "stdout",  // "stdout" or "stderr"
        "encryption": {      // Optional: encryption configuration
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### File Output

    ```json
    {
      "type": "file",
      "config": {
        "path": "/path/to/logfile.log",  // Log file path
        "encryption": {                  // Optional: encryption configuration
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### OTLP Output

    OTLP (OpenTelemetry Protocol) output allows you to send logs to OpenTelemetry-compatible backend systems, such as Jaeger, Prometheus, Grafana, and more.

    ```json
    {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4317",  // OTLP receiver endpoint URL
        "protocol": "grpc",                   // Optional: transport protocol, "grpc" (default) or "http"
        "headers": {                          // Optional: custom HTTP headers
          "Authorization": "Bearer token",
          "X-Custom-Header": "value"
        },
        "service_name": "my-service"          // Optional: service name for identifying log source
      }
    }
    ```

    Configuration parameters:

    * **endpoint**: OTLP receiver endpoint URL (required)
    * **protocol**: Transport protocol, supports `"grpc"` or `"http"`, defaults to `"grpc"`
    * **headers**: Custom HTTP request headers for authentication or metadata (optional)
    * **service\_name**: Service name for identifying different services in log aggregation systems (optional)
  </div>
</div>

### Configuration Examples

#### Example 1: Multi-level Log Separation

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/errors.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### Example 2: Category-based Log Separation

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "database"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/database.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info",
              "category": "api"
            }
          ],
          "formatter": {
            "type": "plain"
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### Example 3: Encrypted Log Output

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/secure/logs/app.log",
              "encryption": {
                "enabled": true,
                "algorithm": "AES-CTR",
                "params": {
                  "key": "your-32-character-encryption-key",
                  "nonce": "your-16-character-nonce"
                }
              }
            }
          }
        }
      ]
    }
  }
}
```

#### Example 4: OTLP Log Output to OpenTelemetry Collector

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "http://localhost:4317",
              "protocol": "grpc",
              "service_name": "ten-framework-app"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "https://api.monitoring.example.com/v1/logs",
              "protocol": "http",
              "headers": {
                "Authorization": "Bearer your-api-token",
                "X-Environment": "production"
              },
              "service_name": "ten-framework-app"
            }
          }
        }
      ]
    }
  }
}
```

### Configuration Notes

1. **Multiple Handlers**: You can define multiple handlers to implement complex log routing strategies
2. **Matching Priority**: Handlers are matched in the order they appear in the configuration file
3. **Encryption Security**: Encryption keys and nonces should be stored securely, not hardcoded in configuration files
4. **File Permissions**: Ensure log file paths have appropriate write permissions
5. **Performance Considerations**: JSON format may produce larger log files than plain format
6. **OTLP Endpoints**:
   * When using gRPC protocol, the default port is typically 4317
   * When using HTTP protocol, the default port is typically 4318
   * Ensure the OTLP collector endpoint is accessible and properly configured
7. **Authentication**: When using OTLP, if the backend requires authentication, add appropriate credentials in the headers

## Best Practices

1. **Choose appropriate log levels**:
   * Use `DEBUG` for detailed debugging information
   * Use `INFO` for normal business processes
   * Use `WARN` for potential issues
   * Use `ERROR` for error information

2. **Provide meaningful log messages**:
   * Include contextual information (such as command names, parameter values, etc.)
   * Use descriptive messages rather than simple identifiers
   * Use the category parameter appropriately to classify logs

3. **Avoid excessive logging**:
   * Don't use too many logs in high-frequency loops
   * Appropriately adjust log levels in production environments
   * Use advanced log configuration matchers to filter unnecessary logs

4. **Thread safety**:
   * The TEN framework's logging API is thread-safe
   * Log methods can be safely called from any thread

5. **Advanced configuration recommendations**:
   * Use different categories for different modules to facilitate log analysis
   * Consider using file output in production environments for log persistence
   * Use encryption functionality for sensitive logs
   * Regularly rotate log files to control disk usage
   * Use JSON format for easier integration with log analysis tools
   * Consider using OTLP output to integrate logs into modern observability platforms (e.g., Grafana, Jaeger, Datadog)
   * OTLP output seamlessly integrates with the OpenTelemetry ecosystem, enabling distributed tracing and log correlation

The overall effect is shown in the image below:

![Unified log output for multi-language extensions](https://ten-framework-assets.s3.amazonaws.com/doc-assets/log.png)


# required (/docs/ten_framework/api/required)

## Principles of the `required` Field

1. **In a communication link, the source's `required` field must be a superset of the target's `required` field**

   This means that if a target extension requires a field to be mandatory, then the source extension must also mark that field as required. This ensures that messages will not lack necessary fields during transmission.

2. **Fields with the same name in both source and target `required` fields must have compatible types**

   The framework validates field type compatibility to ensure data can be correctly transmitted and parsed. Type compatibility checks include basic type matching, array item type matching, and object property type matching.

3. **Graph Check validates the integrity of the entire message transmission path**

   * Checks whether the output message schema of the source extension is compatible with the input message schema of the target extension
   * Verifies the consistency of `required` fields throughout the transmission path
   * Ensures that message transformation rules (if any) correctly handle all required fields

Graph Check is performed before deployment, helping developers identify potential message transmission issues early and avoid runtime errors.

## Usage of the `required` Field

### When Extensions Send Messages

When an extension calls `send_<foo>(msg_X)` or `return_result(result_Y)`, the framework checks `msg_X` or `result_Y` against the corresponding schema in the extension. If `msg_X` or `result_Y` lacks any field marked as `required` in the schema, the schema check will fail.

The following three scenarios are handled in the same way:

1. **When sending a command and schema check fails**

   `send_<foo>` will immediately return `false`, and the error message will contain the schema check failure details.

2. **When `return_result` fails schema check**

   `return_result` will return `false`, and the error message will contain the schema check failure details, typically showing the missing required fields, such as: "the required properties are absent: 'foo'".

3. **When sending data-like messages (such as data, audio frames, or video frames)**

   `send_<foo>` will return `false`, and the error message will contain the schema check failure details.

**Note:** The framework's schema system only takes effect when fields are present. If a field is defined in the schema but does not exist in the message, and that field is not listed in `required`, no type checking will be performed. Type validation only occurs when fields exist both in the message and in the schema definition.

### When Extensions Receive Messages

Before the TEN runtime passes `msg_X` or `result_Y` to the extension's `on_<foo>()` or result handler, it checks whether all `required` fields defined in the schema of `msg_X` or `result_Y` are present. If any `required` fields are missing, the schema check will fail.

1. **When the incoming message is a command**

   The TEN runtime will return an error to the previous extension, with an error message indicating the missing required fields.

2. **When the incoming message is a TEN command result**

   The TEN runtime will change the result's `status_code` to error, add the missing `required` fields, and set the values of these fields to default values based on their types.

3. **When the incoming message is a data-type TEN message**

   The TEN runtime will directly discard the data-type message.

### Error Message Format

When `required` field validation fails, the framework provides detailed error information:

* Single missing field: `"the required properties are absent: 'field_name'"`
* Multiple missing fields: `"the required properties are absent: 'field1', 'field2'"`
* Missing fields in nested objects: `".nested_object: the required properties are absent: 'field_name'"`


# Type System (/docs/ten_framework/api/type_system)

## Types

The TEN framework type system is the system used within the TEN framework to define the data types of values. Developers can declare the types of messages or extension properties through TEN schema.

The TEN framework type system includes basic types and composite types. The basic types include the following:

| Type    | Description                                               |
| ------- | --------------------------------------------------------- |
| null    | Null value type.                                          |
| bool    | Boolean value, true or false.                             |
| int8    | 8-bit signed integer.                                     |
| int16   | 16-bit signed integer.                                    |
| int32   | 32-bit signed integer.                                    |
| int64   | 64-bit signed integer.                                    |
| uint8   | 8-bit unsigned integer.                                   |
| uint16  | 16-bit unsigned integer.                                  |
| uint32  | 32-bit unsigned integer.                                  |
| uint64  | 64-bit unsigned integer.                                  |
| float32 | Single precision (32-bit) IEEE 754 floating-point number. |
| float64 | Double precision (64-bit) IEEE 754 floating-point number. |
| string  | Unicode (UTF-8) character sequence.                       |
| buf     | Sequence of 8-bit unsigned bytes.                         |
| ptr     | Pointer to a memory address.                              |

Composite types include the following:

| Type   | Description                                                    |
| ------ | -------------------------------------------------------------- |
| array  | Collection of elements of the same type.                       |
| object | Represents complex key/value pairs. Key type is always string. |

For basic types, properties can be accessed or set using methods such as `get_property()` / `set_property()`. For example:

```cpp
// Get property value
int32_t value = cmd.get_property_int32("property_name");

// Set property value
cmd.set_property("property_name", 100);
```

For composite types, it is typically necessary to use related serialization methods. For example:

```go
type MyProp struct {
    Name string `json:"name"`
}

var prop MyProp
bytes, _ := json.Marshal(&prop)
cmd.SetPropertyFromJSONBytes("property_name", bytes)
```

## JSON Type Mapping

When creating TEN values from JSON, the TEN framework uses the following mapping rules:

| JSON Type | TEN Type Mapping Rules                                                          |
| --------- | ------------------------------------------------------------------------------- |
| Integer   | Maps to int32 if the value is within int32 range; otherwise maps to int64       |
| Float     | Maps to float32 if the value is within float32 range; otherwise maps to float64 |
| String    | Maps to string                                                                  |
| Boolean   | Maps to bool                                                                    |
| null      | Maps to null                                                                    |
| Array     | Maps to array                                                                   |
| Object    | Maps to object                                                                  |

It's important to note that JSON numeric types, when no explicit schema is defined, will automatically select the appropriate TEN type based on the numeric value size. This ensures data precision and efficiency.

## Types and Schema

If a TEN schema is specified, the property type will be determined according to the corresponding TEN schema. If no TEN schema is specified, the property type will be determined based on the type of the initial value assignment. For example, if the initial assignment is C++'s `int32_t`, the property type will be `int32`; if the initial assignment is done using JSON, the type will be determined according to JSON processing rules.

When a TEN schema is defined, JSON data will be forcibly converted to the type specified by the schema. If the conversion fails, an error will be reported.

## Conversion Rules

The TEN framework supports flexible automatic conversion between different types of values. Conversions are divided into two types: safe conversions and unsafe conversions.

### Type Compatibility

The TEN framework uses type compatibility rules to determine whether conversion between two types is possible:

* **Integer type compatibility**: All integer types (int8, int16, int32, int64, uint8, uint16, uint32, uint64) are mutually compatible
* **Float compatible with integer**: Floating-point types can accept values of integer types (when the integer value is within the floating-point representation range)
* **Other types**: bool, ptr, string, array, object, buf, null types are only compatible with themselves

### Safe Conversion (Must Succeed)

Converting a lower precision type to a higher precision type is always safe and guaranteed to succeed, as the higher precision type can fully accommodate the value of the lower precision type without data loss.

In the TEN type system, safe conversion rules are as follows:

1. Within signed integer types, from lower to higher precision
2. Within unsigned integer types, from lower to higher precision
3. From smaller range unsigned integers to larger range signed integers (when the value is within the target type range)
4. Within floating-point types, from float32 to float64

| From    | To (Allowed Safe Conversions)                    |
| ------- | ------------------------------------------------ |
| int8    | int16 / int32 / int64                            |
| int16   | int32 / int64                                    |
| int32   | int64                                            |
| uint8   | uint16 / uint32 / uint64 / int16 / int32 / int64 |
| uint16  | uint32 / uint64 / int32 / int64                  |
| uint32  | uint64 / int64                                   |
| float32 | float64                                          |

For example:

```cpp
// Set property value. The type of `property_name` in TEN runtime is `int32`.
cmd.set_property("property_name", 100);

// Get property value. Correct.
int32_t value = cmd.get_property_int32("property_name");

// Get property value. Correct. TEN type system will automatically convert the type to `int64`.
int64_t value2 = cmd.get_property_int64("property_name");

// Get property value. Incorrect, an error will be thrown because this is an unsafe conversion.
int16_t error_type = cmd.get_property_int16("property_name");
```

### Unsafe Conversion (May Fail)

Converting a higher precision type to a lower precision type is unsafe because the value of the higher precision type may exceed the range of the lower precision type, leading to data loss. This conversion is called unsafe conversion. When performing unsafe conversion, the TEN runtime performs strict range checking. If an overflow or precision loss occurs, the TEN type system will throw an error.

In the TEN framework type system, unsafe conversions include:

1. Converting higher precision integers to lower precision integers
2. Converting signed integers to unsigned integers (when the value is negative)
3. Converting unsigned integers to signed integers (when the value exceeds the signed integer range)
4. Converting `float64` to `float32`
5. Converting floating-point numbers to integers (when there are decimal parts or the value exceeds integer range)
6. Converting integers to floating-point numbers (when precision loss may occur)

| Conversion Type                | Success Condition                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------ |
| High precision → Low precision | Value must be within target type range                                               |
| Signed → Unsigned              | Value must be non-negative and within target type range                              |
| Unsigned → Signed              | Value must be within target type range                                               |
| float64 → float32              | Value must be within float32 range without precision loss                            |
| Floating-point → Integer       | Value must be within target integer type range and have no decimal part              |
| Integer → Floating-point       | Integer value must be exactly representable in floating-point without precision loss |

#### Range Check Details

TEN framework implements precise range checking algorithms:

* **Integer range checking**: Uses standard MIN/MAX constants for comparison
* **Floating-point range checking**: Uses `ldexp` function to calculate precise boundary values
* **Precision checking**: For integer to floating-point conversion, performs round-trip conversion verification to ensure no precision loss

### When Conversion Is Triggered

The TEN runtime performs type conversion in the following situations:

1. **When loading `property.json`**
   * JSON integers are parsed to appropriate integer types based on numeric value size
   * JSON floating-point numbers are parsed to appropriate floating-point types based on numeric value size
   * If the property has a defined TEN schema, forced type conversion will be performed according to the schema (may be unsafe conversion)

2. **When calling methods like `set_property_from_json()`**
   * When passing serialized JSON strings, type conversion will be performed according to the schema

3. **During property access**
   * When using `get_property_xxx()` methods to get properties, the system will attempt to convert the stored type to the requested type
   * This may trigger safe or unsafe conversions

4. **During message passing**
   * When message properties need to be converted between different types

### Error Handling

When type conversion fails, the TEN framework will:

* Return error status codes and detailed error information
* Provide source type, target type, and failure reason in the error object
* For out-of-range conversions, the error message format is: "out of range of \[target\_type]"
* For incompatible type conversions, the error message format is: "unsupported conversion from `[source_type]` to `[target_type]`"

#### Common Error Examples

```cpp
// Range overflow error
int8_t small_value = cmd.get_property_int8("large_number");  // If large_number = 200
// Error: out of range of int8

// Type incompatibility error
std::string str_value = cmd.get_property_string("integer_prop");
// Error: unsupported conversion from `int32` to `string`

// Precision loss error
float float_value = cmd.get_property_float32("precise_double");  // If precise_double exceeds float32 precision
// Error: out of range of float32
```

## Best Practices

1. **Use appropriate types**: Choose suitable integer and floating-point types based on the actual data range
2. **Explicitly define schema**: Clearly define property types in manifest.json to avoid type conversion ambiguity
3. **Handle conversion errors**: Properly handle potential errors from type conversions in your code
4. **Understand JSON mapping**: Be aware of JSON data to TEN type mapping rules to avoid unexpected type conversions


# Development Process (/docs/ten_framework/development/development_process)

Basically, by using the TEN framework to complete the desired scenarios, there are two development approaches or perspectives:

1. From small extensions to large scenarios
2. From large scenarios to small extensions

**Note:** Regardless of which of these two development approaches you take, you can either write your own extensions from scratch or download pre-built extensions from the TEN cloud store.

In the first approach, **from small extensions to large scenarios**, the development process typically follows these steps:

1. Develop extensions.
2. Assemble the extensions into a graph, and further create an agent, for example, [TEN-Agent](https://github.com/TEN-framework/TEN-Agent).

In the second approach, **from large scenarios to small extensions**, the development process typically follows these steps:

1. Select the desired agent template, for example, [TEN-Agent](https://github.com/TEN-framework/TEN-Agent).
2. Replace some of the extensions within the chosen agent template with your own.

In this second development approach, you can ensure that the behavior of the replaced extension matches the original one by using the framework for **standalone testing** of extensions. In this process, the developer only needs to understand concepts related to TEN extensions, including how to independently develop and test a TEN extension. All other TEN-related concepts outside of extension development and testing do not need to be understood, as these concepts are encapsulated within the selected TEN agent.


# Complete Guide to TEN Extension Development (/docs/ten_framework/development/how_to_develop_with_ext)

TEN Framework provides rich extension templates to help developers quickly create extensions and complete the entire process from development to testing. This guide will demonstrate in detail how to use C++, Go, Python, and Node.js for the complete extension development workflow through practical operations.

## Pre-development Preparation

### Environment Requirements

Before starting extension development, please ensure your development environment is properly configured. Verify the installation with the following command:

```bash
tman --version
```

Under normal circumstances, you should see version information output similar to the following:

```bash
TEN Framework version: <version>
```

> **Important Note**: Please ensure you are using `tman` version >= 0.10.12. If your version is too low, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to install the latest version.

### Development Workflow Overview

Regardless of which programming language you use, TEN extension development follows the following standardized workflow:

1. **Project Creation** - Generate extension project skeleton using official templates
2. **Dependency Installation** - Configure runtime environment and install necessary dependency packages
3. **Core Development** - Implement extension business logic and functional code
4. **Build and Test** - Compile project (if needed) and execute unit tests
5. **Debug and Optimize** - Use professional debugging tools to locate and solve problems

***

## C++ Extension Development

C++ extensions are suitable for application scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency data computation, low-level system operations, etc.

### Create Project

Use the official C++ extension template provided by TEN to quickly create a new project:

```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```

After successful project creation, you will get the following complete project structure:

```bash
my_example_ext_cpp/
├── BUILD.gn              # GN build system configuration file
├── manifest.json         # Extension metadata and configuration information
├── property.json         # Extension properties and parameter configuration
├── src/
│   └── main.cc           # Extension core implementation code
├── tests/
│   ├── basic.cc          # Basic functionality test cases
│   └── gtest_main.cc     # Test framework program entry
├── include/              # Header file storage directory
├── tools/                # Auxiliary tools and scripts
└── .vscode/
    └── launch.json       # VSCode debugging configuration file
```

### Environment Configuration Verification

Enter the project directory and verify that the build tools are working properly:

```bash
cd my_example_ext_cpp
tgn --version
```

> **Expected Output**:

```bash
0.1.0
```

If the command cannot be executed, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to check whether the TEN Framework development environment is correctly installed.

### Dependency Package Installation

Install all dependency packages required for extension runtime:

```bash
tman install --standalone
```

After successful installation, you will see detailed installation logs similar to the following:

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done

🏆  Install successfully in 1 second
```

### Project Build

TEN framework provides two convenient build methods for you to choose from:

#### Method 1: Manual Step-by-step Build

```bash
# Step 1: Generate build configuration files (enable standalone test mode)
tgn gen linux x64 debug -- ten_enable_standalone_test=true

# Step 2: Execute project build
tgn build linux x64 debug
```

#### Method 2: Use tman Shortcut Command

```bash
tman run build
```

After the build is complete, check the generated executable test file:

```bash
ls -la bin/
# You should see: my_example_ext_cpp_test
```

### Run Tests

Verify that the extension functionality is working properly:

#### Method 1: Execute Test File Directly

```bash
./bin/my_example_ext_cpp_test
```

#### Method 2: Use tman Unified Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN      ] Test.Basic
[       OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[  PASSED  ] 1 test.
```

> **Success Indicator**: When you see `[  PASSED  ] 1 test.` it means all tests have passed.

### Core Code Structure Explanation

#### Extension Main Implementation (src/main.cc)

The core class of C++ extensions needs to inherit from the `ten::extension_t` base class and implement complete lifecycle management methods:

```cpp
class my_example_ext_cpp_t : public ten::extension_t {
 public:
  // Extension initialization phase - basic configuration and resource pre-allocation
  void on_init(ten::ten_env_t &ten_env) override;

  // Extension startup phase - officially start processing business logic
  void on_start(ten::ten_env_t &ten_env) override;

  // Command handler - handle command requests from other extensions or applications
  void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr<ten::cmd_t> cmd) override;

  // Data handler - handle general data flow
  void on_data(ten::ten_env_t &ten_env, std::unique_ptr<ten::data_t> data) override;

  // Audio frame handler - handle real-time audio stream data
  void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::audio_frame_t> frame) override;

  // Video frame handler - handle real-time video stream data
  void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::video_frame_t> frame) override;

  // Extension stop phase - clean up resources and graceful shutdown
  void on_stop(ten::ten_env_t &ten_env) override;
};
```

#### Test Framework Implementation (tests/basic.cc)

Use TEN-specific test framework to write complete unit tests:

```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    // Create command object for testing
    auto new_cmd = ten::cmd_t::create("foo");

    // Send command to extension and verify response results
    ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
      // Verify the correctness of test results here
    });
  }
};
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

1. Install the **CodeLLDB** extension plugin for VSCode
2. Set breakpoints in source code
3. Select the "standalone test (lldb, launch)" debug configuration
4. Press F5 to start debugging session

Debug configuration file `.vscode/launch.json` content:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (lldb, launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

#### Command Line Debugging

Use the classic GDB debugger for command-line debugging:

```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt  # View complete function call stack
```

***

## Go Extension Development

Go extensions provide a good balance between high performance and development efficiency, particularly suitable for building network services, concurrent processing, microservice architecture and other application scenarios.

### Create Project

Create a new project using the Go extension template:

```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```

After successful project creation, it will display:

```bash
🏆  Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```

The complete project structure is as follows:

```bash
my_example_ext_go/
├── extension.go             # Extension core implementation code
├── go.mod                   # Go module dependency management configuration
├── manifest.json            # Extension metadata information
├── property.json            # Extension property configuration
├── README.md                # Project documentation
├── tests/                   # Test-related files
│   ├── basic_tester.go      # Test logic implementation
│   ├── basic_tester_test.go # Test case definition
│   ├── main_test.go         # Test program entry
│   └── bin/start            # Test startup script
└── .vscode/launch.json      # VSCode debug configuration
```

### Dependency Package Installation

Install dependency packages required for project runtime:

```bash
tman install --standalone
```

### Run Tests

Verify the correctness of extension functionality:

#### Method 1: Use Startup Script

```bash
./tests/bin/start
```

#### Method 2: Use tman Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
=== RUN   TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```

### Core Code Structure Explanation

#### Extension Implementation (extension.go)

Core implementation structure of Go extensions:

```go
import (
    ten "ten_framework/ten_runtime"
)

type ExampleExtension struct {
    ten.DefaultExtension
}

// Extension startup lifecycle
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStart")
    tenEnv.OnStartDone()
}

// Command handling logic
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    tenEnv.LogDebug("OnCmd")
    cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
    tenEnv.ReturnResult(cmdResult, nil)
}

// Extension stop lifecycle
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStop")
    tenEnv.OnStopDone()
}
```

#### Test Framework (tests/basic\_tester.go)

Go extension test framework implementation:

```go
type BasicExtensionTester struct {
    ten.DefaultExtensionTester
}

func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
    // Create command object for testing
    cmdTest, _ := ten.NewCmd("test")

    // Send command and handle response results
    tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
        if err != nil {
            panic(err)
        }
        statusCode, _ := cr.GetStatusCode()
        if statusCode != ten.StatusCodeOk {
            panic(statusCode)
        }
        tenEnvTester.StopTest(nil)
    })

    tenEnvTester.OnStartDone()
}
```

### Development Environment Optimization

To improve development experience and debugging convenience, it is recommended to create a `go.work` workspace file in the extension root directory:

```go
go 1.18

use (
    .
    .ten/app/ten_packages/system/ten_runtime_go/interface
)
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure the official Go extension is installed, then use the following debug configuration:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (go, launch)",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}/tests/",
      "env": {
        "CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
        "LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
      },
      "args": ["-test.v"]
    }
  ]
}
```

***

## Python Extension Development

Python extensions have the highest development efficiency, particularly suitable for rapid prototyping, AI/ML application integration, complex business logic implementation and other scenarios.

### Create Project

Create a project using the Python async extension template:

```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```

Complete project structure:

```bash
my_example_ext_python/
├── extension.py         # Extension core implementation code
├── addon.py             # Extension plugin registration entry
├── __init__.py          # Python package initialization file
├── requirements.txt     # Python dependency package list
├── manifest.json        # Extension metadata configuration
├── property.json        # Extension property configuration
├── tests/               # Test-related files
│   ├── test_basic.py    # Basic test cases
│   ├── conftest.py      # pytest configuration file
│   └── bin/start        # Test startup script
└── .vscode/launch.json  # VSCode debug configuration
```

### Dependency Package Installation

Install Python dependency packages required for the project:

```bash
tman install --standalone
```

### Run Tests

Verify Python extension functionality:

#### Method 1: Use Startup Script

```bash
./tests/bin/start
```

#### Method 2: Use tman Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py .                                                                                [100%]
============================================ 1 passed in 0.04s =======================================
```

### Core Code Structure Explanation

#### Extension Implementation (extension.py)

Python extensions recommend using modern async programming patterns for better performance and concurrent processing capabilities:

```python
from ten_runtime import (
    AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
    Cmd, StatusCode, CmdResult, Data
)

class ExampleExtension(AsyncExtension):
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_init")

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_start")
        # TODO: Read configuration files here, initialize necessary resources

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_name = cmd.get_name()
        ten_env.log_debug(f"on_cmd name {cmd_name}")

        # TODO: Implement specific business logic processing here
        cmd_result = CmdResult.create(StatusCode.OK, cmd)
        await ten_env.return_result(cmd_result)

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_stop")
        # TODO: Clean up resources here, perform graceful shutdown
```

#### Plugin Registration Entry (addon.py)

Extension plugin registration and creation logic:

```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension

@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
    def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
        ten_env.log_info("on_create_instance")
        ten_env.on_create_instance_done(ExampleExtension(name), context)
```

#### Test Implementation (tests/test\_basic.py)

Complete async test framework implementation:

```python
from ten_runtime import (
    AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
    TenError, TenErrorCode
)

class ExtensionTesterBasic(AsyncExtensionTester):
    async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
        # Create command object for testing
        new_cmd = Cmd.create("hello_world")

        ten_env.log_debug("send hello_world")
        result, err = await ten_env.send_cmd(new_cmd)

        # Verify the correctness of test results
        if (err is not None or result is None
            or result.get_status_code() != StatusCode.OK):
            ten_env.stop_test(TenError.create(
                TenErrorCode.ErrorCodeGeneric,
                "Failed to send hello_world"
            ))
        else:
            ten_env.stop_test()

def test_basic():
    tester = ExtensionTesterBasic()
    tester.set_test_mode_single("my_example_ext_python")
    err = tester.run()
    if err is not None:
        assert False, err.error_message()
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure you have installed the Python extension and debugpy debugger, use the following configuration for debugging:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (debugpy, launch)",
      "type": "debugpy",
      "request": "launch",
      "python": "/usr/bin/python3",
      "module": "pytest",
      "args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
      "env": {
        "TEN_ENABLE_PYTHON_DEBUG": "true",
        "PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
      },
      "console": "integratedTerminal"
    }
  ]
}
```

***

## Node.js Extension Development

Node.js extensions provide a modern JavaScript/TypeScript development experience, particularly suitable for Web application integration, rapid prototyping, frontend technology stack extensions and other scenarios. Thanks to Node.js's async characteristics and rich ecosystem, developers can easily build efficient real-time applications.

### Create Project

Use the official Node.js extension template provided by TEN to quickly create a new project:

```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```

After successful project creation, you will get the following complete project structure:

```bash
my_example_ext_nodejs/
├── manifest.json         # Extension metadata and configuration information
├── property.json         # Extension properties and parameter configuration
├── package.json          # Node.js dependency package management configuration
├── tsconfig.json         # TypeScript compiler configuration
├── src/
│   └── index.ts          # Extension core implementation code
├── tests/                # Test-related files
│   ├── src/
│   │   ├── index.ts      # Tester implementation
│   │   ├── index.spec.ts # Test case definition
│   │   └── main.spec.ts  # Test framework configuration
│   ├── bin/start         # Test startup script
│   ├── package.json      # Test dependency configuration
│   └── tsconfig.json     # Test TypeScript configuration
├── tools/                # Auxiliary tools and scripts
└── .vscode/
    └── launch.json       # VSCode debug configuration file
```

### Environment Configuration Verification

Enter the project directory and verify the development environment:

```bash
cd my_example_ext_nodejs
node --version
npm --version
```

> **Expected Output**:

```bash
v18.0.0 or higher
8.0.0 or higher
```

### Dependency Package Installation

Install all dependency packages required for extension runtime:

```bash
tman install --standalone
```

After successful installation, you will see detailed installation logs similar to the following:

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       2/2       Done

🏆  Install successfully in 1 second
```

> **Important Note**: `tman install --standalone` will create a `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory in the project directory. Subsequent build and test operations need to be performed in this directory.

### Project Build

Node.js extensions are developed using TypeScript and need to install standalone mode dependencies first, then compile to JavaScript:

#### Method 1: Manual Build

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Install standalone mode Node.js dependencies
npm run standalone-install

# Compile TypeScript code
npm run build
```

#### Method 2: Use tman Shortcut Command

```bash
tman run build
```

After the build is complete, check the generated compilation results:

```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# You should see: index.js and related mapping files
```

### Run Tests

Verify that the extension functionality is working properly:

#### Method 1: Use Startup Script

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Run tests
./tests/bin/start
```

#### Method 2: Use tman Unified Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case2 (1010ms)

  2 passing (2s)
```

> **Success Indicator**: When you see all test cases showing `✓` and finally displaying `passing`, it means all tests have passed.

### Core Code Structure Explanation

#### Extension Main Implementation (src/index.ts)

The core class of Node.js extensions needs to inherit from the `Extension` base class and implement complete lifecycle management methods:

```typescript
import {
  Addon,
  RegisterAddonAsExtension,
  Extension,
  TenEnv,
  Cmd,
  CmdResult,
  StatusCode,
} from "ten-runtime-nodejs";

class DefaultExtension extends Extension {
  constructor(name: string) {
    super(name);
  }

  // Extension configuration phase - read and validate configuration parameters
  async onConfigure(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onConfigure");
  }

  // Extension initialization phase - basic configuration and resource pre-allocation
  async onInit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onInit");
  }

  // Extension startup phase - officially start processing business logic
  async onStart(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStart");
  }

  // Command handler - handle command requests from other extensions or applications
  async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
    console.log("DefaultExtension onCmd", cmd.getName());

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("detail", "This is a demo");
    tenEnv.returnResult(cmdResult);
  }

  // Extension stop phase - clean up resources and graceful shutdown
  async onStop(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStop");
  }

  // Extension destruction phase - final cleanup and resource release
  async onDeinit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onDeinit");
  }
}
```

#### Plugin Registration Entry

Extension plugin registration and creation logic:

```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
  async onCreateInstance(
    _tenEnv: TenEnv,
    instanceName: string,
  ): Promise<Extension> {
    return new DefaultExtension(instanceName);
  }
}
```

#### Test Framework Implementation (tests/src/index.ts)

Use TEN-specific test framework to write complete unit tests:

```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";

export class MyExtensionTester extends ExtensionTester {
  async onStart(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStart");

    // Simulate async operations and test logic
    new Promise((resolve) => {
      setTimeout(() => {
        resolve(true);
      }, 1000);
    }).then(() => {
      // Stop test and return results
      tenEnvTester.stopTest();
    });
  }

  async onStop(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStop");
  }

  async onDeinit(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onDeinit");
  }
}
```

#### Test Case Definition (tests/src/index.spec.ts)

Use Mocha test framework to write specific test cases:

```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";

const test_addon_name = "default_extension_nodejs";

describe("MyExtensionTester", () => {
  it("case1", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });

  it("case2", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });
});
```

### TypeScript Configuration

Node.js extensions use modern TypeScript configuration, supporting the latest language features:

```json
{
  "compilerOptions": {
    "allowJs": false,
    "composite": true,
    "module": "NodeNext",
    "target": "es6",
    "moduleResolution": "NodeNext",
    "outDir": "build",
    "removeComments": false,
    "sourceMap": true,
    "noImplicitAny": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "strictNullChecks": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "rootDir": "src",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
  },
  "exclude": [
    "node_modules"
  ],
  "include": [
    "src/**/*"
  ]
}
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure you have installed the official Node.js extension, then use the following debug configuration:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (nodejs) (mocha, launch)",
      "type": "node",
      "request": "launch",
      "program": "node_modules/mocha/bin/_mocha",
      "stopOnEntry": true,
      "args": [
        "--no-timeouts",
        "--package",
        "package.json",
      ],
      "cwd": "${workspaceFolder}/tests",
      "env": {
        "NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
      },
      "runtimeArgs": [
        "--expose-gc",
        "--loader",
        "ts-node/esm",
        "--no-warnings",
      ]
    }
  ]
}
```

#### Command Line Debugging

Use Node.js built-in debugger for command-line debugging:

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Start debug mode
node --inspect-brk ./tests/bin/start

# Or use Chrome DevTools
node --inspect ./tests/bin/start
```

### Complete Development Process Summary

To help developers get started quickly, here is a complete Node.js extension development process summary:

```bash
# 1. Create extension project
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example

# 2. Enter project directory
cd my_example_ext_nodejs

# 3. Install dependencies
tman install --standalone

# 4. Enter extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 5. Install standalone mode dependencies
npm run standalone-install

# 6. Build project
npm run build

# 7. Run tests
./tests/bin/start
```

> **Working Directory Note**:
>
> * Extension source code is located in the `src/` folder at the project root directory
> * Actual build, test, and run operations are performed in the `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory
> * This design ensures extension independence and proper dependency management

### Development Best Practices

#### Async Programming

Node.js extensions fully utilize async programming patterns for better performance:

```typescript
// Recommended: Use async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    const result = await processCommand(cmd);
    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("result", result);
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

#### Error Handling

Implement comprehensive error handling mechanisms:

```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    // Business logic processing
    const result = await this.handleBusinessLogic(cmd);

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("data", JSON.stringify(result));
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    console.error("Command processing error:", error);

    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

***

## Development Summary

By following the complete development process provided in this guide, you can efficiently develop, test, and debug TEN extensions. Whether you choose C++, Go, Python, or Node.js, TEN Framework provides you with a complete toolchain and best practices to help you fully leverage the powerful features of TEN Framework and build high-performance, high-reliability extension applications.

Each language has its unique advantages and applicable scenarios:

* **C++**: Suitable for scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency computation
* **Go**: Provides a balance between high performance and development efficiency, suitable for network services, concurrent processing
* **Python**: Has the highest development efficiency, particularly suitable for AI/ML applications, rapid prototyping
* **Node.js**: Provides modern Web development experience, suitable for frontend technology stack extensions, real-time applications

Please choose the most suitable development solution based on your specific needs and team technology stack. During the development process, it is recommended to fully utilize the debugging tools and testing frameworks provided by TEN Framework to ensure the quality and stability of your extensions.


# TEN Framework Quick Start Guide (/docs/ten_framework/getting-started/quick-start)

## TEN Framework Quick Start Guide

> 🎯 **Goal**: Set up your development environment and run your first TEN app in 5 minutes

## System Requirements

**Supported Operating Systems**:

* Linux (x64)
* Linux (arm64)
* macOS Intel (x64)
* macOS Apple Silicon (arm64)
* Windows (x64)

**Required Software**:

* Python 3.10
* Go 1.20+
* Node.js / npm (for managing JavaScript dependencies)

> 💡 **Windows users**: It's recommended to use PowerShell to run the commands in this guide. Some commands may not be compatible with CMD.

## Step 1: Check Your Environment

Before you begin, make sure the following software is installed on your system:

### Python 3.10

```bash
python3 --version
# Should display: Python 3.10.x
```

> 💡 **Important**: TEN Framework currently only supports Python 3.10. It's recommended to use `pyenv` or `venv` to manage your Python environment:
>
> **Linux / macOS:**
>
> ```bash
> # Install and manage Python 3.10 using pyenv (recommended)
> pyenv install 3.10.14
> pyenv local 3.10.14
>
> # Or create virtual environment using venv
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows:**
>
> Download the Windows installer from the bottom table at [https://www.python.org/downloads/release/python-31011/](https://www.python.org/downloads/release/python-31011/) and run it to install Python 3.10.
>
> Make sure to check "Add Python to PATH" before clicking Install Now.
>
> ```bash
> # On Windows, configure the python3 command:
>
> # First, find where python is installed:
> where.exe python
> # Example output: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # Then, open PowerShell as Administrator and create a symlink in the same directory:
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # Replace the paths above with the actual output from where.exe python
>
> # Verify:
> python3 --version
> ```
>
> ```powershell
> # It's recommended to create a venv after installation
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # Activate the environment
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # If you encounter a permission error, close the terminal/IDE, right-click and select "Run as Administrator" to reopen
> # Or change the execution policy to allow ps1 scripts
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```

### Go 1.20+

```bash
go version
# Should display: go version go1.20 or higher

$env:CGO_ENABLED = "1"
# CGO must be explicitly enabled on Windows
```

### Node.js / npm

```bash
node --version
npm --version
# Ensure node and npm commands are available
```

### GCC (MinGW)

💡 Only required on Windows

```bash
# First, make sure winget is available.
winget --version

# If not, install it from <https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US>
# (Requires Windows 10 version 1709 (Build 16299) or later, or Windows 11)

# Install MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT

# Or search and choose a suitable version to install
winget search "mingw"

# Verify installation
gcc --version
```

> 💡 **Tip**: If any of the above is missing, please install the required version before continuing.

## Step 2: Install TEN Manager (tman)

TEN Manager (tman) is the command-line tool for TEN Framework, used to create projects, manage dependencies, and run applications.

Option 1: Install via Package Manager (Recommended)

**Linux (Ubuntu/Debian):**

```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```

**macOS:**

```bash
brew install TEN-framework/ten-framework/tman
```

**Windows:**

```powershell
winget install TEN-framework.tman
```

Option 2: Install via Script

**Linux / macOS:**

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```

**Windows:**

```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```

Option 3: Install from cloned repository

```bash
cd ten-framework

# Linux/macOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```

> 💡 **Note**: If tman is already installed on your system, the installation script will ask whether you want to reinstall/upgrade it. Press `y` to continue or `n` to cancel.
>
> **Non-interactive Installation** (for automation scripts or CI environments, not available on Windows):
>
> ```bash
> # Remote installation
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # Local installation
> yes y | bash tools/tman/install_tman.sh
> ```

**Verify Installation**:

```bash
tman --version
```

> 💡 **Tip**: If you see `tman: command not found` (Linux/macOS) or a similar unrecognized command error (Windows), make sure the tman directory is in your PATH:
>
> **Linux / macOS:**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc  # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc   # macOS
> source ~/.bashrc  # or source ~/.zshrc
> ```
>
> **Windows:**
>
> ```powershell
> # Check if tman is in PATH
> $env:Path -split ";" | Select-String "tman"
> # If not, either:
> # 1. Add it manually via "System Properties → Environment Variables", then restart your terminal
> # 2. Or temporarily add the path with:
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```

## Step 3: Create and Run the Demo App

<div className="fd-steps">
  <div className="fd-step">
    ### Create App

    ```bash
    # Create a new transcriber_demo app
    tman install app transcriber_demo
    cd transcriber_demo
    ```
  </div>

  <div className="fd-step">
    ### Install Dependencies

    ```bash
    # Install TEN package dependencies
    tman install

    # Install Python and npm package dependencies
    tman run install_deps
    ```

    > ⏱️ **Estimated Time**: 1-2 minutes
  </div>

  <div className="fd-step">
    ### Build the App

    ```bash
    tman run build
    ```

    > ⏱️ **Estimated Time**: 30 seconds
  </div>

  <div className="fd-step">
    ### Configure Environment Variables

    Before running the app, you need to configure the ASR (Automatic Speech Recognition) service credentials. The current example uses Azure ASR extension. You need to fill in the configuration in the `transcriber_demo/.env` file:

    **Linux / macOS:**

    ```bash
    # Create .env file
    cat > .env << EOF
    # Azure Speech Service Configuration
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region      # e.g., eastus
    AZURE_STT_LANGUAGE=en-US                # Set according to your audio language or real-time recording language, e.g., zh-CN, ja-JP, ko-KR, etc.
    EOF
    ```

    **Windows (PowerShell):**

    ```powershell
    # Create .env file
    @"
    # Azure Speech Service Configuration
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region
    AZURE_STT_LANGUAGE=en-US
    "@ | Out-File -Encoding utf8 .env
    ```

    > 💡 **Tip**: Windows users can also create the `.env` file directly with a text editor and fill in the configuration above.
    >
    > 💡 **Tip**: If you want to use other ASR extensions (such as OpenAI Whisper, Google Speech, etc.), you can download and replace them from the cloud store. Similarly, configure the corresponding API keys and environment variables in the `.env` file.
  </div>

  <div className="fd-step">
    ### Run the App

    ```bash
    tman run start
    ```

    If everything is working correctly, you should see output similar to:

    ```text
    [web_audio_control_go] Web server started on port 8001
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```
  </div>

  <div className="fd-step">
    ### Experience the Demo

    Open your browser and visit:

    ```text
    http://localhost:8001
    ```

    You should see the Transcriber Demo web interface. Try:

    * Click the microphone button for real-time voice transcription
    * Upload an audio file for transcription
    * View real-time transcription and subtitle results
  </div>
</div>

## Congratulations! 🎉

You've successfully run your first TEN application!

### Understanding the App Architecture

This `transcriber_demo` app showcases TEN Framework's multi-language extension capabilities, consisting of:

* **Go** - WebSocket server extension (`web_audio_control_go`)
* **Python** - ASR speech recognition extension (`azure_asr_python`)
* **TypeScript** - VTT subtitle generation and audio recording extension (`vtt_nodejs`)

🎯 **You can now run extensions in multiple languages!**

### Next Steps

Now you can:

1. **Explore and download more extensions from the cloud store, design and orchestrate your app**

   ```bash
   tman designer  # Launch TMAN Designer to explore extensions, download them, and design your app
   ```

2. **Choose a language and develop your own extension**
   * Supports Go, Python, TypeScript/JavaScript, C++, and more
   * Check out the [TEN Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) for details

## Advanced: Developing and Building C++ Extensions

If you want to develop and use C++ extensions, you'll need to install the TEN build toolchain (tgn). Here's the complete process:

<div className="fd-steps">
  <div className="fd-step">
    ### Install tgn Build Tool

    tgn is TEN Framework's C/C++ build system, based on Google's GN.

    Option 1: One-line Installation (Recommended)

    **Linux / macOS:**

    ```bash
    curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
    ```

    **Windows:**

    ```powershell
    irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
    ```

    Option 2: Install from Cloned Repository

    **Linux / macOS:**

    ```bash
    # If you've already cloned the TEN Framework repository
    cd ten-framework
    bash tools/tgn/install_tgn.sh
    ```

    **Windows:**

    ```powershell
    # If you've already cloned the TEN Framework repository
    & ".\tools\tgn\install_tgn.ps1"
    ```

    After installation, ensure tgn is added to PATH:

    **Linux / macOS:**

    ```bash
    # Temporarily add to current session
    export PATH="/usr/local/ten_gn:$PATH"

    # Or permanently add to shell configuration (recommended)
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc  # Linux
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc   # macOS
    source ~/.bashrc  # or source ~/.zshrc
    ```

    **Windows:**

    ```powershell
    # The install script adds to user PATH automatically. To add manually:
    $env:Path += ";$env:LOCALAPPDATA\ten_gn"
    ```

    Verify installation:

    ```bash
    tgn --help
    ```
  </div>

  <div className="fd-step">
    ### Install C++ Extension

    Using WebRTC VAD (Voice Activity Detection) extension as an example, install a C++ extension from the cloud store:

    ```bash
    cd transcriber_demo
    tman install extension webrtc_vad_cpp
    ```

    > 💡 **Note**: `webrtc_vad_cpp` is a voice activity detection extension implemented in C++. It can filter out non-speech segments in real-time speech recognition scenarios.
  </div>

  <div className="fd-step">
    ### Compile C++ Extension

    After installing the C++ extension, rebuild the app to compile the C++ code into a dynamic library:

    ```bash
    tman run build
    ```

    > ⏱️ **Expected Time**: First-time compilation of C++ extensions may take 1-3 minutes, depending on your machine's performance.
  </div>

  <div className="fd-step">
    ### Run App with VAD Functionality

    ```bash
    tman run start_with_vad
    ```

    If everything works correctly, you should see:

    ```text
    [web_audio_control_go] Web server started on port 8001
    [vad] WebRTC VAD initialized with mode 2
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```

    Now open your browser at `http://localhost:8001` and navigate to the microphone real-time transcription page. You'll see the silence state changes after VAD processing. When the silence state is true, it indicates there is no speech in the current audio.
  </div>
</div>

### C++ Development Environment Requirements

Developing and compiling C++ extensions requires installing a C++ compiler (gcc or clang):

**Linux:**

```bash
# Ubuntu/Debian
sudo apt-get install gcc g++

# Or use clang
sudo apt-get install clang
```

**macOS:**

```bash
# Install Xcode Command Line Tools (includes clang)
xcode-select --install
```

**Windows:**

```powershell
# Option 1: Install Visual Studio Build Tools (recommended)
# Download from https://visualstudio.microsoft.com/visual-cpp-build-tools/

# Option 2: Install via winget
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

# 💡 Note:
#  - Select the "Desktop development with C++" workload during installation
#  - Make sure to check "C++ Clang tools for Windows"
```

Verify compiler installation:

```bash
# Check gcc (Linux/macOS/MSYS2)
gcc --version
g++ --version

# Or check clang (Linux/macOS)
clang --version
```

**Windows (MSVC):**

```powershell
# Run in "Developer PowerShell for VS"
clang-cl --version
```

### Troubleshooting (C++ Extensions)

1. tgn command not found

   Ensure you've run the installation script and added tgn to PATH:

   ```bash
   export PATH="/usr/local/ten_gn:$PATH"
   ```

2. Compilation failed: Compiler not found

   Please refer to the "C++ Development Environment Requirements" section above to install the compiler.

### Learn More

* [ten\_gn Build System](https://github.com/TEN-framework/ten_gn) - TEN's C/C++ build tool
* [C++ Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) - Complete C++ extension development documentation

## Troubleshooting

<div className="fd-steps">
  <div className="fd-step">
    ### Python Library Loading Failure on macOS

    **Problem**: Error indicating `libpython3.10.dylib` cannot be found when running the app

    **Solution**:

    ```bash
    export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
    ```

    Consider adding this line to your `~/.zshrc` or `~/.bash_profile`.
  </div>

  <div className="fd-step">
    ### tman Download Failed or Slow

    **Problem**: Network connection to GitHub is restricted

    **Solution**:

    * Manual download: Visit the [Releases page](https://github.com/TEN-framework/ten-framework/releases) to download the `tman` binary for your platform
  </div>

  <div className="fd-step">
    ### Port 8001 Already in Use

    **Problem**: Port conflict error when starting the app

    **Solution**:

    * Find the process using the port:
      * Linux/macOS: `lsof -i :8001`
      * Windows: `netstat -ano | findstr :8001`
    * Kill the process:
      * Linux/macOS: `kill -9 <PID>`
      * Windows: `taskkill /PID <PID> /F`
    * Or modify the port number in the app configuration file (`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`)
  </div>

  <div className="fd-step">
    ### Go Build Failed

    **Problem**: Go module related errors during build

    **Solution**:

    ```
    # Set proxy
    # Linux/macOS
    export GOPROXY=https://goproxy.cn,direct
    # Windows PowerShell
    $env:GOPROXY = "https://goproxy.cn,direct"

    # Clean Go module cache
    go clean -modcache

    # Reinstall dependencies
    cd transcriber_demo
    tman run build
    ```
  </div>

  <div className="fd-step">
    ### Python Dependencies Installation Failed

    **Problem**: pip installation timeout or failure

    **Solution**: Use a mirror source (for users in China)

    ```bash
    pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
    ```
  </div>

  <div className="fd-step">
    ### Azure Speech Service Error

    **Problem**: Azure Speech Service related errors, such as authentication failure

    **Solution**: Check that the configuration in your `.env` file is correct, and ensure `AZURE_STT_KEY` and `AZURE_STT_REGION` are filled in accurately
  </div>

  <div className="fd-step">
    ### Permission Error on Windows

    **Problem**: PermissionError when accessing files on Windows

    **Solution**: Right-click PowerShell and select "Run as Administrator"
  </div>
</div>

## Get Help

* **GitHub Issues**: [https://github.com/TEN-framework/ten-framework/issues](https://github.com/TEN-framework/ten-framework/issues)
* **Documentation**: [https://theten.ai/docs](https://theten.ai/docs)
* **Contributing Guide**: [contributing.md](../code-of-conduct/contributing.md)


# WebSocket Voice Assistant Quickstart (/docs/ten_framework/getting-started/websocket-voice-assistant-quick-start)

## WebSocket Voice Assistant Quickstart

Build and run a real-time WebSocket voice assistant in under 10 minutes.

## Overview

This guide helps you run a complete voice assistant demo locally. It supports real-time voice conversations and tool calling. The demo includes:

| Component | Vendor     | Purpose                   |
| --------- | ---------- | ------------------------- |
| ASR       | Deepgram   | Speech-to-text            |
| LLM       | OpenAI     | Response generation       |
| TTS       | ElevenLabs | Text-to-speech            |
| Tool      | WeatherAPI | Weather lookup (optional) |

## Requirements

### Prerequisites

Follow the [TEN Framework Quick Start](./quick-start.md) to install the base environment, then make sure the check below passes:

```bash
tman check env
```

Expected output (at least Python and Go are Ready):

```text
✅ Operating System: Supported
✅ Python:   Ready
✅ Go:       Ready
```

### Extra tools

Install the following tools for build & run:

#### Task (task runner)

**Linux / macOS:**

```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version  # verify
```

**Windows:**

```powershell
# Option 1: Install via winget (recommended)
# Install winget from: https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# Requires Windows 10 version 1709 (Build 16299) or higher, or Windows 11
winget --version  # verify winget
winget install Task.Task
task --version  # verify

# Option 2: Install via scoop
irm get.scoop.sh | iex
scoop --version  # verify scoop
scoop install task
task --version  # verify
```

#### Bun (JavaScript package manager)

**Linux / macOS:**

```bash
curl -fsSL https://bun.sh/install | bash
bun --version  # verify
```

**Windows:**

```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version  # verify
```

#### uv (Python package manager)

**Linux / macOS:**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version  # verify
```

**Windows:**

```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version  # verify
```

> 💡 After installation, reload your shell config:
>
> * Linux/macOS: `source ~/.zshrc`
> * Windows: reopen your PowerShell terminal

## Quickstart

<div className="fd-steps">
  <div className="fd-step">
    ### Clone the repo

    ```bash
    git clone https://github.com/TEN-framework/ten-framework.git
    cd ten-framework/ai_agents/agents/examples/websocket-example
    ```
  </div>

  <div className="fd-step">
    ### Install dependencies

    ```bash
    task install
    ```

    > ⏱️ Estimated time: 2–3 minutes

    This command will:

    * Install TEN extension packages (45+)
    * Build the Go app and API server
    * Install Python and frontend dependencies
  </div>

  <div className="fd-step">
    ### Configure API keys

    Configure vendor keys in `ai_agents/.env`:

    **Linux / macOS:**

    ```bash
    cd ../../../  # back to ai_agents
    vim .env
    ```

    **Windows:**

    ```powershell
    cd ..\..\..   # back to ai_agents
    notepad .env
    ```

    Copy all content from .env.example, then fill in:

    ```bash
    # Deepgram - ASR
    DEEPGRAM_API_KEY=your_deepgram_api_key

    # OpenAI - LLM
    OPENAI_API_KEY=your_openai_api_key
    OPENAI_MODEL=your_openai_model

    # ElevenLabs - TTS
    ELEVENLABS_TTS_KEY=your_elevenlabs_api_key

    # WeatherAPI - Tool (optional)
    WEATHERAPI_API_KEY=your_weatherapi_key
    ```
  </div>

  <div className="fd-step">
    ### Start services

    ```bash
    cd agents/examples/websocket-example
    task run
    ```

    After startup, the following services are available:

    * **Frontend UI**: [http://localhost:3000](http://localhost:3000)
    * **API server**: [http://localhost:8080](http://localhost:8080)
    * **TMAN Designer**: [http://localhost:49483](http://localhost:49483)
  </div>

  <div className="fd-step">
    ### Talk to your assistant

    Open [http://localhost:3000](http://localhost:3000), click the microphone button, and start speaking.

    Try:

    * "Hello, who are you?"
    * "What's the weather in Beijing today?" (requires WeatherAPI)
    * "Tell me a joke."

    **Congratulations!** Your TEN Agent voice assistant is running.
  </div>
</div>

## Advanced

### Swap extensions

TEN Framework lets you swap ASR/LLM/TTS extensions flexibly.

#### Option 1: Use TMAN Designer (recommended)

Open [http://localhost:49483](http://localhost:49483), then in the visual editor:

1. Click an extension node to inspect it
2. Choose a replacement extension
3. Configure parameters
4. Save and restart

#### Option 2: Edit configuration file

Edit `tenapp/property.json` and modify the extension configs under `predefined_graphs`.

##### Example: switch ASR to Azure

```json
{
  "type": "extension",
  "name": "stt",
  "addon": "azure_asr_python",
  "extension_group": "stt",
  "property": {
    "params": {
      "key": "${env:AZURE_STT_KEY}",
      "region": "${env:AZURE_STT_REGION}",
      "language": "en-US"
    }
  }
}
```

##### Example: switch TTS to Minimax

```json
{
  "type": "extension",
  "name": "tts",
  "addon": "minimax_tts_websocket_python",
  "extension_group": "tts",
  "property": {
    "params": {
      "api_key": "${env:MINIMAX_TTS_API_KEY|}",
      "group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
      "model": "speech-02-turbo",
      "audio_setting": {
        "sample_rate": 16000
      },
      "voice_setting": {
        "voice_id": "female-shaonv"
      }
    }
  }
}
```

Restart after changes (`Ctrl+C` then `task run`).

### Monitoring (Grafana)

Use Grafana to observe application metrics and logs.

<div className="fd-steps">
  <div className="fd-step">
    #### Start monitoring stack

    **Linux / macOS:**

    ```bash
    cd tools/grafana-monitoring
    docker-compose -f docker-compose.push.yml up -d
    ```

    **Windows:**

    ```powershell
    cd tools\grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    # If Docker is not installed, install it from https://apps.microsoft.com/detail/xp8cbj40xlbwkx, restart your PC, then launch Docker Desktop
    ```

    Services:

    * **Grafana** (port 3001) - UI
    * **Prometheus** (port 9091) - metrics store
    * **Loki** (port 3100) - log store
    * **OpenTelemetry Collector** (ports 4317/4318) - OTLP receiver
  </div>

  <div className="fd-step">
    #### Configure app telemetry

    Edit `agents/examples/websocket-example/tenapp/property.json`:
  </div>
</div>

#### Logs (export)

Add to the `ten.log.handlers` array:

```json
{
  "matchers": [{ "level": "info" }],
  "formatter": { "type": "json", "colored": false },
  "emitter": {
    "type": "otlp",
    "config": {
      "endpoint": "http://localhost:4317",
      "service_name": "websocket-example"
    }
  }
}
```

#### Metrics (export)

Add to `ten.services`:

```json
{
  "telemetry": {
    "enabled": true,
    "metrics": {
      "enabled": true,
      "exporter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
  }
}
```

<div className="fd-steps">
  <div className="fd-step">
    #### Restart and open Grafana

    ```bash
    cd ../../../agents/examples/websocket-example
    task run
    ```

    Open [http://localhost:3001](http://localhost:3001) (username/password: `admin/admin`).
  </div>
</div>

#### Metrics (Dashboards)

Go to `Dashboards` → **"TEN Framework"** to view:

* Extension startup latency
* Cmd processing latency
* Message queue wait time

#### Logs (Explore/Loki)

Go to `Explore` → select `Loki`, then query:

* `{service_name="websocket-example"}` - all logs
* `{service_name="websocket-example", ten_extension_name="stt"}` - ASR logs
* `{service_name="websocket-example", ten_extension_name="llm"}` - LLM logs
* `{service_name="websocket-example", ten_extension_name="tts"}` - TTS logs

> 💡 Use the `ten_extension_name` label to filter logs per extension.

Stop the monitoring stack:

**Linux / macOS:**

```bash
cd tools/grafana-monitoring
docker-compose -f docker-compose.push.yml down
```

**Windows:**

```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```

## Troubleshooting

<details>
<summary><strong>Frontend dependency install fails</strong></summary>

If `bun install` fails due to missing versions, switch to the official npm registry:

```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```

</details>

<details>
<summary><strong>macOS Python library loading fails</strong></summary>

Set the Python dynamic library path:

```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# Add to ~/.zshrc for persistence
```

</details>

<details>
<summary><strong>Windows Python library loading fails</strong></summary>

Make sure the Python 3.10 installation path is in your system PATH:

```powershell
# Check Python path
python -c "import sys; print(sys.prefix)"

# Add Python DLL directory to PATH (adjust to your actual path)
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```

If you didn't check "Add Python to PATH" during installation, re-run the installer and choose "Modify" to add it.

</details>

<details>
<summary><strong>Port already in use</strong></summary>

Find and kill the process:

**Linux / macOS:**

```bash
lsof -i :3000  # or :8080
kill -9 <PID>
```

**Windows:**

```powershell
netstat -ano | findstr :3000   # or :8080
taskkill /PID <PID> /F
```

</details>

<details>
<summary><strong>Go module related errors</strong></summary>

```bash
# Set proxy
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"

# Clean Go module cache
go clean -modcache

# Reinstall dependencies
cd ai_agents/agents/examples/websocket-example
task install
```

</details>

## Next steps

* **[Extension development guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext)** - build custom extensions
* **[More examples](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - explore other agent demos
* **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - report bugs or request features
* **[Documentation hub](https://theten.ai/docs)** - full TEN Framework documentation


# Graph (/docs/ten_framework/graph/graph)

In the TEN framework, there are two types of graphs:

1. Dynamic Graph
2. Predefined Graph (`predefined_graph`)

|               | Dynamic Graph                                       | Predefined Graph                                                       |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------- |
| Startup Time  | When the TEN app receives the `start_graph` command | When the TEN app starts, or when it receives the `start_graph` command |
| Graph Content | Specified in the `start_graph` command              | Predefined in the TEN app's properties                                 |
| Graph ID      | Randomly generated UUID                             | Randomly generated UUID                                                |

![Two types of graphs](https://ten-framework-assets.s3.amazonaws.com/doc-assets/two_types_of_graph.png)

Predefined graphs have an `auto_start` property to determine whether the graph automatically starts when the TEN app launches.

Additionally, predefined graphs have a `singleton` property that indicates whether only *one* instance of the graph can be generated within the TEN app.

## Graph ID and Graph Name

For each graph instance, the TEN app internally uses a unique UUID4 string as an identifier, called the **Graph ID**.

For predefined graphs, you can assign a meaningful and easy-to-remember name, called the **Graph Name**. When you need to specify a particular predefined graph, you can directly use its graph name. If a predefined graph has the `singleton` property, it means that only one instance of this predefined graph can exist within the TEN app. In this case, the TEN runtime uses the graph name to uniquely identify the single instance generated from that predefined graph.

## Dynamic Graph

When a TEN app receives a `start_graph` command and creates a dynamic graph, the system assigns a random UUID as the graph's ID. If other clients obtain this graph ID, they can also connect to this graph.

Dynamic Graph ID example:

`123e4567-e89b-12d3-a456-426614174000`

## Predefined Graph

Predefined graphs work similarly to dynamic graphs, with the main difference being how the content is defined. The content of a dynamic graph is included in the `start_graph` command, while the content of a predefined graph is defined in advance by the TEN app. Clients only need to specify the name of the predefined graph in the `start_graph` command to start it.

The main advantage of predefined graphs is simplifying usage and protecting sensitive information. With predefined graphs, clients don't need to know the detailed structure of the graph, which both improves usability and avoids exposing potentially sensitive information contained in the graph.

Predefined Graph name example:

`http_server`

When a TEN app starts, all predefined graphs with the `auto_start` property set will automatically start.

## Graph Definition

Whether it's a dynamic graph or a predefined graph, the definition structure is the same:

```json
{
  "nodes": [
    // Definition of nodes
  ],
  "connections": [
    // Definition of communication links
  ]
}
```

Key points:

1. If there is only one TEN app, you can omit the `app` field. If there are multiple apps, you must explicitly specify the `app` field.

2. The `nodes` field defines the nodes in the graph, such as various extensions.

3. Each node can appear only once in the `nodes` field. If the same node appears multiple times, the TEN framework will report an error during validation.

4. The way to define an extension in the `nodes` field is as follows, where the `property` field is optional:

   ```json
   {
     "type": "extension",
     "name": "simple_http_server_cpp",
     "addon": "simple_http_server_cpp",
     "extension_group": "default_extension_group",
     "app": "msgpack://127.0.0.1:8001/",
     "property": {
       "root_key": "player",
       "extra_keys": ["playerName"]
     }
   }
   ```

   The `addon` field indicates that the extension is an instance generated by the corresponding plugin.

5. The `connections` field defines the communication links between nodes in the graph, where the `extension` value represents the name of the corresponding node.

Complete example:

```json
{
  "nodes": [
    {
      "type": "extension",
      "app": "msgpack://127.0.0.1:8001/",
      "name": "simple_http_server_cpp",
      "addon": "simple_http_server_cpp",
      "extension_group": "default_extension_group",
      "property": {
        "root_key": "player",
        "extra_keys": ["playerName"]
      }
    }
  ],
  "connections": [
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "simple_http_server_cpp",
      "cmd": [
        {
          "name": "start",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        },
        {
          "name": "stop",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        }
      ]
    },
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "gateway",
      "cmd": [
        {
          "name": "push_status_online",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "uap"
            }
          ]
        }
      ]
    }
  ]
}
```

## Predefined Graph Definition

Predefined graphs are defined under the `predefined_graphs` field in the TEN app's configuration file (`property.json`), including properties such as `name`, `auto_start`, and `singleton`:

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        // Complete graph definition
      }
    ]
  }
}
```

Complete example:

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        "nodes": [
          {
            "type": "extension",
            "name": "simple_http_server_cpp",
            "addon": "simple_http_server_cpp",
            "extension_group": "default_extension_group",
            "property": {
              "root_key": "player",
              "extra_keys": [
                "playerName"
              ]
            }
          }
        ],
        "connections": [
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "simple_http_server_cpp",
            "cmd": [
              {
                "name": "start",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              },
              {
                "name": "stop",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              }
            ]
          },
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "gateway",
            "cmd": [
              {
                "name": "push_status_online",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "uap"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## `start_graph` Command Definition

Dynamic graphs are created by sending a `start_graph` command to the TEN app, placing the graph definition in the command's `ten` field:

```json
{
  "ten": {
    "type": "start_graph",
    // Complete graph definition
  }
}
```

Complete example:

```json
{
  "ten": {
    "type": "start_graph",
    "nodes": [
      {
        "type": "extension",
        "name": "simple_http_server_cpp",
        "addon": "simple_http_server_cpp",
        "extension_group": "default_extension_group",
        "property": {
          "root_key": "player",
          "extra_keys": ["playerName"]
        }
      }
    ],
    "connections": [
      {
        "app": "msgpack://127.0.0.1:8001/",
        "extension": "simple_http_server_cpp",
        "cmd": [
          {
            "name": "start",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          },
          {
            "name": "stop",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          }
        ]
      },
      {
        "extension": "gateway",
        "cmd": [
          {
            "name": "push_status_online",
            "dest": [
              {
                "extension": "uap"
              }
            ]
          }
        ]
      }
    ]
  }
}
```

### Using `import_uri` to Reference Graph Definitions

When defining predefined graphs, you can use the `import_uri` field to reference external graph definition files:

```json
{
  "ten": {
    "uri": "http://localhost:8001",
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": false,
        "import_uri": "../graph.json"
      }
    ]
  }
}
```

`import_uri` can be a relative path, absolute path, or URL:

* Relative path: relative to the directory containing the `property.json` file
* Absolute path: relative to the root directory of the environment where the TEN app is located
* URL: treated directly as a URL path

## Graph Definition Specifications

* **`nodes` Field**: The `nodes` array must be provided in the graph definition. In contrast, while the `connections` array is optional, it is recommended to provide it to define communication between nodes.

* **Node `app` Field**: The `app` field cannot be set to `localhost`. In single-app graphs, the `app` URI should be omitted; in multi-app graphs, the `app` field value must match the `ten.uri` value in each app's `property.json`.

* **Node Uniqueness**: Each node in the `nodes` array represents a specific extension instance and must be uniquely identified by the combination of `app` and `name`. Duplicate definitions are not allowed, as in the following invalid example:

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_1",
        "extension_group": "test"
      },
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_2",
        "extension_group": "test"
      }
    ]
  }
  ```

* **Connection Definition Consistency**: All extension instances referenced in `connections` must be explicitly defined in `nodes`. The following example is invalid because `ext_2` is not defined in `nodes`:

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "ext_1",
        "addon": "addon_1",
        "extension_group": "some_group"
      }
    ],
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **Communication Link Consolidation**: All messages from the same source extension should be grouped in a single section, not scattered across multiple definitions. The following example is incorrect:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      },
      {
        "extension": "ext_1",
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  The correct approach is to consolidate all messages from the same source extension together:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ],
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **Message Destination Consolidation**: For each specific type of message, all destination extensions should be grouped under a single entry. The following example is incorrect:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          },
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  The correct approach is to consolidate all destinations for the same message together:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              },
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  Note that messages with the same name can coexist in different types (such as `cmd` and `data`) without causing conflicts.

## Subgraph

The core mechanism of the TEN framework is based on graph structures, which consist of nodes and connections. Subgraphs are a powerful reuse mechanism that allows complex graph structures to be split into reusable modules, thereby improving code organization and maintainability.

### Subgraph Design Philosophy

The essence of a graph is to define how data flows between extensions. Subgraphs do not change this fundamental principle in their implementation, but serve as syntactic sugar that is ultimately flattened into the parent graph and started using the same mechanisms as regular graphs. This design simplifies the development of complex systems while not adding runtime complexity.

#### Design Principles

Subgraph design follows these core principles:

1. **Independence**: Any graph is a complete graph that can run independently or be embedded as a component into other graphs.

2. **Tool-Friendly**: Subgraphs provide additional information to help development tools better understand graph structures, improving the development experience while not adding runtime complexity.

3. **Flattening Mechanism**: Subgraphs are ultimately flattened into standard graph structures to ensure performance and compatibility.

4. **Simplicity**: Subgraph design aims to simplify the development process rather than add complexity, avoiding increased JSON structure complexity.

The purpose of subgraph design is to allow developers to use a graph as a black box without needing to understand its internal complexity. Therefore, at the point of referencing subgraphs, no special patching mechanisms are provided to adjust the internal state of subgraphs, to reduce complexity. If subgraph definitions need to be modified, the original definitions should be modified directly rather than patching at the reference point. This is done to avoid complexity spread and increased maintenance costs. For example, when introducing subgraph A into large graph B, if you need to adjust the extension property schema definitions inside subgraph A, you should directly modify the schema definitions of that extension rather than providing modification mechanisms at the reference point, because this might lead to adverse chain reactions, only modifying the extension's schema definitions without patching other places, thus causing errors elsewhere.

In other words, subgraphs primarily provide a mapping mechanism to expose internal elements of subgraphs to the outside, rather than providing mechanisms to patch internal elements of subgraphs.

### Subgraph Implementation Mechanism

Next, we will detail the implementation mechanism of subgraphs, including subgraph definition, reference methods, and the flattening process.

#### Subgraph Definition Example

The following is a `subgraph.json` example that can be used as an independent graph or as a subgraph referenced by other graphs:

```json
{
  "nodes": [
    {
      // Define extension node named ext_c
      "type": "extension",
      "name": "ext_c",
      "addon": "extension_c"
    },
    {
      // Define extension node named ext_d
      "type": "extension",
      "name": "ext_d",
      "addon": "extension_d"
    }
  ],
  "connections": [
    {
      // ext_c sends B command to ext_d
      "extension": "ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_d"
            }
          ]
        }
      ]
    }
  ],
  "exposed_messages": [
    // Represents message interfaces exposed by the graph to the outside, mainly for development tools to provide intelligent hints
    {
      // ext_d's B command exposed to the outside
      "extension": "ext_d",
      "type": "cmd_in",
      "name": "B"
    }
  ],
  "exposed_properties": [
    // Represents properties exposed by the graph to the outside
    {
      "extension": "ext_c",
      "name": "a",
      "alias": "a1"
    }
  ]
}
```

#### Subgraph Reference Example

Example of `graph.json` referencing subgraphs:

```json
{
  "nodes": [
    {
      // Define extension node named ext_a
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // Define extension node named ext_b
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // Reference subgraph, named subgraph_1
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "./ten_packages/extension/aaa/subgraph.json",
      "property": {
        // Properties written in this property field must exist in the subgraph's exposed_properties field,
        // otherwise it will be an error condition
        "app_id": "${env:AGORA_APP_ID}",
        "token": "<agora_token>",
        "channel": "ten_agent_test"
      }
    },
    {
      // Reference subgraph, named subgraph_2
      "type": "subgraph",
      "name": "subgraph_2",
      "import_uri": "./ten_packages/extension/bbb/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // First target is ext_b
              "extension": "ext_b"
            },
            {
              // Second target is ext_d in the subgraph, this is advanced usage
              "extension": "subgraph_1:ext_d"
            },
            {
              // Third target is the subgraph, but this requires the subgraph to explicitly expose the B cmd_in interface,
              // otherwise it will be treated as an error condition
              "subgraph": "subgraph_2"
            }
          ]
        }
      ]
    },
    {
      // ext_c in the subgraph sends cmd H to ext_a, this is advanced usage
      "extension": "subgraph_1:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Subgraph subgraph_2 sends cmd H to ext_a, but this requires the subgraph to expose the H
      // cmd_out interface, otherwise it will be treated as an error condition
      "subgraph": "subgraph_2",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

#### Key Concepts

The subgraph mechanism introduces three key concepts:

1. **Message Exposure** (exposed\_messages):
   * Subgraphs declare externally exposed interfaces through the `exposed_messages` field
   * Mainly used by development tools to implement intelligent hints and checks
   * Hides internal details of subgraphs, improving development experience

2. **Property Exposure** (exposed\_properties):
   * Subgraphs declare externally exposed properties through the `exposed_properties` field
   * Mainly used by development tools to implement intelligent hints and checks
   * Hides internal details of subgraphs, improving development experience

3. **Subgraph Reference and Naming**:
   * Reference other graph files through `"type": "subgraph"`
   * Each subgraph has a unique identifier name serving as a namespace
   * Prevents conflicts between elements with the same name in different subgraphs

4. **Cross-Graph Connections**:
   * Reference elements within subgraphs through namespace syntax (such as `subgraph_1:ext_d`), or reference subgraphs through the `subgraph` field
   * Makes elements within subgraphs interact with the main graph
   * Builds complex cross-graph message flows

In summary, the `exposed_messages` and `exposed_properties` fields describe how messages flow into the graph from the graph boundary, and development tools use these fields to provide intelligent hints, making it convenient for developers to specify how messages flow to the graph boundary.

#### Flattening Mechanism

Ultimately, graphs referencing subgraphs are flattened into regular graph structures, ensuring runtime uniformity and efficiency. However, this flattening result is not seen by users, but is flattened into memory for use in the TEN runtime or TEN manager. Note that in the flattened graph definition, there are the original extensions and the connections between these extensions.

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // ext_c from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "extension_c"
    },
    {
      // ext_d from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "extension_d",
      "property": {
        // The property field defined when referencing the subgraph will be flattened into the corresponding extension's property field
        "app_id": "${env:AGORA_APP_ID}"
      }
    },
    {
      // ext_e from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "extension_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b"
            },
            {
              "extension": "subgraph_1_ext_d"
            },
            {
              "extension": "subgraph_2_ext_e"
            }
          ]
        }
      ]
    },
    {
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Internal connections within subgraphs are also flattened and included
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "subgraph_1_ext_d"
            }
          ]
        }
      ]
    }
  ]
  // exposed_messages and exposed_properties fields are discarded during flattening
}
```

The subgraph flattening mechanism follows these rules:

1. Before flattening, the colon (`:`) symbol indicates that an element is located within a subgraph (such as `subgraph_1:ext_d`).

2. After flattening, elements within subgraphs have the subgraph name added as a prefix to their names (such as `subgraph_1_ext_c`), ensuring global uniqueness.

3. The flattened graph definition no longer contains colon symbols, distinguishing between pre-flattening and post-flattening states.

4. Internal connections within subgraphs are preserved and included in the flattened graph, ensuring functional completeness.

### Advanced Features and Applications

As project complexity increases, advanced features of subgraphs can help better organize and manage systems.

#### Message Conversion and Subgraphs

Subgraphs fully support the message conversion (msg\_conversion) mechanism for handling message format conversion between different interfaces:

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "http://a.b.c.d/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1:ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "subgraph": "subgraph_2",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Development tools can use `exposed_messages` information to suggest compatibility and provide message conversion configuration interfaces. After configuration, conversion rules are automatically written into the graph definition, simplifying the development process.

After flattening, message conversion rules are correctly preserved, ensuring runtime behavior matches design intent:

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "addon_c"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "addon_d"
    },
    {
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "addon_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1_ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_2_ext_e",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Through this approach, developers can flexibly handle message format differences between different components without needing to understand underlying implementation details.

## Meaning of "subgraph\_1:ext\_c"

In the previous examples, `subgraph_1` in `subgraph_1:ext_c` does not represent the subgraph itself, but references an element in the `nodes` field. In subgraph applications, `subgraph_1` can be viewed as a namespace.

Through this mechanism, you can specify extensions at different locations as sources or targets in graph connections, enabling connections across multiple graphs.

The core concept is: graph connections are essentially connections between extensions. Through this mechanism, you can specify elements of different types in the `nodes` field as sources or targets of connections, enabling connections across multiple subgraphs or even multiple graphs.

## Connections Across Multiple Graphs

In addition to subgraphs, you can also specify connections to other graphs within a single graph through similar mechanisms.

### Connecting to Predefined Graphs in the Same TEN App

Example: How to connect to predefined graphs within the same TEN app in a graph:

```json
{
  "nodes": [
    {
      // Define extension named ext_a
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // Define extension named ext_b
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // Reference other graph, graph ID is default
      "type": "graph",
      "graph_id": "default"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // First target is ext_b
              "extension": "ext_b"
            },
            {
              // Second target is ext_d in another graph
              "extension": "default:ext_d"
            }
          ]
        }
      ]
    },
    {
      // ext_c in another graph sends cmd H to ext_a
      "extension": "default:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Another graph sends cmd X to ext_a, but this requires the other graph to expose the X cmd_out interface,
      // otherwise it will be treated as an error condition. This error condition can be reported during static
      // checking if it can be statically detected, or at runtime.
      "graph": "default",
      "cmd": [
        {
          "name": "X",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

### Types of Graph Nodes

1. Subgraph

   ```json
   {
     "type": "subgraph",
     "name": "subgraph_1", // Meaning of namespace
     "import_uri": "http://a.b.c.d/subgraph.json"
   }
   ```

2. Local graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": true
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": false
   }
   ```

3. Remote graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": true,
     "app": "msgpack://127.0.0.1:8002/"
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": false,
     "app": "msgpack://127.0.0.1:8002/"
   }
   ```


# Message Conversion (/docs/ten_framework/graph/msg_conversion)

## Message Conversion

TEN Framework provides powerful message conversion capabilities that allow you to transform and modify message content during message passing. By configuring conversion rules, you can achieve message field renaming, value replacement, and complex data structure transformations.

### Basic Configuration

Message conversion is configured through the `msg_conversion` configuration item:

```json
{
  "msg_conversion": {
    "type": "per_property",
    "keep_original": false,
    "rules": [
      {
        "path": "user.name",
        "conversion_mode": "fixed_value",
        "value": "default_user"
      },
      {
        "path": "data.items[0].id",
        "conversion_mode": "from_original",
        "original_path": "original_id"
      }
    ]
  }
}
```

### Configuration Parameters

* **type**: Conversion type, currently supports `per_property` (per-property conversion)
* **keep\_original**: Whether to retain original message content; when set to `false`, only converted fields are preserved
* **rules**: Array of conversion rules that define specific conversion logic

### Conversion Rules

Each conversion rule contains the following fields:

* **path**: Target field path, specifying the message field location to be set
* **conversion\_mode**: Conversion mode
  * `fixed_value`: Set a fixed value
  * `from_original`: Copy value from a specified path in the original message
* **value**: Fixed value used when conversion mode is `fixed_value`
* **original\_path**: When conversion mode is `from_original`, specifies the source field path in the original message

### Path Expressions

Path expressions in message conversion support multiple access methods:

* **Object Property Access**: `user.profile.name` - Access properties of nested objects
* **Array Index Access**: `items[0].name` - Access elements at specific array indices
* **Mixed Path Access**: `user.tags[1].value` - Combine object and array access

### Use Cases

Message conversion functionality is suitable for the following scenarios:

* **Field Renaming**: Convert field names in messages to formats required by target systems
* **Data Format Conversion**: Adjust message data structures to adapt to different processing modules
* **Default Value Setting**: Provide default values for missing fields
* **Data Cleansing**: Filter or transform data content that doesn't meet requirements


# Go (/docs/ten_framework/language_binding/go)

Let's build a Go application using the TEN framework.

## Creating the TEN App

First, we'll create a basic TEN Go app by integrating several pre-built TEN packages. Follow these steps:

```shell title="Terminal"
tman install app default_app_go
cd default_app_go

tman install protocol msgpack
tman install extension_group default_extension_group
```

### Installing a Default TEN Extension

Next, install a default TEN extension written in Go:

```shell title="Terminal"
tman install extension default_extension_go
```

### Declaring the Prebuilt Graph for Auto-Start

Now, we'll modify the `property.json` file of the TEN app to include a graph declaration. This will ensure the default extension starts automatically when the TEN app is launched.

```json title=".json"
"predefined_graphs": [
  {
    "name": "default",
    "auto_start": true,
    "nodes": [
      {
        "type": "extension_group",
        "name": "default_extension_group",
        "addon": "default_extension_group"
      },
      {
        "type": "extension",
        "name": "default_extension_go",
        "addon": "default_extension_go",
        "extension_group": "default_extension_group"
      }
    ]
  }
]
```

## Building the App

Unlike standard Go projects, the TEN Go app uses CGo, so you need to set up certain environment variables before building. A build script is already provided in the TEN runtime Go binding system package, so you can build the app with a single command:

```shell title="Terminal"
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```

The compiled binary, `main`, will be generated in the `/bin` folder.

## Starting the App

Since some environment variables need to be set, it is recommended to start the app using the provided script:

```shell title="Terminal"
./bin/start
```

## Debugging

If you are using Visual Studio Code (VSCode) as your development environment, you can use the following configurations to debug Go/C code.

### Debugging Go Code

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Go app (launch)",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "cwd": "${workspaceFolder}",
      "output": "${workspaceFolder}/bin/tmp",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

### Debugging C Code

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "C (launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/main",
      "cwd": "${workspaceFolder}",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

## CGO

### Generated Code

When interfacing Go with C, the cgo tool is crucial. When a C function is called from Go, cgo converts the Go source files into multiple output files, including both Go and C source files. These generated C source files are then compiled using compilers like gcc or clang.

Use the following command to generate these C/Go source files:

```bash title="Terminal"
mkdir out
go tool cgo -objdir out
```

Example:

```bash title="Terminal"
go tool cgo -objdir out cmd.go error.go
```

The generated files include:

```bash title="Terminal"
├── _cgo_export.c
├── _cgo_export.h
├── _cgo_flags
├── _cgo_gotypes.go
├── _cgo_main.c
├── _cgo_.o
├── cmd.cgo1.go
├── cmd.cgo2.c
├── error.cgo1.go
└── error.cgo2.c
```

* **\_cgo\_export.h** is key to the interoperability between Go and C within the cgo context. It contains necessary declarations for Go functions accessible from C.

  This file is automatically generated based on Go functions explicitly exported using the `//export` directive. It also includes definitions for C types corresponding to the Go types used in these functions, ensuring compatibility between the two languages.

  Example of type definitions:

```c title=".h"
typedef signed char GoInt8;
typedef unsigned char GoUint8;
typedef short GoInt16;
typedef unsigned short GoUint16;
typedef int GoInt32;
typedef unsigned int GoUint32;
typedef long long GoInt64;
typedef unsigned long long GoUint64;
typedef GoInt64 GoInt;
typedef GoUint64 GoUint;
typedef size_t GoUintptr;
typedef float GoFloat32;
typedef double GoFloat64;
```

* **\_cgo\_gotypes.go** contains the corresponding Go types defined in C and used in Go.

  For example, if you define a struct `ten_go_error_t` in a header file `common.h` and use `C.ten_go_error_t` in Go, there will be a corresponding Go type in `_cgo_gotypes.go`:

```go title=".go"
package ten

type _Ctype_ten_go_status_t = _Ctype_struct_ten_go_status_t

type _Ctype_struct_ten_go_status_t struct {
    err_no   _Ctype_int
    msg_size _Ctype_uint8_t
    err_msg  [256]_Ctype_char
    _        [3]byte
}
```

* **cmd.cgo1.go** is a cgo-generated counterpart to the original Go source file `cmd.go`. It replaces direct calls to C functions and types with calls to the generated Go functions and types provided by cgo.

  Example:

```go title=".go"
package ten

  func NewCmd(cmdName string) (Cmd, error) {
      // ...
      cStatus := func() _Ctype_struct_ten_go_status_t {
          var _cgo0 _cgo_unsafe.Pointer = unsafe.Pointer(unsafe.StringData(cmdName))
          var _cgo1 _Ctype_int = _Ctype_int(len(cmdName))
          _cgoBase2 := &bridge
          _cgo2 := _cgoBase2
          _cgoCheckPointer(_cgoBase2, 0 == 0)
          return _Cfunc_ten_go_cmd_create_custom_cmd(_cgo0, _cgo1, _cgo2)
      }()
      // ...
  }
```

* **cmd.cgo2.c** is a wrapper of the original C function called from Go.

  Example:

```c title=".c"
  CGO_NO_SANITIZE_THREAD void _cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd(void *v) {
      struct {
          const void* p0;
          int p1;
          char __pad12[4];
          ten_go_msg_t** p2;
          ten_go_error_t r;
      } __attribute__((__packed__, __gcc_struct__)) *_cgo_a = v;
      char *_cgo_stktop = _cgo_topofstack();
      __typeof__(_cgo_a->r) _cgo_r;
      _cgo_tsan_acquire();
      _cgo_r = ten_go_cmd_create_cmd(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2);
      _cgo_tsan_release();
      _cgo_a = (void*)((char*)_cgo_a + (_cgo_topofstack() - _cgo_stktop));
      _cgo_a->r = _cgo_r;
      _cgo_msan_write(&_cgo_a->r, sizeof(_cgo_a->r));
  }
```

So, the calling sequence of `C.ten_go_cmd_create_cmd()` from Go is:

```text
_Cfunc_ten_go_cmd_create_custom_cmd (auto-generated Go)
           |
           V
   _cgo_runtime_cgocall (Go)
           |
           V
      entrysyscall     // switch Go stack to C stack
           |
           V
_cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd (auto-generated C)
           |
           V
   ten_go_cmd_create_cmd (C)
           |
           V
      exitsyscall
```

### Incomplete Type

As mentioned earlier, the cgo tool generates corresponding Go types in `_cgo_gotypes.go` based on C header files imported via `import "C"`. If a struct is opaque in the C header, the cgo tool generates an incomplete type instead.

Example of an opaque C struct:

```c title=".h"
typedef struct ten_go_msg_t ten_go_msg_t;
```

The cgo tool will generate an incomplete type in Go:

```go title=".go"
type _Ctype_ten_go_msg_t = _Ctype_struct_ten_go_msg_t
type _Ctype_struct_ten_go_msg_t _cgopackage.Incomplete
```

#### What happens if you use the incomplete type in Go?

* **Incomplete types cannot be allocated.**

  Since `sys.NotInHeap` cannot be allocated on the Go heap, operations like `new` or `make` won't work. Attempting to create a new instance of an opaque struct in Go will result in a compiler error:

```go title=".go"
msg := new(C.ten_go_msg_t) // Error: can't be allocated in Go; it is incomplete (or unallocatable)
```

* **Pointers to incomplete types cannot be passed to C directly.**

  If you have a C function with a pointer to an opaque struct as a parameter, passing a Go pointer to this incomplete type directly to a C function will not work according to cgo rules. The Go compiler will require the pointer to be "pinned" to ensure it adheres to Go's garbage collector (GC) constraints.

### Rules for Using C.uintptr\_t Instead of a Pointer to an Opaque Struct

* **Benefits:**

  * `C.uintptr_t` and `uintptr` in Go are integers large enough to hold a C pointer, avoiding memory allocation or conversion when passing from Go to C.

* **Limitations:**
  * `uintptr` is an integer, not a pointer. It represents the bit pattern of an address with no pointer arithmetic.
  * `uintptr` cannot be dereferenced in Go. Converting it to `unsafe.Pointer` can cause issues with Go's GC.
  * Since `uintptr` is an integer, `nil` or `NULL` cannot be passed to C. Use `0` instead of `nil` to represent a null address.


# Manifest (/docs/ten_framework/metadata_system/manifest)

## What is a Manifest File

A Manifest file is a crucial configuration file in the TEN Framework used to store metadata for software packages. It defines essential information about Extensions or Apps, including basic information, dependencies, API interfaces, script configurations, and other key details.

## Purpose of Manifest Files

* **Package Identification**: Define the package type, name, and version information
* **Dependency Management**: Declare required dependencies and their version requirements
* **API Definition**: Describe data interfaces and command interfaces provided by the package
* **Package Configuration**: Specify files to be included in the release version
* **Script Management**: Define automation scripts for building, testing, and other tasks

## Extension Manifest Example

Here's a complete Manifest example for an AI chat extension:

```json
{
  "type": "extension",
  "name": "openai_chatgpt_python",
  "version": "0.1.0",
  "tags": ["ai", "llm", "python"],
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "^0.8.0"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "*.py",
      "README.md"
    ]
  },
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        }
      }
    },
    "data_in": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "data_out": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_in": [
      {
        "name": "flush"
      },
      {
        "name": "tool_register",
        "property": {
          "properties": {
            "tool": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "description": {
                  "type": "string"
                },
                "parameters": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                }
              },
              "required": ["name", "description", "parameters"]
            }
          }
        },
        "result": {
          "property": {
            "properties": {
              "response": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "tool_call",
        "property": {
          "properties": {
            "name": {
              "type": "string"
            },
            "args": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    ]
  },
  "scripts": {
    "test": "python -m pytest tests/",
    "build": "python setup.py build"
  }
}
```

## App Manifest Example

Here's a Manifest example for a voice assistant application:

```json
{
  "type": "app",
  "name": "voice_assistant_app",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "^0.8.0"
    }
  ],
  "api": {
    "property": {
      "properties": {
        "app_id": {
          "type": "string"
        },
        "log_level": {
          "type": "uint8"
        },
        "server_config": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "int32"
            },
            "ssl_enabled": {
              "type": "bool"
            }
          },
          "required": ["host", "port"]
        }
      }
    }
  }
}
```

## Key Field Descriptions

### Basic Information

* **type**: Package type, can be `extension` or `app`
* **name**: Unique identifier name for the package
* **version**: Version number following semantic versioning standards
* **tags**: Array of tags for categorization and search

### Dependency Configuration

* **dependencies**: List of dependencies, each containing type, name, and version requirements

### Package Configuration

* **package.include**: Specifies files and directories to include during packaging

### API Configuration

* **api.property**: Defines configuration properties for the extension
* **api.data\_in/data\_out**: Defines data input and output interfaces
* **api.cmd\_in/cmd\_out**: Defines command input and output interfaces

### Script Configuration

* **scripts**: Defines various automation scripts such as testing, building, etc.

## Best Practices

1. **Version Management**: Use semantic versioning to ensure version consistency
2. **Dependency Declaration**: Clearly declare all required dependencies and their version ranges
3. **API Design**: Clearly define input and output interfaces for easy integration with other components
4. **Complete Documentation**: Ensure that configurations in the Manifest are consistent with actual implementation


# Metadata System (/docs/ten_framework/metadata_system/metadata_system)

The TEN framework utilizes a consistent metadata system across various types of packages, including:

* App
* Extension Group
* Extension
* Protocol
* System

## Types of Metadata

In the TEN metadata system, there are two primary types of metadata:

### 1. **Manifest**

* **Location**: Stored in the root directory of the associated TEN package with the file name `manifest.json`.
* **Contents**:
  * **Package Name**: The name of the TEN package.
  * **Package Version**: The version of the TEN package, following semantic versioning.
  * **API Schema**: Defines schemas related to the package's properties and input/output messages.
    * **Property Schema**: These schemas are often defined in a `property.json` file in the root directory. The manifest can specify the schema for these properties.
    * **Message Schema**: Defines the schema for input/output messages handled by the package.

> ⚠️ **Note**: The API schema in `manifest.json` is *not* a JSON schema. Instead, it describes the metadata of TEN values, which are central to the TEN runtime, while JSON is merely a representation format.

### 2. **Property**

* **Location**: Typically stored in a `property.json` file in the root directory of the TEN package.
* **Purpose**: The `property.json` file stores initial property values, which are read-write during runtime. This means that properties can be modified while the TEN runtime is executing.

#### Example of `property.json`

```json
{
  "Darth Vader": "I am your father"
}
```

## Manifest File

The `manifest.json` file serves as the blueprint for the TEN package, defining its metadata, properties, and the input/output messages it handles.

### Example of `manifest.json`

```json
{
  "type": "app",
  "name": "default_app_cpp",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.0"
    }
  ],
  "api": {
    "property": {
      "exampleInt8": {
        "type": "int8"
      },
      "exampleString": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "detail": {
              "type": "string"
            },
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

### Purpose of TEN Schema in Manifest

The TEN schema in `manifest.json` provides the TEN runtime with metadata about the extension's external API, including the type information for properties and messages. This helps the runtime platform correctly handle the extension's properties and messages during operation.

### When TEN Schema is Used

1. **Property Validation**: When the TEN runtime gets/sets properties of a TEN package or message.
2. **Data Conversion**: When converting a JSON document to a TEN package or message property using the `from_json` API.
3. **Compatibility Check**: When the TEN manager checks if a message can be output from a source extension to a destination extension according to the TEN schema.

## Property Management

In the TEN framework, there are two types of properties:

1. **Message Properties**: These properties are associated with the messages exchanged between extensions within the framework. Message properties define the specific data or parameters that are carried within a message, such as command parameters, data payloads, or metadata related to audio/video frames.

2. **TEN Package Properties**: These properties are associated with the TEN packages themselves, such as extensions. TEN package properties define the configuration or settings specific to a package. For example, an extension might have properties that configure its behavior, such as runtime settings, initialization parameters, or other configuration data.

Both types of properties are managed within the TEN framework but serve different purposes, one focuses on the communication between components (message properties), while the other pertains to the configuration and operation of the components themselves (TEN package properties).

<figure><img src="..https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_system.png" alt=""><figcaption><p>Property system</p></figcaption></figure>

### Defining TEN Package Properties

The `property.json` file defines the properties of a TEN package. Here's an example:

```json
{
  "prop_1_name": 0,
  "prop_2_name": "prop_2_value",
  "prop_3_name": ["hello", "prop_3_sub_value"],
  "prop_4_name": {
    "prop_4_1_name": 1,
    "prop_4_2_name": "prop_4_sub_value"
  }
}
```

> ⚠️ **Note**: Each property name in the `property.json` file must be unique.

### TEN Schema for Properties

You can define a TEN schema for properties in the `manifest.json` file, enabling the TEN runtime to handle these properties more effectively. If a property does not have a corresponding TEN schema, the runtime platform will use the default JSON handling method (e.g., treating all JSON numbers as `float64`). If a TEN schema is provided, the runtime platform will use it to validate and process the property accordingly.

| Property | TEN Schema | Effect                                                                                          |
| -------- | ---------- | ----------------------------------------------------------------------------------------------- |
| Yes      | Yes        | The TEN runtime validates the property value based on the TEN schema (e.g., checking the type). |
| Yes      | No         | The TEN runtime uses default handling, treating all JSON numbers as `float64`.                  |

### Accessing Properties from an Extension

The TEN runtime provides APIs for extensions to access various properties.

### Accessing App Properties from an Extension

> ⚠️ **Note**: The behavior described in this section has not yet been implemented and is part of the future roadmap.

The TEN framework provides a powerful property access mechanism that allows extensions to read and modify app properties. Through the get/set property API, you can access all properties, including app properties. When the API parameter begins with the `app:` prefix, the system identifies this as a request to operate on app properties.

API design considerations for different language implementations:

* **Go**: Providing a synchronous version without callbacks is sufficient, making it concise and efficient.
* **Node.js**: Implemented as an async function without callbacks, aligning with Node.js's asynchronous programming model.
* **Python**: Ideally, versions both with and without callbacks should be provided. Currently, a synchronous version without callbacks can be implemented first, but accessing `app:` properties will use event wait, which may impact performance.
* **Async Python**: Implemented as an async function without callbacks, consistent with Python's asynchronous model.
* **C++**: Both versions with and without callbacks should be provided. Currently, a synchronous version without callbacks can be implemented first, but accessing `app:` properties will use event wait, which may impact performance.

This design enables CRUD operations on app properties through the unified get/set\_property API, maintaining interface consistency without affecting message property-related APIs.

Regarding the handling of `ten:` namespace fields, the following rules are adopted:

1. `get/set_property_to/from_json(nullptr)` operations do not process `ten:` fields
2. To access `ten:` fields, you must explicitly specify the complete path such as `get/set_property("ten:xxx")`
3. When using `get/set_property_to/from_json()`, `ten:` fields are only processed when a key in the form of `"ten:xxx"` is explicitly specified

This design maintains API simplicity while avoiding the complexity of creating additional commands for specific requirements.

### Specifying Property Values in the `start_graph` Command

Property values related to the TEN package can be specified in the `start_graph` command. The TEN runtime processes these properties according to the TEN schema (if defined) and stores them within the corresponding TEN package instance.

#### Example of Property Specification in `start_graph` Command

```json
{
  "nodes": [
    {
      "type": "extension_group",
      "name": "foo_extension_group",
      "addon": "foo_extension_group_addon"
    },
    {
      "type": "extension",
      "name": "bar_extension",
      "extension_group": "foo_extension_group",
      "property": {
        "prop_1_name": 0,
        "prop_2_name": "prop_2_value",
        "prop_3_name": ["hello", "prop_3_sub_value"],
        "prop_4_name": {
          "prop_4_1_name": 1,
          "prop_4_2_name": "prop_4_sub_value"
        }
      }
    }
  ]
}
```


# Property (/docs/ten_framework/metadata_system/property)

## Property File Overview

The Property file is a configuration file in the TEN Framework used to store attribute values for extensions. It defines runtime parameters and configuration options for extensions, serving as an essential component for their proper operation.

## Property File Format

Property files use JSON format and contain various configuration parameters required for extension operation, such as API keys, model configurations, runtime parameters, etc.

### Basic Property Example

Here's a simple Property file example:

```json
{
  "api_key": "your-api-key-here",
  "max_tokens": 1000,
  "temperature": 0.7,
  "model_config": {
    "name": "gpt-4",
    "version": "2024-01-01"
  },
  "allowed_languages": ["en", "zh", "es"]
}
```

### AI Model Configuration Example

Property file configuration example for AI model extensions:

```json
{
  "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxx",
  "model": "gpt-4",
  "max_tokens": 4096,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "prompt": "You are a helpful AI assistant.",
  "greeting": "Hello! How can I help you today?",
  "max_memory_length": 10,
  "vendor": "openai"
}
```

## Graph Configuration

Property files can also contain Graph Configuration, which defines connection relationships and data flow between extensions.

### Predefined Graph Configuration Example

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "nodes": [
          {
            "type": "extension",
            "name": "my_extension",
            "addon": "my_extension_addon",
            "property": {
              "setting1": "value1",
              "setting2": 42
            }
          }
        ],
        "connections": [
          {
            "extension": "source_extension",
            "cmd": [
              {
                "name": "process_cmd",
                "dest": [
                  {
                    "extension": "target_extension"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## Usage Guidelines

* Property files must be in valid JSON format
* Property values are loaded when the extension starts
* Extension behavior can be adjusted by modifying the Property file
* The graph configuration section defines connections and data flow between extensions


# Addon System (/docs/ten_framework/software_packages/addon_system)

The TEN framework is a software module system, where each software module is defined by the concept of an **addon**. Essentially, every software module in the TEN framework is an addon. For example, users can define the following types of TEN framework addons:

1. TEN **extension** addon
2. TEN **extension group** addon
3. TEN **protocol** addon

## Information Required to Load Addon

To load an addon into the TEN runtime, three pieces of information are needed:

1. **Addon name**

   It must be globally unique within the entire TEN app. Loading multiple addons with the same name into the TEN runtime is undefined behavior, and it may result in the second instance of the addon not being properly loaded into the TEN runtime.

2. **Addon folder location**

   In the TEN framework, when loading an addon, users must inform the TEN runtime where the addon's directory is located. This directory serves as the addon's `base_dir`. The TEN runtime will use this base directory to load other files related to the addon, such as the addon's `manifest.json` and `property.json` files.

   Basically, when designing the TEN framework's addon register API, it aims to automatically detect the directory where the addon is located, so the user does not need to explicitly specify the `base_dir`. However, in some special cases where the TEN framework cannot automatically detect the `base_dir`, the user will need to explicitly specify the folder where the addon is located for the TEN runtime.

3. **Addon instance itself**

## Timing for Registering Addon to TEN Runtime

There are two ways to register an addon into the TEN runtime:

1. Explicitly call the API to register the addon's `base_dir` to the TEN runtime.

   **Note** Currently, only the standalone testing framework provides such an API.

2. Place the addon in a designated location within the TEN app folder tree.

By placing the addon in the specified location, the TEN app will automatically load these addons when it starts. Currently, this designated location is under the `ten_packages/` directory in the TEN app. If the addons are placed in this format, they will be automatically loaded when the TEN app starts.

```text
.
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    └── protocol/
        └── <protocol_a>/
```


# App (/docs/ten_framework/software_packages/app)

App is the container in the TEN world, containing groups and extensions.
App can function as a process placed inside a Docker container (primarily used in the cloud).
App can also function as a thread embedded within an existing app (on client or server).
⇒ TEN can perform localized modifications (minor enhancements) on an existing project without requiring the entire project to be rewritten according to TEN.

![Run TEN as a Process or a Thread](https://ten-framework-assets.s3.amazonaws.com/doc-assets/run_ten_as_process_or_thread.png)

![Different mode of app running](https://ten-framework-assets.s3.amazonaws.com/doc-assets/different_mode_of_app_runnings.png)

## TEN App Folder Structure

In the TEN framework, developers can create TEN packages using various programming languages. Regardless of the language used, the overall folder structure of a TEN app remains consistent, ensuring uniformity across different implementations.

![TEN package folder structure](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_package_folder_structure.png)


# Extension (/docs/ten_framework/software_packages/extension)

## Life Cycle

The life cycle of an extension is divided into the following stages:

1. `on_configure`
2. `on_init`
3. `on_start`
4. `on_stop`
5. `on_deinit`

At all these stages, the extension can send messages, and if the sent message is a command type, the extension can also receive the result of the command it sent. In other words, there is no stage where the extension *cannot* send messages or receive the results of its own sent commands. So, in all these stages, the extension can send messages to other extensions, which helps implement dependencies between extensions in all stages.

Each life cycle stage corresponds to a callback function, and there is a corresponding `on_xxx_done()` function to mark the end of that life cycle stage.

| Life Cycle Callback | End of Life Cycle Function |
| ------------------- | -------------------------- |
| `on_configure`      | `on_configure_done`        |
| `on_init`           | `on_init_done`             |
| `on_start`          | `on_start_done`            |
| `on_stop`           | `on_stop_done`             |
| `on_deinit`         | `on_deinit_done`           |

### Stage Transition Graph

```mermaid
graph TD;
  on_configure((on_configure));
  on_init((on_init));
  on_start((on_start));
  on_stop((on_stop));
  on_deinit((on_deinit));

  on_configure-->on_configure_done
  on_configure_done-->on_init;
  on_configure-->on_stop;
  on_init-->on_init_done;
  on_init_done-->on_start;
  on_init-->on_stop;
  on_start-->on_stop;
  on_start-->on_start_done;
  on_stop-->on_stop_done;
  on_stop_done-->on_deinit;
  on_deinit-->on_deinit_done;
```

### on\_configure

Used to set the initial values of the extension's properties. This allows the `get_property` API from `ten_env` to be used in other lifecycle stages to retrieve properties set during the `on_configure()` stage. The `on_configure_done()` function is called to mark the end of the `on_configure` stage.

```c++
  void on_configure(ten::ten_env_t &ten_env) override {
    // Set the initial values of the extension's properties.
    ten_env.init_property_from_json(
      R"({
           "ten": {
             "uri": "msgpack://127.0.0.1:8001/",
             "log_level": 2
           }
         })", nullptr);
    ten_env.on_configure_done();
  }
```

### on\_init

Used to initialize the extension. Since the extension is not yet ready before `on_init_done()`, the TEN runtime will queue all messages sent to the extension until it's ready. Once ready, the queued messages are delivered to the extension. It's important to note that if the result is from a command sent by the extension itself, it is *not* subject to this restriction; the TEN runtime will directly pass the command's result back to the extension. This is because sending a command is a self-initiated action by the extension, so it must be prepared to receive and handle the result of its own commands.

### on\_start

This stage marks the runtime starting to send messages into the extension. The `on_start_done` function doesn't have a special purpose, but for consistency with other life cycle stages and to reduce learning complexity, the TEN framework includes it. Typically, `on_start_done` is invoked at the end of the `on_start` callback function. The TEN runtime will start delivering messages from other extensions to this extension after receiving its `on_start_done()`.

```c++
  void on_start(ten::ten_env_t &ten_env) override {
    // Some operations.
    ten_env.on_start_done();
  }
```

### on\_stop

There are many situations where the extension needs to stop, such as when the app or engine containing the extension is terminating. When the extension is about to stop, the TEN runtime uses this `on_stop` callback to notify the extension that it has entered the `on_stop` life cycle stage. The extension can then perform any necessary actions for termination.

### on\_deinit

After the extension calls `on_stop_done()`, it enters the `on_deinit` stage. During this stage, because the resources within the extension may no longer be fully available, the TEN runtime will not pass any messages from other extensions to this one.

## Relationship Between Extensions at Different Life Cycle Stages

Basically, there is no inherent relationship. Each extension operates independently, switching between its own life cycle stages. Extensions are independent of one another, and any dependencies between them must be explicitly implemented by the extensions themselves. The TEN runtime does not make any assumptions or guarantees. For example, if extension A needs to wait for extension B to complete its initialization before finishing its own, extension A can send a command to extension B during its `on_init`. Once extension B completes initialization and receives the command, it can reply with a result, and when extension A receives the result, it can call `on_init_done`.

```mermaid
sequenceDiagram
  participant A as Extension A
  participant B as Extension B

  Note over A,B: on_init
  activate A
  activate B
  deactivate B
  Note over B: on_init_done

  Note over B: on_start
  activate B
  deactivate B
  Note over B: on_start_done

  A->>B: cmd
  B-->>A: result

  deactivate A
  Note over A: on_init_done
```

Basically, in all other life cycle stages, if you want to implement the order of different extensions, you can achieve it using the same method.

## Interface with TEN Runtime

Extensions interact with the TEN runtime primarily through three interfaces:

1. **Lifecycle Callbacks**

   * These include callbacks like `on_init`, `on_deinit`, `on_start`, and `on_stop`.

2. **Callbacks for Receiving Messages**

   * These include callbacks such as `on_cmd`, `on_data`, `on_audio_frame`, and `on_video_frame`, which handle incoming messages to the extension.

3. **Functions for Sending Messages**

   * These include functions like `send_cmd`, `send_data`, `send_audio_frame`, and `send_video_frame`, which send messages out of the extension.

## Lifecycle Callbacks

The different stages of the extension's lifecycle and their connection to message handling are as follows:

* **on\_init \~ on\_init\_done**: Handles the extension's own initialization. At this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions.
* **on\_start \~ on\_start\_done**: At this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions. Since properties are initialized during `on_configure`, you can perform actions that depend on these properties being set up. However, as this is still part of the initialization phase, the extension will not receive messages initiated by others, avoiding the need for various checks. Active message sending is allowed.
* **After on\_start\_done \~ on\_stop\_done**: During this phase, the extension can normally send and receive all types of messages and their results.
* **on\_deinit \~ on\_deinit\_done**: Handles the extension's de-initialization. Similar to the `on_init` stage, at this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions.

## Implementing Extensions in Different Languages

Within the TEN framework, extensions can be implemented in various languages such as C++, Go, and Python. Developers can use the same conceptual approach to implement extensions in different languages. Learning how to develop an extension in one language makes it relatively easy to do so in other languages as well.

## Asynchronous Message Processing in Extensions

<figure><img src="../../..https://ten-framework-assets.s3.amazonaws.com/doc-assets/asynchronous_message_processing.png" alt=""><figcaption><p>Asynchronous Message Processing</p></figcaption></figure>

Extensions process messages asynchronously. When the TEN runtime delivers a message to an extension through callbacks like `on_cmd`, `on_data`, `on_audio_frame`, or `on_video_frame`, the extension is not required to process the message immediately within the callback. Instead, the extension can delegate the message to other threads, processes, or even machines for processing. This allows for full utilization of multi-core and distributed computing resources.

After processing is complete, the results can be sent back to the TEN runtime through callbacks such as `send_cmd`, `send_data`, `send_audio_frame`, or `send_video_frame`. The entire process is asynchronous, meaning the extension doesn't need to send the processed results back before the `on_cmd`, `on_data`, `on_audio_frame`, or `on_video_frame` callbacks return. The results can be transmitted only when they are actually ready, using the appropriate send functions.


# Overview (/docs/ten_framework/software_packages/overview)

The directory structure of a basic TEN framework software package is shown below.

```text
.
├── bin/
├── src/
├── manifest.json
├── manifest-lock.json
├── property.json
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    ├── protocol/
    │   └── <protocol_a>/
    └── system/
        ├── <system_package_1>/
        └── <system_package_2>/
```

Depending on the programming language, there may be additional required files. For example, in a C++ project, since the build system is `ten_gn`, a `BUILD.gn` file would be present. In a Go project, there would be files such as `go.mod` and `go.sum`. In a Python project, you might find a `requirements.txt` file.

## TEN Framework Software Package Types

| Type            | Description                     |
| --------------- | ------------------------------- |
| App             | Contains a TEN app.             |
| Extension group | Contains a TEN extension group. |
| Extension       | Contains a TEN extension.       |
| Protocol        | Contains a TEN protocol.        |
| System          | Contains a TEN system package.  |

## TEN Framework Software Package Kinds

There are two types of software packages:

* Release package
* Development package

The relationship between these two package types is as follows:

* A release package can be generated from a development package through a build process.
* If we add source-related content to a release package, it becomes a development package.
* If we remove source-related content from a development package, it becomes a release package.

In short, the directory structures of development and release packages are very similar, with the difference being that a development package includes source-related content, whereas a release package does not.

### Development package

The primary purpose of a development package is to develop TEN software packages. For instance, if you want to create a new TEN extension, you can modify an existing development package to achieve this. This is because the development package contains all the source-related content, enabling the creation of new TEN software packages from it.

### Release package

The primary purpose of a release package is to create and run graphs. Since no build tasks are required, a release package can be used to perform these actions. However, a development package can also be used for these tasks. Essentially, the application scope of a development package is broader than that of a release package.


# Python Async Extension (/docs/ten_framework/software_packages/python_async_extension)

A **Python async extension** is a modular component that using Python's `asyncio` framework. If you want to wrap existing Python codes that use `asyncio` into a TEN extension, using the Python async extension is the simplest and most convenient option.

## Example: The Default Python Async Extension

Here’s how a Python async extension is structured:

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    async def on_configure(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_json = cmd.to_json()
        ten_env.log_debug(f"DefaultAsyncExtension on_cmd: {cmd_json}")

        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

        # Send a new command to other extensions and wait for the result. The
        # result will be returned to the original sender.
        new_cmd = Cmd.create("hello")
        cmd_result = await ten_env.send_cmd(new_cmd)
        ten_env.return_result(cmd_result, cmd)
```

Each method simulates a delay using `await asyncio.sleep()`.

## FAQ

### Aysnc loop for event handling

* Create a queue: use `asyncio.Queue`.
* Create an async task for event handling.

Here is the sample code:

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    queue = asyncio.Queue()
    loop:asyncio.AbstractEventLoop = None

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        self.loop = asyncio.get_event_loop()

        self.loop.create_task(self._consume())

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        self.queue.put(None)

    async def _consume(self) -> None:
        while True
            try:
                value = await self.queue.get()
                if value is None:
                    self.ten_env.log_info("async loop exit")
                    break

                # Code for processing values retrieved from the queue.

            except Exception as e:
                self.ten_env.log_error(f"Failed to handle {e}")
```

## Conclusion

TEN's Python async extension provide a powerful way to handle long-running tasks asynchronously. By integrating Python’s `asyncio` framework, the extensions ensure that operations such as network calls or file handling are efficient and non-blocking. This makes TEN a great choice for building scalable, modular applications with asynchronous capabilities.


# Standard Interface (/docs/ten_framework/standard_extension/standard_interface)

In the TEN framework, we encourage developers to create their own extensions and publish them to the TEN extension marketplace for other developers to use. To enable developers to easily select and replace extensions, we need to define standardized interface specifications so that developers can quickly find extensions that meet their requirements based on these standard interfaces.

In short, standard interfaces are syntactic sugar that allows TEN tools (such as TEN Manager) to more conveniently assist developers in selecting and replacing extensions.

TEN's interface definitions are similar to TypeScript's `.d.ts` files, allowing the TypeScript compiler to perform type checking, but these interface definitions are not used at runtime. Therefore, the functionality and code enhancements brought by standard interface design are mainly reflected in development tools like TEN Manager, rather than the runtime TEN Runtime. For example, TEN Manager needs to provide relevant RESTful APIs to tman designer Frontend, allowing users to select and replace extensions through the UI interface based on standard interface definitions.

## What are Standard Interfaces

In the TEN framework, each extension's interface consists of the following core components:

1. **Properties** - Configuration parameters of the extension
2. **Command**
   * **Input command** - Input commands
   * **Output command** - Output commands
3. **Data**
   * **Input data** - Input data
   * **Output data** - Output data
4. **Audio frame**
   * **Input audio frame** - Input audio frames
   * **Output audio frame** - Output audio frames
5. **Video frame**
   * **Input video frame** - Input video frames
   * **Output video frame** - Output video frames

### Significance and Purpose of Standard Interfaces

Standard interfaces normalize these components into unified interface standards. Developers can refer to these interface standards to implement their own extensions, while extensions can also declare support for specific standard interfaces. This allows development tools to provide better development support based on the standard interfaces declared by extensions, enabling users to select or replace required extensions based on standard interfaces.

In standard interfaces, Command/Data/Audio frame/Video frame are used to standardize interactions between extensions, achieving interchangeability and compatibility of extensions. Properties are used to standardize extension configuration parameters, achieving configuration compatibility with tman designer.

It should be noted that standardizing interactions between extensions is not the function of the interface itself, but the function of Command/Data/Audio frame/Video frame. Interfaces are just syntactic sugar for aggregation purposes, helping developers quickly specify extension interface definitions.

### Interface Checking and Error Reporting

In an extension's Command/Data/Audio frame/Video frame definitions, if there are two messages with the same name and identical schema definitions, this is not considered an error, but will be treated as duplicate definitions and ignored. This rule also applies to standard interfaces, allowing different standard interfaces to define common functionalities themselves, forming self-contained standard interface definitions. When an extension implements these standard interfaces with identical definitions, it won't cause errors. This feature is very useful in standard interfaces because standard interface definitions can be referenced. So when an extension implements multiple standard interfaces, if these standard interfaces (or other standard interfaces they reference) have the same messages and identical schemas, it won't cause errors.

Since the TEN framework adopts a runtime-development separation architecture (similar to the relationship between JavaScript and TypeScript's .d.ts files), standard interface checking and error reporting mainly occur during the development tool phase, not at runtime in TEN Runtime. Therefore, TEN Runtime will not report errors due to interface errors when starting the graph.

In the tman designer development tool, if an extension's interface definition is incomplete, prompts will be displayed indicating that certain functions are temporarily unavailable for that extension. For example, when using one extension to replace another extension's functionality, if the interface definition is incomplete, a prompt will indicate that this function is temporarily unavailable for this extension and needs to wait for the interface definition to be complete before use.

Additionally, since interface definitions can be independent files that are referenced, incomplete interface definitions are not considered errors during TEN Manager's publish/install operations.

### Interface Compatibility Rules

When replacing an extension that implements interface A with an extension that implements interface B, interface A must be compatible with interface B for the replacement to be possible. There are two modes of compatibility rules between interfaces:

**Strict Mode:**

* All messages in interface A must appear in interface B, and the schema of messages in interface A must be compatible with the schema of corresponding messages in interface B.

**Relaxed Mode:**

* Only messages in interface A that have connections must appear in interface B, and the schemas of these messages in interface A must be compatible with the schemas of messages with the same names in interface B.

Strict mode and relaxed mode are user-selectable, suitable for different scenarios. For example, if it's certain that a specific message X in interface A implemented by an extension will never be used, then being unable to replace with another desired extension due to message X's incompatibility or non-existence might be too strict.

## Implementation of Standard Interfaces

The APIs supported by TEN extensions are defined in their `manifest.json` files. The implementation of standard interfaces involves writing the defined API specifications into independent JSON files and then referencing these files in `manifest.json`.

```json
{
  "type": "extension",
  "name": "foo_extension",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.1"
    }
  ],
  "api": {
    "interface": [
      {
        "import_uri": "https://xxx/asr_interface.json"  // <= Reference to standard interface definition file
      },
      {
        "import_uri": "https://xxx/tts_interface.json"  // <= Reference to standard interface definition file
      }
    ],
    "property": {
      "bar": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "query_vector",
        "property": {
          "collection_name": {
            "type": "string"
          },
          "top_k": {
            "type": "int64"
          },
          "embedding": {
            "type": "array",
            "items": {
              "type": "float64"
            }
          }
        },
        "required": [
          "collection_name",
          "top_k",
          "embedding"
        ],
        "result": {
          "property": {
            "response": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "content": {
                    "type": "string"
                  },
                  "score": {
                    "type": "float64"
                  }
                }
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "publish",
        "property": {
          "message": {
            "type": "buf"
          }
        }
      },
      {
        "name": "set_presence_state"
      }
    ],
    "data_in": [
      {
        "name": "data"
      }
    ]
  }
}
```

### Example of Standard Interface Definition File

When TEN systems (including TEN Runtime, TEN Manager, etc.) parse `manifest.json`, they download the corresponding JSON files based on `import_uri` and merge their contents into `manifest.json`. The example content of the referenced standard interface file is shown below. As you can see, the content is the same as the `api` field in manifest.json, just separated into an independent file for easy reference. This independent standard interface file can also reference other standard interface files.

```json
{
  "interface": [
    {
      "import_uri": "https://xxx/stt_interface.json"// <= Reference to other standard interface definition files
    }
  ],
  "property": {
    "bar": {
      "type": "string"
    }
  },
  "cmd_in": [
    {
      "name": "query_vector",
      "property": {
        "collection_name": {
          "type": "string"
        },
        "top_k": {
          "type": "int64"
        },
        "embedding": {
          "type": "array",
          "items": {
            "type": "float64"
          }
        }
      },
      "required": [
        "collection_name",
        "top_k",
        "embedding"
      ],
      "result": {
        "property": {
          "response": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string"
                },
                "score": {
                  "type": "float64"
                }
              }
            }
          }
        }
      }
    }
  ],
  "cmd_out": [
    {
      "name": "publish",
      "property": {
        "message": {
          "type": "buf"
        }
      }
    },
    {
      "name": "set_presence_state"
    }
  ],
  "data_in": [
    {
      "name": "data"
    }
  ]
}
```

### Interface Merging Rules

The system merges the content of JSON files pointed to by `import_uri` into the `api` field of `manifest.json`, specifically including:

* `property`
* `cmd_in`, `cmd_out`
* `data_in`, `data_out`
* `audio_frame_in`, `audio_frame_out`
* `video_frame_in`, `video_frame_out`

**Important Rules:**

1. **Conflict Detection**: If fields of the same name and type are found during merging (such as `cmd_in` or `property` with the same name), it's not considered an error if the schemas are identical, otherwise the system will report an error.
2. **Recursive Merging**: The merging process is recursive, and the system will report an error if circular references are detected.
3. **Multiple Interface Support**: An extension can support multiple standard interfaces by defining multiple `interface` fields in `manifest.json`.
4. **Duplicate Reference Detection**: If different `interface` fields reference the same `import_uri`, the system will report an error.

## Storage Methods for Standard Interfaces

Standard interface files support the following storage methods:

<div className="fd-steps">
  <div className="fd-step">
    ### TEN System Package

    **Overview**: This method packages standard interface definition files into TEN system packages and then references them in `manifest.json`. You can also depend on this system package in the dependencies field of manifest.json, so that when installing the extension, the dependent standard interface system package will be installed together, providing the best development experience. Through this method, you can use TEN package's version mechanism to implement version management of standard interfaces, and use TEN package's dependency mechanism to implement automatic installation of standard interfaces.

    Using TEN system packages, in addition to standard interface JSON definition files, can also include related resources such as model files, common utility function libraries, supporting tools, etc. Therefore, when any of these resources have incompatible changes, they can be better reflected through TEN system package version number changes. For example, changes in the storage location or file names of JSON files within the system package are also considered breaking changes to the system package, so the major version number of the system package should be upgraded.

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "0.1.0"
        }
      ],
      "api": {
        "interface": [
          {
            "import_uri": "../../../ten_packages/system/foo_interface/interface.json"
          }
        ]
      }
    }
    ```

    **Advantages of Using TEN System Packages:**

    1. **Easy Distribution**: System packages can be uploaded to the TEN extension marketplace for users to download directly
    2. **Rich Resources**: The package not only contains interface definition files but can also include related resources such as:
       * Model files
       * Common utility function libraries
       * Supporting tools, etc.
    3. **Dependency Management**: System packages can serve as dependencies of extensions. When extensions are downloaded, related system packages will also be automatically downloaded
    4. **Version Management**: Not limited to standard interface protocols themselves, any resources within the system package are part of the system package's version management

    It should be noted that the version number of TEN system packages does not only represent the version number of standard interfaces, but represents the version number of the entire system package. Therefore, if resources within the system package have breaking changes, the major version number of the system package should be upgraded. For example, changes in the storage location or file names of JSON files within the system package, even if the content of JSON files hasn't changed, are considered breaking changes to the system package because the usage method needs to change for users, so the major version number of the system package should be upgraded.

    Therefore, you cannot judge the compatibility of standard interfaces solely based on the version number of system packages. The compatibility of standard interfaces still needs to be judged through interface compatibility rules.

    #### Support for Multi-Version Coexistence

    > **Note**: The functionality described in this section will basically not be implemented. It only describes possible implementation methods and considerations if it were to be done in the future.

    **Overview**: In some scenarios, a TEN application might need to use different versions of the same standard interface simultaneously. To support this requirement, TEN allows installing multiple versions of the same system package within the same application.

    **System Package Configuration:**

    First, declare support for multi-version coexistence in the system package's `manifest.json`:

    ```json
    {
      "type": "system",
      "name": "foo_interface",
      "version": "0.1.0",
      "install_policy": {
        "coinstallable": "true"
      }
    }
    ```

    **Extension Dependency Configuration:**

    In the `manifest.json` of extensions that depend on this system package, you also need to declare the use of multi-version coexistence:

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "=0.1.0", // Note: Only equal syntax can be used here, not >= syntax
          "install_policy": {
            "coinstallable": "true"
          }
        }
      ],
    }
    ```

    And during the entire dependency tree construction process, if a system package's coinstallable is true, then all places that depend on this system package need to declare coinstallable, otherwise the system will report an error.

    **File System Path:**

    After enabling multi-version coexistence, the path of system packages in the file system becomes:

    ```text
    ten_packages/system/<hash_value>_foo_extension_0_1_0/
    ```

    > **Note**: A mapping mechanism might be introduced in the future to allow users not to explicitly specify hash\_value during usage, but to automatically find corresponding system packages through some mapping mechanism. However, for this mechanism to work well, it would also require modifying the module loading mechanisms in languages supported by TEN, so that developers don't need to specify hash\_value in the module loading mechanisms of those languages, but can automatically find corresponding system packages through some mapping mechanism. This kind of deep language-internal modification is also one of the reasons why this single-package multi-version coexistence mechanism will basically not be implemented.
    >
    > **Note**: If there's really a need to use multiple versions of a standard interface within a TEN application, you can use the several methods described below. If a standard interface has accompanying common resources such as model files, common utility function libraries, supporting tools, etc., then many of these resources cannot achieve multi-version coexistence within a TEN application, which is also one of the reasons why this single-package multi-version coexistence mechanism will basically not be implemented.
  </div>

  <div className="fd-step">
    ### Network Address

    **Overview**: Deploy standard interface definition files to network servers and reference them through HTTP/HTTPS addresses:

    ```json
    {
      "interface": [
        {
          "import_uri": "http://xxx/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    > **Note**: When TEN Manager handles this type of reference, it can download files to its own cache and then use some mapping mechanism to allow TEN Manager to find the definitions.

    **Applicable Scenarios:**

    * **Lightweight Interfaces**: Interface definitions without accompanying common resources
    * **Version Flexibility**: Need to reference different versions of the same standard interface in one `manifest.json`
  </div>

  <div className="fd-step">
    ### Local Path

    **Overview**: Reference standard interface definition files in the local file system:

    **Absolute Path:**

    ```json
    {
      "interface": [
        {
          "import_uri": "file://c:/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **Relative Path:**

    ```json
    {
      "interface": [
        {
          "import_uri": "../asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **Applicable Scenarios:**

    Similar to the network address method:

    * **Lightweight Interfaces**: Suitable for interface definitions without accompanying common resources
    * **Version Flexibility**: Need to reference different versions of the same standard interface in one `manifest.json`
  </div>
</div>

## How to Filter Desired Extensions from the Cloud Store

The relationship between standard interfaces and the cloud store is reflected in packaging standard interfaces into TEN system packages. There are two ways to search for extensions that meet specific conditions:

1. **Dependency Search**: Search for extensions that depend on a certain system package. Since the metadata of extensions in the cloud store contains dependency information, the cloud store can filter out all extensions that depend on a certain system package accordingly.

2. **Tag Search**: Quickly search for extensions that meet certain conditions through TEN package tags, select and install them, then perform replacement operations. If they are TEN officially certified extensions, they can even be tagged with `ten:` prefix to indicate that this is an officially certified extension that indeed conforms to the declared standard interface definition.


# Check-Graph (/docs/ten_framework/ten_manager/check_graph)

`tman` provides the `check graph` command to validate predefined graphs or a `start_graph` command for correctness. To see the usage details, use the following command:

```shell
$ tman check graph -h
Check whether the graph content of the predefined graph or start_graph command is correct. For more detailed usage, run 'graph -h'

Usage: tman check graph [OPTIONS] --app <APP>

Options:
      --app <APP>
          The absolute path of the app defined in the graph. By default, the predefined graph will be read from the first one in the list.
      --predefined-graph-name <PREDEFINED_GRAPH_NAME>
          Specify a predefined graph name to check, otherwise, all predefined graphs will be checked.
      --graph <GRAPH>
          Specify the JSON string of a 'start_graph' command to be checked. If not specified, the predefined graph in the first app will be checked.
  -h, --help
          Print help
```

The `check graph` command is designed to handle graphs that may span multiple apps, so it does not require being run from the root directory of a specific app. Instead, it can be executed from any directory, with the `--app` parameter used to specify the folders of the apps involved in the graph.

## Example Usages

* **Check all predefined graphs in `property.json`**:

  ```shell
  tman check graph --app /home/TEN-Agent/agents
  ```

* **Check a specific predefined graph**:

  ```shell
  tman check graph --predefined-graph-name va.openai.azure --app /home/TEN-Agent/agents
  ```

* **Check a `start_graph` command**:

  ```shell
  tman check graph --graph '{
    "type": "start_graph",
    "seq_id": "55",
    "nodes": [
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_2__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      },
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_1__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      }
    ]
  }' --app /home/TEN-Agent/agents
  ```

## Prerequisites

* **Predefined Graph Definition Requirement**: If a predefined graph name is specified, the definition will be extracted from the `property.json` file of the app specified by the first `--app` parameter.

* **Package Installation Requirement**: Before running the `check graph` command, all extensions that the app depends on must be installed using the `tman install` command. This is necessary because the validation process requires information about each extension in the graph, such as APIs defined in their `manifest.json` files.

* **Unique App URI Requirement**: In a multi-app graph, each app's `property.json` must define a unique `ten::uri`. Additionally, the `uri` value cannot be set to `"localhost"`.

## Validation Rules

<div className="fd-steps">
  <div className="fd-step">
    ### Presence of Nodes

    The `nodes` array is required in any graph definition. If absent, an error will be thrown.

    * **Example (No nodes defined)**:

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": []
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        No extension node is defined in graph.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### Uniqueness of Nodes

    Each node in the `nodes` array represents a specific extension instance within a group of an app, created by a specified addon. Therefore, each extension instance should be uniquely represented by a single node. A node must be uniquely identified by the combination of `app`, `extension_group`, and `name`. Multiple entries for the same extension instance are not allowed.

    * **Example (Duplicate nodes)**:

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": [
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_2__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          },
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_1__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        Duplicated extension was found in nodes[1], addon: basic_hello_world_1__test_extension, name: test_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### Extensions used in connections should be defined in nodes

    All extension instances referenced in the `connections` field, whether as a source or destination, must be explicitly defined in the `nodes` field. Any instance not defined in the `nodes` array will cause validation errors.

    * **Example (Source extension not defined)**:

      Imagine that the content of `property.json` of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0] is not defined in nodes, extension_group: producer, extension: some_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

    * **Example (Destination extension not defined)**:

      Imagine that the content of `property.json` of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                },
                {
                  "type": "extension",
                  "name": "some_extension_1",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        }
                      ]
                    },
                    {
                      "name": "world",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        },
                        {
                          "extension": "consumer"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0].cmd[1] is not defined in nodes extension_group: some_group, extension: consumer.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The addons declared in the `nodes` must be installed in the app

    * **Example (The `ten::uri` in property.json is not equal to the `app` field in nodes)**:

      Imagine that the content of property.json of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ],
          "uri": "http://localhost:8001"
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

      The problem is all packages in the app will be stored in a map which key is the `uri` of the app, and each node in the graph is retrieved by the `app` field (which is `localhost` by default). The `app` in node (i.e., localhost) is mismatch with the `uri` of app (i.e., [http://localhost:8001](http://localhost:8001)).

    * **Example (the ten\_packages does not exist as the `tman install` has not executed)**:

      Imagine that the content of property.json of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ]
        }
      }
      ```

      And the `ten_packages` directory does **\_NOT** exist.

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### In connections, messages sent from one extension should be defined in the same section

    * **Example**:

      Imagine that all the packages have been installed.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          },
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello_2",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        extension 'some_extension' is defined in connection[0] and connection[1], merge them into one section.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### In connections, the messages sent out from one extension should have a unique name in each type

    * **Example**:

      Imagine that all the packages have been installed.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              },
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "some_extension"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        - connection[0]:
         - Merge the following cmd into one section:
            'hello' is defined in flow[0] and flow[1].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The messages declared in the connections should be compatible

    The message declared in each message flow in the connections will be checked if the schema is compatible, according to the schema definition in the manifest.json of extensions. The rules of message compatible are as follows.

    * If the message schema is not found neither in the source extension nor the target extension, the message is compatible.

    * If the message schema is only found in one of the extensions, the message is incompatible.

    * The message is compatible only if the following conditions are met.

      * The property type is compatible.
        * If the property is an object, the fields both in the source and target schema must have a compatible type.
        * If the `required` keyword is defined in the target schema, there must be a `required` keyword in the source schema, which is the superset of the `required` in the target.

    * **Example**:

      Imagine that all the packages have been installed.

      The content of manifest.json in `addon_a` is as follows.

      ```json
      {
        "type": "extension",
        "name": "addon_a",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_out": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "string"
                }
              }
            }
          ]
        }
      }
      ```

      And, the content of manifest.json in `addon_b` is as follows.

      ```json
      {
        "type": "extension",
        "name": "addon_b",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_in": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "int8"
                }
              }
            }
          ]
        }
      }
      ```

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        - connections[0]:
          - cmd[0]:  Schema incompatible to [extension_group: some_group, extension: another_ext], { .foo: type is incompatible, source is [string], but target is [int8] }

      All is done.
      ❌ Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The `app` in node must be unambiguous

    The `app` field in each node must met the following rules.

    * The `app` field must be equal to the `ten::uri` of the corresponding TEN app.

    * Either all nodes should have `app` declared, or none should.

    * The `app` field can not be `localhost`.

    * The `app` field can not be an empty string.

    * **Example (some of the nodes specified the `app` field)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        Either all nodes should have 'app' declared, or none should, but not a mix of both.
      ```

    * **Example (No `app` specified in all nodes, but some source extension specified `app` in the connections)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
         {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].the 'app' should not be declared, as not any node has declared it
      ```

    * **Example (No `app` specified in all nodes, but some target extension specified `app` in the connections)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext",
                    "app": "http://localhost:8000"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].cmd[0].dest[0]: the 'app' should not be declared, as not any node has declared it
      ```

    * **Example (The `app` field in nodes is not equal to the `ten::uri` of app)**:

      Same as [Rule 4](#id-4.-the-addons-declared-in-the-nodes-must-be-installed-in-the-app).

    * **Example (The `app` field is `localhost` in a single-app graph)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, and the graph seems to be a single-app graph, just remove the 'app' field
      ```

    * **Example (The `app` field is `localhost` in a multi-app graph)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, change the content of 'app' field to be consistent with 'ten::uri'
      ```
  </div>
</div>


# Overview (/docs/ten_framework/ten_manager/overview)

In the TEN ecosystem, almost everything is a component. For example, a TEN app is a type of component, a TEN extension is another type, and a TEN protocol is yet another. To assist TEN developers in managing all operations related to these components, there is a crucial tool called the TEN Manager.

<Callout type="info">
  `tman`

   is the abbreviation for TEN Manager.
</Callout>

The TEN Manager includes a TEN package manager, designed to simplify the management of TEN packages. This package manager handles tasks such as determining which dependency packages are necessary, fetching those dependencies, and installing them. In short, the TEN Manager streamlines the process of creating new TEN packages from scratch and makes it easier to install TEN packages from the TEN Cloud Store.


# Debugging (/docs/ten_framework/testing_and_debugging/debugging)

## For Developers of the TEN Framework

If you're using VS Code to develop the TEN framework, there is a `launch.json` file in the TEN framework source tree that provides default debug targets.

## For Users of the TEN Framework

If you're using VS Code to develop your own application based on the TEN framework, you can add configurations to your `.vscode/launch.json` file so that you can debug your application with breakpoints and variable inspection whichever programming language (C++/Golang/Python) you are using.

### Debugging in C++ Applications

If the application is written in C++, it means the extensions can be written
in C++ or Python.

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

* **Debugging C++ code with lldb**

  ```json
  {
    "name": "app (C++) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    }
  }
  ```

* **Debugging C++ code with gdb**

  ```json
  {
    "name": "app (C++) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### Debugging Python code with debugpy

You can use `debugpy` to debug Python code. However, since the program is not started directly through the Python interpreter, the Python code is executed by the embedded Python interpreter. Therefore, you can only attach a debugger to a running process.

First, you need to install `debugpy` in the Python environment:

```shell
pip install debugpy
```

Then, you need to start the application with environment variable `TEN_ENABLE_PYTHON_DEBUG` and `TEN_PYTHON_DEBUG_PORT` set. For example:

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

If the environment variable `TEN_ENABLE_PYTHON_DEBUG` is set to `true`, then the application will block until the debugger is attached. If the environment variable `TEN_PYTHON_DEBUG_PORT` is set, then the debugger server will listen on the specified port for incoming connections.

Then, you can attach the debugger to the running process. Here is an example of the configuration:

**Debugging Python code with debugpy**

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### Debugging C++ and Python code at the same time

If you want to start debugging C++ and Python code at the same time with one click in VSCode, you can do the following:

1. Define a task for delaying several seconds before attaching the debugger:

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. Add the following configurations to the `launch.json` file:

   ```json
   "configurations": [
        {
            "name": "app (C++) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch", // "launch" or "attach"
            "program": "${workspaceFolder}/bin/worker", // The executable path
            "cwd": "${workspaceFolder}/",
            "env": {
                "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
                "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (C++) (debugpy, attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "justMyCode": false,
            "preLaunchTask": "delay 3 seconds"
        }
   ],
   "compounds": [
        {
            "name": "app (C++) (lldb, launch) and app (C++) (debugpy, attach)",
            "configurations": [
                "app (C++) (lldb, launch with debugpy)",
                "app (C++) (debugpy, attach with delay)"
            ]
        }
   ]
   ```

Then, you can start debugging C++ and Python code with the compound configuration `app (C++) (lldb, launch) and app (C++) (debugpy, attach)`.

### Debugging in Go applications

If the application is written in Go, it means the extensions can be written in Go or C++ or Python.

#### Debugging Go code with delve

You can use the following configuration to debug in your Go extensions:

```json
{
  "name": "app (golang) (go, launch)",
  "type": "go",
  "request": "launch",
  "mode": "exec",
  "cwd": "${workspaceFolder}/",
  "program": "${workspaceFolder}/bin/worker", // The executable path
  "env": {
    "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
    "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
    "TEN_APP_BASE_DIR": "${workspaceFolder}"
  }
}
```

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

* **debugging C++ code with lldb**

  ```json
  {
    "name": "app (Go) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    },
    "initCommands": ["process handle SIGURG --stop false --pass true"]
  }
  ```

* **debugging C++ code with gdb**

  ```json
  {
    "name": "app (Go) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### Debugging Python code with debugpy

Refer to the section above for debugging Python code with debugpy. Start the application with environment variable `TEN_ENABLE_PYTHON_DEBUG` and `TEN_PYTHON_DEBUG_PORT` set.

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

Then, you can attach the debugger to the running process. Here is an example of the configuration:

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### Debugging C++, Go and Python code at the same time

If you want to start debugging C++, Go and Python code at the same time with one click in VSCode, you can do the following:

1. Define a task for delaying several seconds before attaching the debugger:

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. Add the following configurations to the `launch.json` file:

   ```json
   "configurations": [
        {
            "name": "app (golang) (debugpy, remote attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "delay 3 seconds",
            "justMyCode": false
        },
        {
            "name": "app (golang) (go, launch with debugpy)",
            "type": "go",
            "request": "launch",
            "mode": "exec",
            "cwd": "${workspaceFolder}/",
            "program": "${workspaceFolder}/bin/worker",
            "env": {
                "TEN_APP_BASE_DIR": "${workspaceFolder}",
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (golang) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/bin/worker",
            "cwd": "${workspaceFolder}/",
            "env": {
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            },
            "initCommands": [
                "process handle SIGURG --stop false --pass true"
            ]
        },
   ],
   "compounds": [
        {
            "name": "Mixed Go/Python",
            "configurations": [
                "app (golang) (go, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ],
        },
        {
            "name": "Mixed Go/Python/C++ (lldb)",
            "configurations": [
                "app (golang) (lldb, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ]
        }
   ]
   ```

If you want to debug C++ and Python code at the same time, you can use the compound configuration `Mixed Go/Python/C++ (lldb)`. With this configuration, you can perform breakpoint debugging for C++, Go and Python code simultaneously. However, the variable inspection is only supported for C++ and Python but not for Go code.

If you want to debug Go and Python code at the same time, you can use the compound configuration `Mixed Go/Python`. With this configuration, you can perform breakpoint debugging for Go and Python code simultaneously and inspect variables for both Go and Python code.

### Debugging in Python applications

If the application is written in Python, it means the extensions can be written in Python or C++.

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

```json
{
  "name": "app (Python) (cpp, launch)",
  "type": "cppdbg", //
  "request": "launch", // "launch" or "attach"
  "program": "/usr/bin/python3", // The Python interpreter path
  "args": ["main.py"],
  "cwd": "${workspaceFolder}/",
  "environment": [
    {
      "name": "PYTHONPATH",
      "value": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface"
    },
    {
      "name": "TEN_APP_BASE_DIR",
      "value": "${workspaceFolder}/"
    }
  ],
  "MIMode": "gdb" // "gdb" or "lldb"
}
```

#### Debugging Python code with debugpy

Refer to the section above for debugging Python code with debugpy. You can use
the following configurations to debug Python code:

```json
{
  "name": "app (Python) (debugpy, launch)",
  "type": "debugpy",
  "python": "/usr/bin/python3",
  "request": "launch",
  "program": "${workspaceFolder}/main.py",
  "console": "integratedTerminal",
  "cwd": "${workspaceFolder}/",
  "env": {
    "PYTHONPATH": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface",
    "TEN_APP_BASE_DIR": "${workspaceFolder}/",
    "TEN_ENABLE_PYTHON_DEBUG": "true"
  }
}
```

#### Debugging C++ and Python code at the same time

If you want to start debugging C++ and Python code at the same time, you can launch the application with the configuration `app (Python) (debugpy, launch)` and attach the debugger to the running process with the following configuration:

```json
{
  "name": "app (Python) (cpp, attach)",
  "type": "cppdbg",
  "request": "attach", // "launch" or "attach"
  "program": "${workspaceFolder}/bin/worker", // The executable path
  "cwd": "${workspaceFolder}/",
  "MIMode": "gdb",
  "environment": [
    {
      "name": "LD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    },
    {
      "name": "DYLD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    }
  ]
}
```


# For Developers of TEN framework (/docs/ten_framework/testing_and_debugging/for_developers_of_ten_framework)

The TEN framework includes three types of test suites:

* Unit Tests
* Smoke Tests
* Integration Tests

The TEN framework uses `gtest` as the testing framework for unit tests and smoke tests, and `pytest` as the testing framework for integration tests.

## Unit Tests

The source directory for unit tests is `tests/ten_runtime/unit`. If you need to add new unit test cases, please place them in the `tests/ten_runtime/unit directory`.

For C++ projects, after successful compilation, you can navigate to the output directory (e.g., `out/linux/x64/tests/standalone/`, depending on the target OS and CPU) and execute the following command:

```shell
./ten_runtime_unit_test
```

## Smoke Tests

The source directory for smoke tests is `tests/ten_runtime/smoke`. If you need to add new smoke test cases, please place them in the `tests/ten_runtime/smoke` directory.

Navigate to the output directory (e.g., `out/linux/x64/tests/standalone`, depending on the target OS and CPU) and execute the following command:

```shell
./ten_runtime_smoke_test
```

## Loop Testing

You can perform multiple rounds of testing using the following commands.

To run only unit tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;
```

To run only smoke tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

To run both unit tests and smoke tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

## Integration Tests

Integration tests are black-box tests for the TEN framework. The source directory for integration tests is `tests/ten_runtime/integration`. These tests are used to validate real-world execution scenarios.

The file structure for integration tests is as follows:

```text
tests/ten_runtime/integration/
  ├── test_1/
  │    ├── test_case.py
  │    └── ...
  ├── test_2/
  │    ├── test_case.py
  │    └── ...
  └── ...
```

To execute the integration tests, use the following command:

```shell
cd path/to/out/
pytest tests/ten_runtime/integration/
```

## Testing Tricks

### Linux

On Linux, you can use the following technique to slow down the execution of TEN, which can help in triggering timing-related bugs.

```shell
sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...
```

* `taskset 0x1`: Use only 1 CPU core to run the program.
* `taskset 0x3`: Use only 2 CPU cores to run the program.


# For Users of the TEN Framework (/docs/ten_framework/testing_and_debugging/for_users_of_ten_framework)

## Standalone Testing of Extension

The TEN framework provides an independent extension testing mechanism. This mechanism allows developers to test extensions without relying on other TEN concepts, such as other TEN extensions, TEN graphs, or TEN apps. When developers need to test the behavior of an extension without running the entire TEN app, this independent extension testing mechanism is especially useful.

The standalone testing framework for TEN extensions follows two key principles:

1. Compatibility with any native testing framework of the used language.

   For example, if a TEN extension is developed using C++, the Google gtest/gmock testing framework can be used alongside the TEN extension standalone testing framework to achieve independent testing of the C++ TEN extension.

2. Users don't need to modify any code in the extension under test.

   The exact same extension code used in runtime can be fully tested using this standalone testing framework.

3. The standalone testing flow for extensions developed in different languages follows the same design principles and usage methods.

   This ensures that once you learn the standalone testing concepts and methods for one language, you can quickly grasp the standalone testing concepts and methods for TEN extensions in other languages.

For users, the TEN extension standalone testing framework introduces two main concepts:

1. **extension\_tester**
2. **ten\_env\_tester**

The role of **extension\_tester** is similar to a testing driver, responsible for setting up and executing the entire testing process. **ten\_env\_tester**, on the other hand, behaves like a typical TEN extension's `ten_env` instance, enabling users to invoke functionalities within the standalone testing framework from the callback of `extension_tester`, such as sending messages to the extension under test and receiving messages from the extension.

From the API design of **extension\_tester** and **ten\_env\_tester**, you can see that they are very similar to **TEN extension** and **ten\_env**, and this is by design. The main purpose is to allow users familiar with extension development to quickly get accustomed to the standalone testing framework and then develop standalone test cases for the extension itself.

However, for testing purposes, there are inevitably APIs and features dedicated specifically to testing. To prevent these test-specific functionalities and APIs, which are not needed during actual runtime, from polluting the runtime API set, the standalone testing framework is designed not to directly use or extend the API sets of **extension** and **ten\_env**. Instead, it introduces types and API sets that are exclusive to standalone testing. This separation ensures that the types and API sets used in actual runtime and those used during testing do not interfere with each other, avoiding any negative side effects.

## Standalone Testing Framework Internal

Internally, the TEN extension standalone testing framework implicitly starts a test app, loads the extension addon to be tested (let’s call it extension A), and initializes a graph containing both the extension A to be tested and another extension (let’s call it extension B) dedicated to testing other extensions. All input and output messages of extension A are redirected to extension B, allowing users to customize inputs and outputs during the testing process and complete the actual testing work.

```mermaid
sequenceDiagram
  participant tester as Extension Tester
  participant testing as Extension Testing
  participant tested as Extension Tested

  tester->>testing: send messages
  testing->>tested: send messages

  tested-->>testing: send results
  testing-->>tester: send results
```

Basically, you can think of this testing extension hidden within the testing framework as a proxy between the extension being tested and the tester. It acts as a proxy extension within the TEN graph, facilitating the exchange of messages between the tested extension and the tester using the language of the TEN environment.

## Basic Testing Process

The basic testing process and logic are as follows:

1. Create an extension tester to manage the entire standalone testing process.
2. Inform the standalone testing framework of the folder containing the extension to be tested.
3. Set a testing mode, such as a mode for testing a single extension.
4. Start the testing.

## C++

Here is an example of TEN extension standalone testing using Google gtest:

```c++
class extension_tester_basic : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    auto new_cmd = ten::cmd_t::create("hello_world");
    ten_env.send_cmd(std::move(new_cmd),
                     [](ten::ten_env_tester_t &ten_env,
                        std::unique_ptr<ten::cmd_result_t> result) {
                       if (result->get_status_code() == TEN_STATUS_CODE_OK) {
                         ten_env.stop_test();
                       }
                     });
  }
};

TEST(Test, Basic) {
  // 1. Create an extension tester to manage the entire standalone testing process.
  auto *tester = new extension_tester_basic();

  // 2. Set a testing mode, such as a mode for testing a single extension.
  tester->set_test_mode_single("default_extension_cpp");

  // 3. Start the testing.
  tester->run();

  delete tester;
}
```

## Golang

TODO: To be added.

## Python

```python
class ExtensionTesterBasic(ExtensionTester):
    def check_hello(self, ten_env: TenEnvTester, result: CmdResult):
        statusCode = result.get_status_code()
        print("receive hello_world, status:" + str(statusCode))

        if statusCode == StatusCode.OK:
            ten_env.stop_test()

    def on_start(self, ten_env: TenEnvTester) -> None:
        new_cmd = Cmd.create("hello_world")

        print("send hello_world")
        ten_env.send_cmd(
            new_cmd,
            lambda ten_env, result: self.check_hello(ten_env, result),
        )

        print("tester on_start_done")
        ten_env.on_start_done()


def test_basic():
    # 1. Create an extension tester to manage the entire standalone testing process.
    tester = ExtensionTesterBasic()

    # 2. Set a testing mode, such as a mode for testing a single extension.
    tester.set_test_mode_single("default_extension_python")

    # 3. Start the testing.
    tester.run()
```


# How to Debug with Logs (/docs/ten_framework/testing_and_debugging/how_to_debug_with_logs)

We will introduce how to view logs, as they are now ported to stdout, making them very easy to view.

For example, if a TEN Agent is running on localhost:3000, you might see output similar to the following in the logs:

```bash
...
[GIN] 2024/09/29 - 09:29:42 | 200 |     634.458µs |    192.168.65.1 | POST     "/ping"
[agora_5bnxv1] 09-29 09:29:42.895  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ How about you?], text_recv_ts: 1727602181374572, read_bytes: 37200, sent_frames: 117, first_frame_latency: 306ms, finish_latency: 428ms, synthesis_first_byte_latency: 300ms, synthesis_finish_latency: 332ms
[agora_5bnxv1] 09-29 09:29:43.350  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Anything exciting happening?], text_recv_ts: 1727602181383864, read_bytes: 63600, sent_frames: 199, first_frame_latency: 300ms, finish_latency: 452ms, synthesis_first_byte_latency: 295ms, synthesis_finish_latency: 332ms
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_5bnxv1 requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
2024/09/29 09:29:45 INFO handlerPing end worker="&{ChannelName:agora_5bnxv1 HttpServerPort:10024 LogFile:/tmp/astra/app-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.json Pid:7455 QuitTimeoutSeconds:60 CreateTs:1727602125 UpdateTs:1727602185}" requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |    1.038208ms |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
2024/09/29 09:29:45 ERROR handlerPing channel not existed channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |     442.375µs |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:48 INFO handlerPing start channelName=agora_5bnxv1 requestId=6fb07d85-
...
```

Once you interact with the TEN Agent, you should see entries like the following:

```bash title="Bash"
[agora_5bnxv1] 09-29 09:35:25.049  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ and the World Cup in Russia.], text_recv_ts: 1727602522313774, read_bytes: 64000, sent_frames: 200, first_frame_latency: 418ms, finish_latency: 568ms, synthesis_first_byte_latency: 413ms, synthesis_finish_latency: 458ms
[agora_5bnxv1] 09-29 09:35:25.570  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ In tech,], text_recv_ts: 1727602522347918, read_bytes: 29200, sent_frames: 92, first_frame_latency: 427ms, finish_latency: 515ms, synthesis_first_byte_latency: 419ms, synthesis_finish_latency: 460ms
[agora_5bnxv1] 09-29 09:35:26.502  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ we saw some big advancements like the rise of AI and electric cars becoming more mainstream.], text_recv_ts: 1727602522461866, read_bytes: 178800, sent_frames: 559, first_frame_latency: 510ms, finish_latency: 928ms, synthesis_first_byte_latency: 503ms, synthesis_finish_latency: 800ms
[agora_5bnxv1] 09-29 09:35:27.029  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Personally,], text_recv_ts: 1727602522468652, read_bytes: 34400, sent_frames: 108, first_frame_latency: 442ms, finish_latency: 524ms, synthesis_first_byte_latency: 436ms, synthesis_finish_latency: 463ms
```

When you see logs like these, it means the system is working properly and is logging every sentence you speak.


# Doc Window (/docs/ten_framework/tman_designer/doc_window)

TMAN Designer has a built-in documentation window that allows you to view TEN Framework concepts and terminology. For example, you can select `About App` from the App dropdown menu to open the documentation window and display the App's description.

![TMAN Designer Doc Window](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_doc_window.png)


# Overview (/docs/ten_framework/tman_designer/overview)

TEN Framework offers high flexibility and provides rich functionalities, including TEN Cloud Store, package management, multi-language support, and multi-platform support. However, these features may require a longer learning period and higher technical threshold to master TEN Framework for AI Agent development and customization. To facilitate developers' use, TEN Framework provides a command-line tool called TEN Manager, which enables developers to easily modify and customize AI Agents built with TEN Framework.

Since command-line tools are not as user-friendly as graphical user interfaces, TEN Manager includes a web UI-based visual development interface called TMAN Designer, in addition to the command-line interface. It provides rich functionality to help developers develop and debug TEN Apps and TEN Extensions. This means that developers only need to obtain the TEN Manager tool to access a complete TEN Framework development environment with both command-line and graphical user interfaces.

You can get TMAN Designer from the following link:

```text
https://github.com/TEN-framework/ten-framework/releases
```

TMAN Designer supports the following operating systems and architectures. Please obtain the appropriate version for your platform:

* Windows
  * x86\_64
* Linux
  * x86\_64
* MacOS
  * x86\_64
  * arm64

After downloading and extracting, use the following command to start TMAN Designer:

```shell
tman designer
```

You can execute this command in the root directory of any TEN App, or in any location. If executed in the root directory of a TEN App, TMAN Designer will load that TEN App by default. You can also manually load any TEN App in TMAN Designer. Therefore, starting TMAN Designer in the root directory of a specific TEN App is not mandatory, but it simplifies the initial step of manually loading a TEN App.

After startup, you will see information like the following, indicating that TMAN Designer starts on port 49483 by default:

```text
🏆  Starting server at http://0.0.0.0:49483
```

You can interact with TMAN Designer using the following URL:

```text
http://127.0.0.1:49483/
```

> Note: If your TMAN Designer is running on a remote machine, please use the IP address of the remote machine to access TMAN Designer.

## Overview

After opening the TMAN Designer page in your browser, you will see the following interface:

![TMAN Designer Overview](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_overview.png)

In the upper right corner of TMAN Designer, you can see several function buttons:

* The rightmost button displays the current version of TMAN Designer and automatically checks for new version releases. If an update is available, an upward arrow will be displayed. Clicking it will open the download page for the new version.
* The first button on the left is the [TEN Agent](https://agent.theten.ai/) button. Clicking it will open the TEN Agent page. TEN Agent is a complete AI Agent implementation built using TEN Framework. One of the main goals of TMAN Designer is to enable developers to easily develop, debug, and customize TEN Agent through TMAN Designer.
* The second button on the left is the [TEN Framework](https://github.com/TEN-framework/ten-framework) GitHub page. Clicking it will open the GitHub repository for TEN Framework. You can visit this page to learn more about TEN Framework, view source code, or submit issues.
* On the far left, there are two buttons for switching between light and dark modes and selecting languages.

In the upper left corner of TMAN Designer, there are several dropdown menus that developers can use to access various functions of TMAN Designer.

## Working with TEN Apps

### Loading an Existing TEN App

You can load an existing TEN App using the Load App button in the App menu. After clicking the button, you will see the following interface:

![Loading an Existing TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_load_app.png)

Through this file browser dialog, developers can specify the root directory of an existing TEN App to load it.

### Managing Loaded TEN Apps

You can manage loaded TEN Apps using the Manage Loaded App(s) button in the App menu. After clicking the button, you will see the following Apps Manager interface:

![Managing Loaded TEN Apps](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_manage_loaded_apps.png)

Developers can use this dialog to:

* Unload loaded TEN Apps
* Reload specified TEN Apps
* Reload all loaded TEN Apps
* Install all dependencies for specified Apps
* Run functions for specified Apps

### Installing All Dependencies for an App

Developers can install various TEN packages in a TEN App. All TEN packages that a TEN App depends on are recorded in the `manifest.json` file. Here is an example of a `manifest.json` file:

```json
{
  "type": "app",
  "name": "aaa",
  "version": "0.8.18",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.8.18"
    },
    {
      "type": "extension",
      "name": "ffmpeg_muxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_demuxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_client",
      "version": "^0.1.0"
    },
    {
      "type": "protocol",
      "name": "msgpack",
      "version": "^0.8.18"
    }
  ],
  "scripts": {
    "start": "python3 tools/run_script.py start",
    "build": "python3 tools/run_script.py build"
  }
}
```

From the above configuration, we can see that this TEN App depends on the following TEN packages:

* `ten_runtime`: The system package of TEN Framework
* `ffmpeg_demuxer`: A TEN Extension package for decoding audio and video
* `ffmpeg_muxer`: A TEN Extension package for encoding audio and video
* `ffmpeg_client`: A TEN Extension package for processing audio and video using the other two ffmpeg extension packages
* `msgpack`: A TEN Protocol package for message serialization and deserialization using the MessagePack protocol

After completing the installation of all dependencies for the TEN App, the TEN package file structure in the system looks like this:

```text
├── ten_packages
│   ├── extension
│   │   ├── ffmpeg_client
│   │   ├── ffmpeg_demuxer
│   │   └── ffmpeg_muxer
│   ├── protocol
│   │   └── msgpack
│   └── system
│       ├── ffmpeg
│       └── ten_runtime
```

It's worth noting that even though it's not explicitly declared in the `manifest.json`, there is an additional `ffmpeg` system package in the system. This is because `ffmpeg_demuxer` and `ffmpeg_muxer` depend on the `ffmpeg` system package, and TEN Manager automatically identifies and installs these transitive dependencies.

Through the Apps Manager interface in TMAN Designer, developers can perform the Install All operation on a specified App to easily complete the installation of all dependencies. After clicking the button, you will see the following interface showing the installation progress:

![Installing All Dependencies for an App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_install_all.png)

### Installing TEN Extensions from the Cloud Store

You can open the TEN Cloud Store and install TEN Extensions to the current TEN App using the Open Extension Store button in the Graph menu. After clicking the button, you will see the following interface:

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_store.png)

In the TEN Cloud Store, developers can search for the required TEN Extensions and click the Install button to install them to the current TEN App. After installation, you can view the installed TEN Extensions in the current TEN App.

### Running a TEN App

In the Apps Manager, you can perform preset runtime operations on the specified App. After clicking the Run button in the Apps Manager, you will see the following interface:

![Running a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_run.png)

In the dialog, developers can select the preset operation to execute from the dropdown menu. These typically include the following operations:

* Run App: Start the App and display the App's output information in the TMAN Designer message window.
* Build App: Compile the App. Since TEN Framework supports multiple programming languages, some languages need to be compiled while others do not. For example, TEN Apps developed in C++ need to be compiled, while TEN Apps developed in Python do not.

Below is the build interface for a TEN App developed in C++:

![Building a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_build.png)

After the build is complete, you can continue with the startup operation for the TEN App. The system will display a window showing the TEN App's runtime logs in real-time:

![Starting a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_start.png)

## Working with TEN Graphs

### Loading an Existing TEN Graph

You can load an existing TEN Graph using the Open Existing Graph button in the Graph menu. After successful loading, you will see an interface similar to the following:

![TEN Graph](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_graph_overview.png)

Using this Graph as an example, it contains 3 TEN Extensions, namely:

* `ffmpeg_demuxer`
* `ffmpeg_muxer`
* `ffmpeg_client`

Each TEN Extension has its corresponding inputs and outputs. For example, `ffmpeg_demuxer` has the following inputs and outputs:

* 2 cmd inputs
* 1 cmd output
* 1 audio\_frame output
* 1 video\_frame output

Clicking on the number buttons for inputs and outputs opens a dialog like the following, which is used to manage input and output connections:

![TEN Message Input/Output Dialog](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_msg_info_dialog.png)

### Auto-Layout Graph

After manually adjusting the positions of TEN Extensions in the TEN Graph, developers can click the Auto Layout button in the Graph menu to automatically arrange the TEN Extensions in the TEN Graph.

### Opening the Context Menu for TEN Extensions

In the Graph, right-clicking on each TEN Extension opens the TEN Extension's context menu:

![TEN Extension Context Menu](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_context_menu.png)

In the context menu, you can open the TEN Extension's `manifest.json` file, `property.json` file, or directly start a terminal in the root directory of the TEN Extension, making it convenient for developers to modify and customize the TEN Extension:

![Opening Manifest](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_context_menu_effect.png)


# How to run local AI model in Python extension (/docs/ten_framework/0.11.59/ai_model/how_to_run_local_model_in_python_extensions)

In the TEN framework, extensions can utilize third-party AI services or run AI models locally to improve performance and reduce costs. This tutorial explains how to run a local AI model in a Python extension and how to interact with it within the extension.

## Step 1: Check Hardware Requirements

Before running an AI model locally, ensure that your hardware meets the necessary requirements. Key components to verify include:

* **CPU/GPU**: Check if the model requires specific processing power.
* **Memory**: Ensure sufficient memory to load and run the model.

Verify that your system can support the model’s demands to ensure smooth operation.

## Step 2: Install Necessary Software and Dependencies

Once your hardware is ready, install the required software and dependencies. Follow these steps:

1. **Operating System**: Ensure compatibility with your model. Most AI frameworks support Windows, macOS, and Linux, though specific versions may be required.

2. **Python Version**: Ensure compatibility with the TEN Python runtime and the model.

3. **Required Libraries**: Install necessary libraries such as:

   * TensorFlow
   * PyTorch
   * Numpy
   * vllm

   You can list the required dependencies in a `requirements.txt` file for easy installation.

4. **Download the Model**: Obtain the local version of the AI model you plan to run.

## Step 3: Implement Your Python Extension

Below is an example of how to implement a basic text generation feature using the `vllm` inference engine in a Python extension.

First, initialize the local model within the extension:

```python
from ten_runtime import (
    Extension,
    TenEnv,
    Cmd,
    CmdResult,
)
from vllm import LLM

class TextGenerationExtension(Extension):
    def on_init(self, ten_env: TenEnv) -> None:
        self.llm = LLM(model="<model_path>")
        ten_env.on_init_done()
```

Next, implement the `on_cmd` method to handle text generation based on the provided input:

```python
    def on_cmd(self, ten_env: TenEnv, cmd: Cmd) -> None:
        prompt = cmd.get_property_string("prompt")

        outputs = self.llm.generate(prompt)
        generated_text = outputs[0].outputs[0].text

        cmd_result = CmdResult.create(StatusCode.OK)
        cmd_result.set_property_string("result", generated_text)
        ten_env.return_result(cmd_result, cmd)
```

In this code, the `on_cmd` method retrieves the `prompt`, generates text using the model, and returns the generated text as the command result.

You can adapt this approach to implement other functionalities such as image recognition or speech-to-text by processing the relevant input types.

## Step 4: Unload the Model

It’s important to unload the model during extension cleanup to free resources:

```python
import gc
import torch

class TextGenerationExtension(Extension):
    ...

    def on_deinit(self, ten_env: TenEnv) -> None:
        del self.llm
        gc.collect()
        torch.cuda.empty_cache()
        torch.distributed.destroy_process_group()
        print("Successfully deleted the LLM pipeline and freed GPU memory!")
        ten_env.on_deinit_done()
```

This ensures efficient memory management, especially when working with GPU resources.

## Summary

Running a local model in a TEN Python extension is similar to native Python development. By loading and unloading the model in the appropriate extension lifecycle methods, you can easily integrate local AI models and interact with them effectively.


# API (/docs/ten_framework/0.11.59/api/api)

In the TEN Framework, the API definition of an extension is contained in the `manifest.json` file. When defining APIs, the TEN Framework does not use JSON Schema, but instead uses a JSON-like syntax to define API Schema.

## Why Not Use JSON Schema

The TEN Framework chooses not to use standard JSON Schema for the following reasons:

* **Type System Differences**: The TEN Framework's type system includes many types that JSON Schema does not have, such as `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float32`, `float64`, `buf`, `ptr`, etc.

* **Reduce Redundancy**: In the TEN Framework's API definitions, many places do not need flexible settings. For example, a message's schema must be an object type and cannot be other types. Requiring developers to explicitly write `"type": "object"` would be redundant and confusing.

Based on these reasons, the TEN Framework adopts a more natural definition approach, using JSON-like syntax to define API Schema.

## Top-level API Definition

The API Schema definition is placed in the `api` field of the `manifest.json` file:

```json
{
  "api": {
    "property": {
      "properties": {...},
      "required": [...]
    },
    "cmd_in": [{
      "name": "foo",
      "property": {
        "properties": {...},
        "required": [...]
      },
      "result": {
        "property": {
          "properties": {...},
          "required": [...]
        }
      }
    }]
  }
}
```

## API Schema Definition Patterns

### Expressing Object Type Fields

Use the `properties` field to describe the type of each property within an object type field:

```json
{
  "my_object": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "age": {
        "type": "int32"
      }
    }
  }
}
```

### Expressing Required and Optional Fields

Use a `required` field at the same level as the object type field to describe required and optional fields:

```json
{
  "some_concept": {
    "properties": {
      "foo": {},
      "bar": {}
    },
    "required": ["foo"]
  }
}
```

### Expressing Array Type Fields

Use the `items` field to describe the type of each element within an array type field:

```json
{
  "my_array": {
    "type": "array",
    "items": {
      "type": "string"
    }
  }
}
```

### Usage Principles for type Field

If a field can only be a specific type (constrained by the TEN Framework itself), there is no need to explicitly write the `type` field. If a field can be multiple types, then the `type` field needs to be explicitly written.

For example, if the `some_concept` field below can only be an object type, there is no need to explicitly write the `type` field. However, the `foo` field within it can be multiple types, so the `type` field needs to be explicitly written:

```json
{
  "some_concept": {
    "properties": {
      "foo": {
        "type": "string"
      },
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "string"
          }
        },
        "required": ["baz"]
      }
    },
    "required": ["foo"]
  }
}
```

Use the `required` field appropriately to ensure the existence of critical data:

```json
{
  "name": "critical_cmd",
  "property": {
    "properties": {
      "essential_param": {
        "type": "string"
      },
      "optional_param": {
        "type": "int32"
      }
    },
    "required": ["essential_param"]
  }
}
```

## Expressing Enum Type Fields

Use the `enum` field to describe all possible values of an enumeration type:

```json
{
  "api": {
    "property": {
      "properties": {
        "status": {
          "type": "string",
          "enum": ["CREATING", "ACTIVE", "DELETING", "FAILED"]
        }
      }
    }
  }
}
```

## Extension Property Schema

Complete example of defining extension properties:

```json
{
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        },
        "config": {
          "type": "object",
          "properties": {
            "enabled": {
              "type": "bool"
            },
            "settings": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          },
          "required": ["enabled"]
        }
      },
      "required": ["api_key", "max_tokens", "temperature", "config"]
    }
  }
}
```

## Command-like Message Schema

Commands support input (`cmd_in`) and output (`cmd_out`):

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "process_text",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            },
            "options": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "max_length": {
                  "type": "int32"
                }
              }
            }
          },
          "required": ["text"]
        },
        "result": {
          "property": {
            "properties": {
              "processed_text": {
                "type": "string"
              },
              "confidence": {
                "type": "float32"
              }
            },
            "required": ["processed_text"]
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "notify_completion",
        "property": {
          "properties": {
            "task_id": {
              "type": "string"
            },
            "status": {
              "type": "string"
            }
          },
          "required": ["task_id", "status"]
        }
      }
    ]
  }
}
```

## Data-like Message Schema

Schema definitions for data-type messages:

```json
{
  "api": {
    "data_in": [
      {
        "name": "input_data",
        "property": {
          "properties": {
            "content": {
              "type": "string"
            },
            "metadata": {
              "type": "object",
              "properties": {
                "timestamp": {
                  "type": "int64"
                }
              }
            }
          },
          "required": ["content"]
        }
      }
    ],
    "data_out": [
      {
        "name": "output_data",
        "property": {
          "properties": {
            "result": {
              "type": "string"
            }
          }
        }
      }
    ],
    "audio_frame_in": [
      {
        "name": "audio_input"
      }
    ],
    "audio_frame_out": [
      {
        "name": "audio_output"
      }
    ],
    "video_frame_in": [
      {
        "name": "video_input"
      }
    ],
    "video_frame_out": [
      {
        "name": "video_output"
      }
    ]
  }
}
```

## Common Errors and Solutions

<div className="fd-steps">
  <div className="fd-step">
    ### Property Names Not Following Conventions

    **Error Example**: Property names containing hyphens or starting with numbers

    ```json
    {
      "api-key": {  // Error: contains hyphens
        "type": "string"
      },
      "2nd_param": {  // Error: starts with a number
        "type": "int32"
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "api_key": {
        "type": "string"
      },
      "second_param": {
        "type": "int32"
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### Array Type Missing items Definition

    **Error Example**:

    ```json
    {
      "my_array": {
        "type": "array"
        // Missing items definition
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "my_array": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### Object Type Missing properties Definition

    **Error Example**:

    ```json
    {
      "my_object": {
        "type": "object"
        // Missing properties definition
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "my_object": {
        "type": "object",
        "properties": {
          "field1": {
            "type": "string"
          }
        }
      }
    }
    ```
  </div>
</div>

## Summary

The TEN Framework's Schema system provides a powerful and structured way to define and validate data structures, ensuring consistency and type safety between components. By following design principles and best practices, developers can create robust and maintainable TEN applications and extensions.

The system supports rich type definitions, complex API descriptions, and flexible message transformation capabilities, providing a solid foundation for building large-scale real-time multimodal applications.


# Log (/docs/ten_framework/0.11.59/api/log)

The TEN framework allows extensions developed in different languages to run within the same process. This creates a need to view logs from all these extensions in a unified log with a consistent format and information, making debugging easier.

To address this, the `ten_env` provides a logging API. Within each callback of an extension, an instance of `ten_env` can be accessed. Using this instance, you can call the logging API to output logs from extensions developed in various languages to a unified log output.

## Log Levels

The TEN framework supports the following log levels in increasing order of severity:

* **DEBUG** (1): Debug information
* **INFO** (2): General information
* **WARN** (3): Warning information
* **ERROR** (4): Error information

## API Interface

### C++ Language

```cpp
#include "ten_runtime/binding/cpp/ten.h"

class Extension {
public:
    void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr<ten::cmd_t> cmd) {
        // Using convenience macros
        TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
        TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
        TEN_ENV_LOG_WARN(ten_env, "Warning message");
        TEN_ENV_LOG_ERROR(ten_env, "Error occurred");

        // Direct method call with category and fields support
        ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
                   "Direct log call", nullptr, nullptr);
    }
};
```

C++ provides the following convenience macros:

* `TEN_ENV_LOG_DEBUG(ten_env, msg)`
* `TEN_ENV_LOG_INFO(ten_env, msg)`
* `TEN_ENV_LOG_WARN(ten_env, msg)`
* `TEN_ENV_LOG_ERROR(ten_env, msg)`

### Python Language

```python
from ten_runtime import Extension, TenEnv

class MyExtension(Extension):
    def on_cmd(self, ten_env: TenEnv, cmd):
        # Using different log level methods
        ten_env.log_debug("Debug information")
        ten_env.log_info("General information")
        ten_env.log_warn("Warning information")
        ten_env.log_error("Error information")

        # Supports formatted strings and additional parameters
        test_value = "example"
        ten_env.log_info(f"Processing command, test value: {test_value}")

        # Supports category and fields parameters
        ten_env.log_info("Log with category", category="my_extension")
```

Python provides the following methods:

* `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`

### Go Language

```go
package main

import "ten_runtime/ten"

type MyExtension struct {
    ten.DefaultExtension
}

func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    // Using different log level methods
    tenEnv.LogDebug("Debug information")
    tenEnv.LogInfo("General information")
    tenEnv.LogWarn("Warning information")
    tenEnv.LogError("Error information")

    // Supports formatting
    cmdName := cmd.GetName()
    tenEnv.LogInfo("Processing command: " + cmdName)

    // Using the full Log method for additional parameters
    category := "my_extension"
    tenEnv.Log(ten.LogLevelInfo, "Log with category", &category, nil, nil)
}
```

Go provides the following methods:

* `LogDebug(msg string) error`
* `LogInfo(msg string) error`
* `LogWarn(msg string) error`
* `LogError(msg string) error`
* `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`

### Node.js/TypeScript Language

```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";

class MyExtension extends Extension {
    async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
        // Using different log level methods
        tenEnv.logDebug("Debug information");
        tenEnv.logInfo("General information");
        tenEnv.logWarn("Warning information");
        tenEnv.logError("Error information");

        // Supports string concatenation
        const cmdName = cmd.getName();
        tenEnv.logInfo("Processing command: " + cmdName);

        // Supports category and fields parameters
        tenEnv.logInfo("Log with category", "my_extension");
    }
}
```

Node.js provides the following methods:

* `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
* `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
* `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
* `logError(message: string, category?: string, fields?: Value): TenError | undefined`

## Usage Examples

### Basic Usage

```python
# Python example
def on_start(self, ten_env: TenEnv):
    ten_env.log_info("Extension starting")
    ten_env.on_start_done()

def on_cmd(self, ten_env: TenEnv, cmd):
    cmd_name = cmd.get_name()
    ten_env.log_debug(f"Received command: {cmd_name}")

    # Command processing logic
    if cmd_name == "hello":
        ten_env.log_info("Processing hello command", category="command_handler")
        # ... processing logic
    else:
        ten_env.log_warn(f"Unknown command: {cmd_name}")
```

### Using in Threads

```python
import threading
import time

def log_routine(self, ten_env: TenEnv):
    """Log messages in a separate thread"""
    while not self.stop_flag:
        self.log_count += 1
        ten_env.log_info(f"Log message {self.log_count}")
        time.sleep(0.05)

def on_start(self, ten_env: TenEnv):
    # Start logging thread
    self.log_thread = threading.Thread(
        target=self.log_routine, args=(ten_env,)
    )
    self.log_thread.start()
    ten_env.on_start_done()
```

### Error Handling

```python
def on_cmd(self, ten_env: TenEnv, cmd):
    try:
        # Business logic
        result = self.process_command(cmd)
        ten_env.log_info("Command processed successfully")
    except ValueError as e:
        ten_env.log_error(f"Parameter error: {str(e)}", category="error_handling")
    except Exception as e:
        ten_env.log_error(f"Fatal error: {str(e)}", category="error_handling")
        raise
```

## Log Format and Output

The TEN framework's logging system automatically includes the following information:

* **Timestamp**: Time when the log was recorded
* **Log Level**: DEBUG, INFO, WARN, ERROR
* **Function Name**: Name of the function that called the log (if available)
* **File Name**: Source file name where the log was called (if available)
* **Line Number**: Code line number where the log was called (if available)
* **Category**: Optional log category information
* **Message Content**: User-provided log message

The output format is similar to:

```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] User log message
```

## Advanced Log Configuration

The TEN framework supports advanced logging functionality through the `property.json` file configuration. This allows you to customize the log output format, destination, and filtering rules.

### Configuration Structure

In the `property.json` file, you can define advanced log configuration in the `ten.log` section:

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "my_extension"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

### Configuration Components

<div className="fd-steps">
  <div className="fd-step">
    #### Log Handlers

    Each handler contains three main components:

    * **matchers**: Define which log messages will be processed by this handler
    * **formatter**: Define the log output format
    * **emitter**: Define the log output destination
  </div>

  <div className="fd-step">
    #### Matchers

    Matchers are used to filter log messages:

    ```json
    {
      "level": "info",      // Log level: "off", "debug", "info", "warn", "error"
      "category": "auth"    // Optional: log category (string)
    }
    ```

    Supported log levels:

    * `"off"`: Disable logging
    * `"debug"`: Debug level
    * `"info"`: Info level
    * `"warn"`: Warning level
    * `"error"`: Error level
  </div>

  <div className="fd-step">
    #### Formatter

    Controls the log output format:

    ```json
    {
      "type": "plain",      // Format type: "plain" or "json"
      "colored": true       // Optional: enable colored output (only for plain format)
    }
    ```

    * **plain**: Human-readable plain text format
    * **json**: JSON structured format, convenient for log analysis tools
  </div>

  <div className="fd-step">
    #### Emitter

    Defines the log output destination:

    ##### Console Output

    ```json
    {
      "type": "console",
      "config": {
        "stream": "stdout",  // "stdout" or "stderr"
        "encryption": {      // Optional: encryption configuration
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### File Output

    ```json
    {
      "type": "file",
      "config": {
        "path": "/path/to/logfile.log",  // Log file path
        "encryption": {                  // Optional: encryption configuration
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### OTLP Output

    OTLP (OpenTelemetry Protocol) output allows you to send logs to OpenTelemetry-compatible backend systems, such as Jaeger, Prometheus, Grafana, and more.

    ```json
    {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4317",  // OTLP receiver endpoint URL
        "protocol": "grpc",                   // Optional: transport protocol, "grpc" (default) or "http"
        "headers": {                          // Optional: custom HTTP headers
          "Authorization": "Bearer token",
          "X-Custom-Header": "value"
        },
        "service_name": "my-service"          // Optional: service name for identifying log source
      }
    }
    ```

    Configuration parameters:

    * **endpoint**: OTLP receiver endpoint URL (required)
    * **protocol**: Transport protocol, supports `"grpc"` or `"http"`, defaults to `"grpc"`
    * **headers**: Custom HTTP request headers for authentication or metadata (optional)
    * **service\_name**: Service name for identifying different services in log aggregation systems (optional)
  </div>
</div>

### Configuration Examples

#### Example 1: Multi-level Log Separation

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/errors.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### Example 2: Category-based Log Separation

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "database"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/database.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info",
              "category": "api"
            }
          ],
          "formatter": {
            "type": "plain"
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### Example 3: Encrypted Log Output

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/secure/logs/app.log",
              "encryption": {
                "enabled": true,
                "algorithm": "AES-CTR",
                "params": {
                  "key": "your-32-character-encryption-key",
                  "nonce": "your-16-character-nonce"
                }
              }
            }
          }
        }
      ]
    }
  }
}
```

#### Example 4: OTLP Log Output to OpenTelemetry Collector

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "http://localhost:4317",
              "protocol": "grpc",
              "service_name": "ten-framework-app"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "https://api.monitoring.example.com/v1/logs",
              "protocol": "http",
              "headers": {
                "Authorization": "Bearer your-api-token",
                "X-Environment": "production"
              },
              "service_name": "ten-framework-app"
            }
          }
        }
      ]
    }
  }
}
```

### Configuration Notes

1. **Multiple Handlers**: You can define multiple handlers to implement complex log routing strategies
2. **Matching Priority**: Handlers are matched in the order they appear in the configuration file
3. **Encryption Security**: Encryption keys and nonces should be stored securely, not hardcoded in configuration files
4. **File Permissions**: Ensure log file paths have appropriate write permissions
5. **Performance Considerations**: JSON format may produce larger log files than plain format
6. **OTLP Endpoints**:
   * When using gRPC protocol, the default port is typically 4317
   * When using HTTP protocol, the default port is typically 4318
   * Ensure the OTLP collector endpoint is accessible and properly configured
7. **Authentication**: When using OTLP, if the backend requires authentication, add appropriate credentials in the headers

## Best Practices

1. **Choose appropriate log levels**:
   * Use `DEBUG` for detailed debugging information
   * Use `INFO` for normal business processes
   * Use `WARN` for potential issues
   * Use `ERROR` for error information

2. **Provide meaningful log messages**:
   * Include contextual information (such as command names, parameter values, etc.)
   * Use descriptive messages rather than simple identifiers
   * Use the category parameter appropriately to classify logs

3. **Avoid excessive logging**:
   * Don't use too many logs in high-frequency loops
   * Appropriately adjust log levels in production environments
   * Use advanced log configuration matchers to filter unnecessary logs

4. **Thread safety**:
   * The TEN framework's logging API is thread-safe
   * Log methods can be safely called from any thread

5. **Advanced configuration recommendations**:
   * Use different categories for different modules to facilitate log analysis
   * Consider using file output in production environments for log persistence
   * Use encryption functionality for sensitive logs
   * Regularly rotate log files to control disk usage
   * Use JSON format for easier integration with log analysis tools
   * Consider using OTLP output to integrate logs into modern observability platforms (e.g., Grafana, Jaeger, Datadog)
   * OTLP output seamlessly integrates with the OpenTelemetry ecosystem, enabling distributed tracing and log correlation

The overall effect is shown in the image below:

![Unified log output for multi-language extensions](https://ten-framework-assets.s3.amazonaws.com/doc-assets/log.png)


# required (/docs/ten_framework/0.11.59/api/required)

## Principles of the `required` Field

1. **In a communication link, the source's `required` field must be a superset of the target's `required` field**

   This means that if a target extension requires a field to be mandatory, then the source extension must also mark that field as required. This ensures that messages will not lack necessary fields during transmission.

2. **Fields with the same name in both source and target `required` fields must have compatible types**

   The framework validates field type compatibility to ensure data can be correctly transmitted and parsed. Type compatibility checks include basic type matching, array item type matching, and object property type matching.

3. **Graph Check validates the integrity of the entire message transmission path**

   * Checks whether the output message schema of the source extension is compatible with the input message schema of the target extension
   * Verifies the consistency of `required` fields throughout the transmission path
   * Ensures that message transformation rules (if any) correctly handle all required fields

Graph Check is performed before deployment, helping developers identify potential message transmission issues early and avoid runtime errors.

## Usage of the `required` Field

### When Extensions Send Messages

When an extension calls `send_<foo>(msg_X)` or `return_result(result_Y)`, the framework checks `msg_X` or `result_Y` against the corresponding schema in the extension. If `msg_X` or `result_Y` lacks any field marked as `required` in the schema, the schema check will fail.

The following three scenarios are handled in the same way:

1. **When sending a command and schema check fails**

   `send_<foo>` will immediately return `false`, and the error message will contain the schema check failure details.

2. **When `return_result` fails schema check**

   `return_result` will return `false`, and the error message will contain the schema check failure details, typically showing the missing required fields, such as: "the required properties are absent: 'foo'".

3. **When sending data-like messages (such as data, audio frames, or video frames)**

   `send_<foo>` will return `false`, and the error message will contain the schema check failure details.

**Note:** The framework's schema system only takes effect when fields are present. If a field is defined in the schema but does not exist in the message, and that field is not listed in `required`, no type checking will be performed. Type validation only occurs when fields exist both in the message and in the schema definition.

### When Extensions Receive Messages

Before the TEN runtime passes `msg_X` or `result_Y` to the extension's `on_<foo>()` or result handler, it checks whether all `required` fields defined in the schema of `msg_X` or `result_Y` are present. If any `required` fields are missing, the schema check will fail.

1. **When the incoming message is a command**

   The TEN runtime will return an error to the previous extension, with an error message indicating the missing required fields.

2. **When the incoming message is a TEN command result**

   The TEN runtime will change the result's `status_code` to error, add the missing `required` fields, and set the values of these fields to default values based on their types.

3. **When the incoming message is a data-type TEN message**

   The TEN runtime will directly discard the data-type message.

### Error Message Format

When `required` field validation fails, the framework provides detailed error information:

* Single missing field: `"the required properties are absent: 'field_name'"`
* Multiple missing fields: `"the required properties are absent: 'field1', 'field2'"`
* Missing fields in nested objects: `".nested_object: the required properties are absent: 'field_name'"`


# Type System (/docs/ten_framework/0.11.59/api/type_system)

## Types

The TEN framework type system is the system used within the TEN framework to define the data types of values. Developers can declare the types of messages or extension properties through TEN schema.

The TEN framework type system includes basic types and composite types. The basic types include the following:

| Type    | Description                                               |
| ------- | --------------------------------------------------------- |
| null    | Null value type.                                          |
| bool    | Boolean value, true or false.                             |
| int8    | 8-bit signed integer.                                     |
| int16   | 16-bit signed integer.                                    |
| int32   | 32-bit signed integer.                                    |
| int64   | 64-bit signed integer.                                    |
| uint8   | 8-bit unsigned integer.                                   |
| uint16  | 16-bit unsigned integer.                                  |
| uint32  | 32-bit unsigned integer.                                  |
| uint64  | 64-bit unsigned integer.                                  |
| float32 | Single precision (32-bit) IEEE 754 floating-point number. |
| float64 | Double precision (64-bit) IEEE 754 floating-point number. |
| string  | Unicode (UTF-8) character sequence.                       |
| buf     | Sequence of 8-bit unsigned bytes.                         |
| ptr     | Pointer to a memory address.                              |

Composite types include the following:

| Type   | Description                                                    |
| ------ | -------------------------------------------------------------- |
| array  | Collection of elements of the same type.                       |
| object | Represents complex key/value pairs. Key type is always string. |

For basic types, properties can be accessed or set using methods such as `get_property()` / `set_property()`. For example:

```cpp
// Get property value
int32_t value = cmd.get_property_int32("property_name");

// Set property value
cmd.set_property("property_name", 100);
```

For composite types, it is typically necessary to use related serialization methods. For example:

```go
type MyProp struct {
    Name string `json:"name"`
}

var prop MyProp
bytes, _ := json.Marshal(&prop)
cmd.SetPropertyFromJSONBytes("property_name", bytes)
```

## JSON Type Mapping

When creating TEN values from JSON, the TEN framework uses the following mapping rules:

| JSON Type | TEN Type Mapping Rules                                                          |
| --------- | ------------------------------------------------------------------------------- |
| Integer   | Maps to int32 if the value is within int32 range; otherwise maps to int64       |
| Float     | Maps to float32 if the value is within float32 range; otherwise maps to float64 |
| String    | Maps to string                                                                  |
| Boolean   | Maps to bool                                                                    |
| null      | Maps to null                                                                    |
| Array     | Maps to array                                                                   |
| Object    | Maps to object                                                                  |

It's important to note that JSON numeric types, when no explicit schema is defined, will automatically select the appropriate TEN type based on the numeric value size. This ensures data precision and efficiency.

## Types and Schema

If a TEN schema is specified, the property type will be determined according to the corresponding TEN schema. If no TEN schema is specified, the property type will be determined based on the type of the initial value assignment. For example, if the initial assignment is C++'s `int32_t`, the property type will be `int32`; if the initial assignment is done using JSON, the type will be determined according to JSON processing rules.

When a TEN schema is defined, JSON data will be forcibly converted to the type specified by the schema. If the conversion fails, an error will be reported.

## Conversion Rules

The TEN framework supports flexible automatic conversion between different types of values. Conversions are divided into two types: safe conversions and unsafe conversions.

### Type Compatibility

The TEN framework uses type compatibility rules to determine whether conversion between two types is possible:

* **Integer type compatibility**: All integer types (int8, int16, int32, int64, uint8, uint16, uint32, uint64) are mutually compatible
* **Float compatible with integer**: Floating-point types can accept values of integer types (when the integer value is within the floating-point representation range)
* **Other types**: bool, ptr, string, array, object, buf, null types are only compatible with themselves

### Safe Conversion (Must Succeed)

Converting a lower precision type to a higher precision type is always safe and guaranteed to succeed, as the higher precision type can fully accommodate the value of the lower precision type without data loss.

In the TEN type system, safe conversion rules are as follows:

1. Within signed integer types, from lower to higher precision
2. Within unsigned integer types, from lower to higher precision
3. From smaller range unsigned integers to larger range signed integers (when the value is within the target type range)
4. Within floating-point types, from float32 to float64

| From    | To (Allowed Safe Conversions)                    |
| ------- | ------------------------------------------------ |
| int8    | int16 / int32 / int64                            |
| int16   | int32 / int64                                    |
| int32   | int64                                            |
| uint8   | uint16 / uint32 / uint64 / int16 / int32 / int64 |
| uint16  | uint32 / uint64 / int32 / int64                  |
| uint32  | uint64 / int64                                   |
| float32 | float64                                          |

For example:

```cpp
// Set property value. The type of `property_name` in TEN runtime is `int32`.
cmd.set_property("property_name", 100);

// Get property value. Correct.
int32_t value = cmd.get_property_int32("property_name");

// Get property value. Correct. TEN type system will automatically convert the type to `int64`.
int64_t value2 = cmd.get_property_int64("property_name");

// Get property value. Incorrect, an error will be thrown because this is an unsafe conversion.
int16_t error_type = cmd.get_property_int16("property_name");
```

### Unsafe Conversion (May Fail)

Converting a higher precision type to a lower precision type is unsafe because the value of the higher precision type may exceed the range of the lower precision type, leading to data loss. This conversion is called unsafe conversion. When performing unsafe conversion, the TEN runtime performs strict range checking. If an overflow or precision loss occurs, the TEN type system will throw an error.

In the TEN framework type system, unsafe conversions include:

1. Converting higher precision integers to lower precision integers
2. Converting signed integers to unsigned integers (when the value is negative)
3. Converting unsigned integers to signed integers (when the value exceeds the signed integer range)
4. Converting `float64` to `float32`
5. Converting floating-point numbers to integers (when there are decimal parts or the value exceeds integer range)
6. Converting integers to floating-point numbers (when precision loss may occur)

| Conversion Type                | Success Condition                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------ |
| High precision → Low precision | Value must be within target type range                                               |
| Signed → Unsigned              | Value must be non-negative and within target type range                              |
| Unsigned → Signed              | Value must be within target type range                                               |
| float64 → float32              | Value must be within float32 range without precision loss                            |
| Floating-point → Integer       | Value must be within target integer type range and have no decimal part              |
| Integer → Floating-point       | Integer value must be exactly representable in floating-point without precision loss |

#### Range Check Details

TEN framework implements precise range checking algorithms:

* **Integer range checking**: Uses standard MIN/MAX constants for comparison
* **Floating-point range checking**: Uses `ldexp` function to calculate precise boundary values
* **Precision checking**: For integer to floating-point conversion, performs round-trip conversion verification to ensure no precision loss

### When Conversion Is Triggered

The TEN runtime performs type conversion in the following situations:

1. **When loading `property.json`**
   * JSON integers are parsed to appropriate integer types based on numeric value size
   * JSON floating-point numbers are parsed to appropriate floating-point types based on numeric value size
   * If the property has a defined TEN schema, forced type conversion will be performed according to the schema (may be unsafe conversion)

2. **When calling methods like `set_property_from_json()`**
   * When passing serialized JSON strings, type conversion will be performed according to the schema

3. **During property access**
   * When using `get_property_xxx()` methods to get properties, the system will attempt to convert the stored type to the requested type
   * This may trigger safe or unsafe conversions

4. **During message passing**
   * When message properties need to be converted between different types

### Error Handling

When type conversion fails, the TEN framework will:

* Return error status codes and detailed error information
* Provide source type, target type, and failure reason in the error object
* For out-of-range conversions, the error message format is: "out of range of \[target\_type]"
* For incompatible type conversions, the error message format is: "unsupported conversion from `[source_type]` to `[target_type]`"

#### Common Error Examples

```cpp
// Range overflow error
int8_t small_value = cmd.get_property_int8("large_number");  // If large_number = 200
// Error: out of range of int8

// Type incompatibility error
std::string str_value = cmd.get_property_string("integer_prop");
// Error: unsupported conversion from `int32` to `string`

// Precision loss error
float float_value = cmd.get_property_float32("precise_double");  // If precise_double exceeds float32 precision
// Error: out of range of float32
```

## Best Practices

1. **Use appropriate types**: Choose suitable integer and floating-point types based on the actual data range
2. **Explicitly define schema**: Clearly define property types in manifest.json to avoid type conversion ambiguity
3. **Handle conversion errors**: Properly handle potential errors from type conversions in your code
4. **Understand JSON mapping**: Be aware of JSON data to TEN type mapping rules to avoid unexpected type conversions


# Development Process (/docs/ten_framework/0.11.59/development/development_process)

Basically, by using the TEN framework to complete the desired scenarios, there are two development approaches or perspectives:

1. From small extensions to large scenarios
2. From large scenarios to small extensions

**Note:** Regardless of which of these two development approaches you take, you can either write your own extensions from scratch or download pre-built extensions from the TEN cloud store.

In the first approach, **from small extensions to large scenarios**, the development process typically follows these steps:

1. Develop extensions.
2. Assemble the extensions into a graph, and further create an agent, for example, [TEN-Agent](https://github.com/TEN-framework/TEN-Agent).

In the second approach, **from large scenarios to small extensions**, the development process typically follows these steps:

1. Select the desired agent template, for example, [TEN-Agent](https://github.com/TEN-framework/TEN-Agent).
2. Replace some of the extensions within the chosen agent template with your own.

In this second development approach, you can ensure that the behavior of the replaced extension matches the original one by using the framework for **standalone testing** of extensions. In this process, the developer only needs to understand concepts related to TEN extensions, including how to independently develop and test a TEN extension. All other TEN-related concepts outside of extension development and testing do not need to be understood, as these concepts are encapsulated within the selected TEN agent.


# Complete Guide to TEN Extension Development (/docs/ten_framework/0.11.59/development/how_to_develop_with_ext)

TEN Framework provides rich extension templates to help developers quickly create extensions and complete the entire process from development to testing. This guide will demonstrate in detail how to use C++, Go, Python, and Node.js for the complete extension development workflow through practical operations.

## Pre-development Preparation

### Environment Requirements

Before starting extension development, please ensure your development environment is properly configured. Verify the installation with the following command:

```bash
tman --version
```

Under normal circumstances, you should see version information output similar to the following:

```bash
TEN Framework version: <version>
```

> **Important Note**: Please ensure you are using `tman` version >= 0.10.12. If your version is too low, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to install the latest version.

### Development Workflow Overview

Regardless of which programming language you use, TEN extension development follows the following standardized workflow:

1. **Project Creation** - Generate extension project skeleton using official templates
2. **Dependency Installation** - Configure runtime environment and install necessary dependency packages
3. **Core Development** - Implement extension business logic and functional code
4. **Build and Test** - Compile project (if needed) and execute unit tests
5. **Debug and Optimize** - Use professional debugging tools to locate and solve problems

***

## C++ Extension Development

C++ extensions are suitable for application scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency data computation, low-level system operations, etc.

### Create Project

Use the official C++ extension template provided by TEN to quickly create a new project:

```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```

After successful project creation, you will get the following complete project structure:

```bash
my_example_ext_cpp/
├── BUILD.gn              # GN build system configuration file
├── manifest.json         # Extension metadata and configuration information
├── property.json         # Extension properties and parameter configuration
├── src/
│   └── main.cc           # Extension core implementation code
├── tests/
│   ├── basic.cc          # Basic functionality test cases
│   └── gtest_main.cc     # Test framework program entry
├── include/              # Header file storage directory
├── tools/                # Auxiliary tools and scripts
└── .vscode/
    └── launch.json       # VSCode debugging configuration file
```

### Environment Configuration Verification

Enter the project directory and verify that the build tools are working properly:

```bash
cd my_example_ext_cpp
tgn --version
```

> **Expected Output**:

```bash
0.1.0
```

If the command cannot be executed, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to check whether the TEN Framework development environment is correctly installed.

### Dependency Package Installation

Install all dependency packages required for extension runtime:

```bash
tman install --standalone
```

After successful installation, you will see detailed installation logs similar to the following:

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done

🏆  Install successfully in 1 second
```

### Project Build

TEN framework provides two convenient build methods for you to choose from:

#### Method 1: Manual Step-by-step Build

```bash
# Step 1: Generate build configuration files (enable standalone test mode)
tgn gen linux x64 debug -- ten_enable_standalone_test=true

# Step 2: Execute project build
tgn build linux x64 debug
```

#### Method 2: Use tman Shortcut Command

```bash
tman run build
```

After the build is complete, check the generated executable test file:

```bash
ls -la bin/
# You should see: my_example_ext_cpp_test
```

### Run Tests

Verify that the extension functionality is working properly:

#### Method 1: Execute Test File Directly

```bash
./bin/my_example_ext_cpp_test
```

#### Method 2: Use tman Unified Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN      ] Test.Basic
[       OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[  PASSED  ] 1 test.
```

> **Success Indicator**: When you see `[  PASSED  ] 1 test.` it means all tests have passed.

### Core Code Structure Explanation

#### Extension Main Implementation (src/main.cc)

The core class of C++ extensions needs to inherit from the `ten::extension_t` base class and implement complete lifecycle management methods:

```cpp
class my_example_ext_cpp_t : public ten::extension_t {
 public:
  // Extension initialization phase - basic configuration and resource pre-allocation
  void on_init(ten::ten_env_t &ten_env) override;

  // Extension startup phase - officially start processing business logic
  void on_start(ten::ten_env_t &ten_env) override;

  // Command handler - handle command requests from other extensions or applications
  void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr<ten::cmd_t> cmd) override;

  // Data handler - handle general data flow
  void on_data(ten::ten_env_t &ten_env, std::unique_ptr<ten::data_t> data) override;

  // Audio frame handler - handle real-time audio stream data
  void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::audio_frame_t> frame) override;

  // Video frame handler - handle real-time video stream data
  void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::video_frame_t> frame) override;

  // Extension stop phase - clean up resources and graceful shutdown
  void on_stop(ten::ten_env_t &ten_env) override;
};
```

#### Test Framework Implementation (tests/basic.cc)

Use TEN-specific test framework to write complete unit tests:

```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    // Create command object for testing
    auto new_cmd = ten::cmd_t::create("foo");

    // Send command to extension and verify response results
    ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
      // Verify the correctness of test results here
    });
  }
};
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

1. Install the **CodeLLDB** extension plugin for VSCode
2. Set breakpoints in source code
3. Select the "standalone test (lldb, launch)" debug configuration
4. Press F5 to start debugging session

Debug configuration file `.vscode/launch.json` content:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (lldb, launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

#### Command Line Debugging

Use the classic GDB debugger for command-line debugging:

```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt  # View complete function call stack
```

***

## Go Extension Development

Go extensions provide a good balance between high performance and development efficiency, particularly suitable for building network services, concurrent processing, microservice architecture and other application scenarios.

### Create Project

Create a new project using the Go extension template:

```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```

After successful project creation, it will display:

```bash
🏆  Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```

The complete project structure is as follows:

```bash
my_example_ext_go/
├── extension.go             # Extension core implementation code
├── go.mod                   # Go module dependency management configuration
├── manifest.json            # Extension metadata information
├── property.json            # Extension property configuration
├── README.md                # Project documentation
├── tests/                   # Test-related files
│   ├── basic_tester.go      # Test logic implementation
│   ├── basic_tester_test.go # Test case definition
│   ├── main_test.go         # Test program entry
│   └── bin/start            # Test startup script
└── .vscode/launch.json      # VSCode debug configuration
```

### Dependency Package Installation

Install dependency packages required for project runtime:

```bash
tman install --standalone
```

### Run Tests

Verify the correctness of extension functionality:

#### Method 1: Use Startup Script

```bash
./tests/bin/start
```

#### Method 2: Use tman Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
=== RUN   TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```

### Core Code Structure Explanation

#### Extension Implementation (extension.go)

Core implementation structure of Go extensions:

```go
import (
    ten "ten_framework/ten_runtime"
)

type ExampleExtension struct {
    ten.DefaultExtension
}

// Extension startup lifecycle
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStart")
    tenEnv.OnStartDone()
}

// Command handling logic
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    tenEnv.LogDebug("OnCmd")
    cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
    tenEnv.ReturnResult(cmdResult, nil)
}

// Extension stop lifecycle
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStop")
    tenEnv.OnStopDone()
}
```

#### Test Framework (tests/basic\_tester.go)

Go extension test framework implementation:

```go
type BasicExtensionTester struct {
    ten.DefaultExtensionTester
}

func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
    // Create command object for testing
    cmdTest, _ := ten.NewCmd("test")

    // Send command and handle response results
    tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
        if err != nil {
            panic(err)
        }
        statusCode, _ := cr.GetStatusCode()
        if statusCode != ten.StatusCodeOk {
            panic(statusCode)
        }
        tenEnvTester.StopTest(nil)
    })

    tenEnvTester.OnStartDone()
}
```

### Development Environment Optimization

To improve development experience and debugging convenience, it is recommended to create a `go.work` workspace file in the extension root directory:

```go
go 1.18

use (
    .
    .ten/app/ten_packages/system/ten_runtime_go/interface
)
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure the official Go extension is installed, then use the following debug configuration:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (go, launch)",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}/tests/",
      "env": {
        "CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
        "LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
      },
      "args": ["-test.v"]
    }
  ]
}
```

***

## Python Extension Development

Python extensions have the highest development efficiency, particularly suitable for rapid prototyping, AI/ML application integration, complex business logic implementation and other scenarios.

### Create Project

Create a project using the Python async extension template:

```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```

Complete project structure:

```bash
my_example_ext_python/
├── extension.py         # Extension core implementation code
├── addon.py             # Extension plugin registration entry
├── __init__.py          # Python package initialization file
├── requirements.txt     # Python dependency package list
├── manifest.json        # Extension metadata configuration
├── property.json        # Extension property configuration
├── tests/               # Test-related files
│   ├── test_basic.py    # Basic test cases
│   ├── conftest.py      # pytest configuration file
│   └── bin/start        # Test startup script
└── .vscode/launch.json  # VSCode debug configuration
```

### Dependency Package Installation

Install Python dependency packages required for the project:

```bash
tman install --standalone
```

### Run Tests

Verify Python extension functionality:

#### Method 1: Use Startup Script

```bash
./tests/bin/start
```

#### Method 2: Use tman Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py .                                                                                [100%]
============================================ 1 passed in 0.04s =======================================
```

### Core Code Structure Explanation

#### Extension Implementation (extension.py)

Python extensions recommend using modern async programming patterns for better performance and concurrent processing capabilities:

```python
from ten_runtime import (
    AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
    Cmd, StatusCode, CmdResult, Data
)

class ExampleExtension(AsyncExtension):
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_init")

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_start")
        # TODO: Read configuration files here, initialize necessary resources

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_name = cmd.get_name()
        ten_env.log_debug(f"on_cmd name {cmd_name}")

        # TODO: Implement specific business logic processing here
        cmd_result = CmdResult.create(StatusCode.OK, cmd)
        await ten_env.return_result(cmd_result)

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_stop")
        # TODO: Clean up resources here, perform graceful shutdown
```

#### Plugin Registration Entry (addon.py)

Extension plugin registration and creation logic:

```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension

@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
    def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
        ten_env.log_info("on_create_instance")
        ten_env.on_create_instance_done(ExampleExtension(name), context)
```

#### Test Implementation (tests/test\_basic.py)

Complete async test framework implementation:

```python
from ten_runtime import (
    AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
    TenError, TenErrorCode
)

class ExtensionTesterBasic(AsyncExtensionTester):
    async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
        # Create command object for testing
        new_cmd = Cmd.create("hello_world")

        ten_env.log_debug("send hello_world")
        result, err = await ten_env.send_cmd(new_cmd)

        # Verify the correctness of test results
        if (err is not None or result is None
            or result.get_status_code() != StatusCode.OK):
            ten_env.stop_test(TenError.create(
                TenErrorCode.ErrorCodeGeneric,
                "Failed to send hello_world"
            ))
        else:
            ten_env.stop_test()

def test_basic():
    tester = ExtensionTesterBasic()
    tester.set_test_mode_single("my_example_ext_python")
    err = tester.run()
    if err is not None:
        assert False, err.error_message()
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure you have installed the Python extension and debugpy debugger, use the following configuration for debugging:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (debugpy, launch)",
      "type": "debugpy",
      "request": "launch",
      "python": "/usr/bin/python3",
      "module": "pytest",
      "args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
      "env": {
        "TEN_ENABLE_PYTHON_DEBUG": "true",
        "PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
      },
      "console": "integratedTerminal"
    }
  ]
}
```

***

## Node.js Extension Development

Node.js extensions provide a modern JavaScript/TypeScript development experience, particularly suitable for Web application integration, rapid prototyping, frontend technology stack extensions and other scenarios. Thanks to Node.js's async characteristics and rich ecosystem, developers can easily build efficient real-time applications.

### Create Project

Use the official Node.js extension template provided by TEN to quickly create a new project:

```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```

After successful project creation, you will get the following complete project structure:

```bash
my_example_ext_nodejs/
├── manifest.json         # Extension metadata and configuration information
├── property.json         # Extension properties and parameter configuration
├── package.json          # Node.js dependency package management configuration
├── tsconfig.json         # TypeScript compiler configuration
├── src/
│   └── index.ts          # Extension core implementation code
├── tests/                # Test-related files
│   ├── src/
│   │   ├── index.ts      # Tester implementation
│   │   ├── index.spec.ts # Test case definition
│   │   └── main.spec.ts  # Test framework configuration
│   ├── bin/start         # Test startup script
│   ├── package.json      # Test dependency configuration
│   └── tsconfig.json     # Test TypeScript configuration
├── tools/                # Auxiliary tools and scripts
└── .vscode/
    └── launch.json       # VSCode debug configuration file
```

### Environment Configuration Verification

Enter the project directory and verify the development environment:

```bash
cd my_example_ext_nodejs
node --version
npm --version
```

> **Expected Output**:

```bash
v18.0.0 or higher
8.0.0 or higher
```

### Dependency Package Installation

Install all dependency packages required for extension runtime:

```bash
tman install --standalone
```

After successful installation, you will see detailed installation logs similar to the following:

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       2/2       Done

🏆  Install successfully in 1 second
```

> **Important Note**: `tman install --standalone` will create a `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory in the project directory. Subsequent build and test operations need to be performed in this directory.

### Project Build

Node.js extensions are developed using TypeScript and need to install standalone mode dependencies first, then compile to JavaScript:

#### Method 1: Manual Build

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Install standalone mode Node.js dependencies
npm run standalone-install

# Compile TypeScript code
npm run build
```

#### Method 2: Use tman Shortcut Command

```bash
tman run build
```

After the build is complete, check the generated compilation results:

```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# You should see: index.js and related mapping files
```

### Run Tests

Verify that the extension functionality is working properly:

#### Method 1: Use Startup Script

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Run tests
./tests/bin/start
```

#### Method 2: Use tman Unified Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case2 (1010ms)

  2 passing (2s)
```

> **Success Indicator**: When you see all test cases showing `✓` and finally displaying `passing`, it means all tests have passed.

### Core Code Structure Explanation

#### Extension Main Implementation (src/index.ts)

The core class of Node.js extensions needs to inherit from the `Extension` base class and implement complete lifecycle management methods:

```typescript
import {
  Addon,
  RegisterAddonAsExtension,
  Extension,
  TenEnv,
  Cmd,
  CmdResult,
  StatusCode,
} from "ten-runtime-nodejs";

class DefaultExtension extends Extension {
  constructor(name: string) {
    super(name);
  }

  // Extension configuration phase - read and validate configuration parameters
  async onConfigure(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onConfigure");
  }

  // Extension initialization phase - basic configuration and resource pre-allocation
  async onInit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onInit");
  }

  // Extension startup phase - officially start processing business logic
  async onStart(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStart");
  }

  // Command handler - handle command requests from other extensions or applications
  async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
    console.log("DefaultExtension onCmd", cmd.getName());

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("detail", "This is a demo");
    tenEnv.returnResult(cmdResult);
  }

  // Extension stop phase - clean up resources and graceful shutdown
  async onStop(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStop");
  }

  // Extension destruction phase - final cleanup and resource release
  async onDeinit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onDeinit");
  }
}
```

#### Plugin Registration Entry

Extension plugin registration and creation logic:

```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
  async onCreateInstance(
    _tenEnv: TenEnv,
    instanceName: string,
  ): Promise<Extension> {
    return new DefaultExtension(instanceName);
  }
}
```

#### Test Framework Implementation (tests/src/index.ts)

Use TEN-specific test framework to write complete unit tests:

```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";

export class MyExtensionTester extends ExtensionTester {
  async onStart(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStart");

    // Simulate async operations and test logic
    new Promise((resolve) => {
      setTimeout(() => {
        resolve(true);
      }, 1000);
    }).then(() => {
      // Stop test and return results
      tenEnvTester.stopTest();
    });
  }

  async onStop(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStop");
  }

  async onDeinit(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onDeinit");
  }
}
```

#### Test Case Definition (tests/src/index.spec.ts)

Use Mocha test framework to write specific test cases:

```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";

const test_addon_name = "default_extension_nodejs";

describe("MyExtensionTester", () => {
  it("case1", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });

  it("case2", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });
});
```

### TypeScript Configuration

Node.js extensions use modern TypeScript configuration, supporting the latest language features:

```json
{
  "compilerOptions": {
    "allowJs": false,
    "composite": true,
    "module": "NodeNext",
    "target": "es6",
    "moduleResolution": "NodeNext",
    "outDir": "build",
    "removeComments": false,
    "sourceMap": true,
    "noImplicitAny": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "strictNullChecks": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "rootDir": "src",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
  },
  "exclude": [
    "node_modules"
  ],
  "include": [
    "src/**/*"
  ]
}
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure you have installed the official Node.js extension, then use the following debug configuration:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (nodejs) (mocha, launch)",
      "type": "node",
      "request": "launch",
      "program": "node_modules/mocha/bin/_mocha",
      "stopOnEntry": true,
      "args": [
        "--no-timeouts",
        "--package",
        "package.json",
      ],
      "cwd": "${workspaceFolder}/tests",
      "env": {
        "NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
      },
      "runtimeArgs": [
        "--expose-gc",
        "--loader",
        "ts-node/esm",
        "--no-warnings",
      ]
    }
  ]
}
```

#### Command Line Debugging

Use Node.js built-in debugger for command-line debugging:

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Start debug mode
node --inspect-brk ./tests/bin/start

# Or use Chrome DevTools
node --inspect ./tests/bin/start
```

### Complete Development Process Summary

To help developers get started quickly, here is a complete Node.js extension development process summary:

```bash
# 1. Create extension project
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example

# 2. Enter project directory
cd my_example_ext_nodejs

# 3. Install dependencies
tman install --standalone

# 4. Enter extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 5. Install standalone mode dependencies
npm run standalone-install

# 6. Build project
npm run build

# 7. Run tests
./tests/bin/start
```

> **Working Directory Note**:
>
> * Extension source code is located in the `src/` folder at the project root directory
> * Actual build, test, and run operations are performed in the `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory
> * This design ensures extension independence and proper dependency management

### Development Best Practices

#### Async Programming

Node.js extensions fully utilize async programming patterns for better performance:

```typescript
// Recommended: Use async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    const result = await processCommand(cmd);
    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("result", result);
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

#### Error Handling

Implement comprehensive error handling mechanisms:

```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    // Business logic processing
    const result = await this.handleBusinessLogic(cmd);

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("data", JSON.stringify(result));
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    console.error("Command processing error:", error);

    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

***

## Development Summary

By following the complete development process provided in this guide, you can efficiently develop, test, and debug TEN extensions. Whether you choose C++, Go, Python, or Node.js, TEN Framework provides you with a complete toolchain and best practices to help you fully leverage the powerful features of TEN Framework and build high-performance, high-reliability extension applications.

Each language has its unique advantages and applicable scenarios:

* **C++**: Suitable for scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency computation
* **Go**: Provides a balance between high performance and development efficiency, suitable for network services, concurrent processing
* **Python**: Has the highest development efficiency, particularly suitable for AI/ML applications, rapid prototyping
* **Node.js**: Provides modern Web development experience, suitable for frontend technology stack extensions, real-time applications

Please choose the most suitable development solution based on your specific needs and team technology stack. During the development process, it is recommended to fully utilize the debugging tools and testing frameworks provided by TEN Framework to ensure the quality and stability of your extensions.


# TEN Framework Quick Start Guide (/docs/ten_framework/0.11.59/getting-started/quick-start)

## TEN Framework Quick Start Guide

> 🎯 **Goal**: Set up your development environment and run your first TEN app in 5 minutes

## System Requirements

**Supported Operating Systems**:

* Linux (x64)
* Linux (arm64)
* macOS Intel (x64)
* macOS Apple Silicon (arm64)
* Windows (x64)

**Required Software**:

* Python 3.10
* Go 1.20+
* Node.js / npm (for managing JavaScript dependencies)

> 💡 **Windows users**: It's recommended to use PowerShell to run the commands in this guide. Some commands may not be compatible with CMD.

## Step 1: Check Your Environment

Before you begin, make sure the following software is installed on your system:

### Python 3.10

```bash
python3 --version
# Should display: Python 3.10.x
```

> 💡 **Important**: TEN Framework currently only supports Python 3.10. It's recommended to use `pyenv` or `venv` to manage your Python environment:
>
> **Linux / macOS:**
>
> ```bash
> # Install and manage Python 3.10 using pyenv (recommended)
> pyenv install 3.10.14
> pyenv local 3.10.14
>
> # Or create virtual environment using venv
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows:**
>
> Download the Windows installer from the bottom table at [https://www.python.org/downloads/release/python-31011/](https://www.python.org/downloads/release/python-31011/) and run it to install Python 3.10.
>
> Make sure to check "Add Python to PATH" before clicking Install Now.
>
> ```bash
> # On Windows, configure the python3 command:
>
> # First, find where python is installed:
> where.exe python
> # Example output: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # Then, open PowerShell as Administrator and create a symlink in the same directory:
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # Replace the paths above with the actual output from where.exe python
>
> # Verify:
> python3 --version
> ```
>
> ```powershell
> # It's recommended to create a venv after installation
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # Activate the environment
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # If you encounter a permission error, close the terminal/IDE, right-click and select "Run as Administrator" to reopen
> # Or change the execution policy to allow ps1 scripts
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```

### Go 1.20+

```bash
go version
# Should display: go version go1.20 or higher

$env:CGO_ENABLED = "1"
# CGO must be explicitly enabled on Windows
```

### Node.js / npm

```bash
node --version
npm --version
# Ensure node and npm commands are available
```

### GCC (MinGW)

💡 Only required on Windows

```bash
# First, make sure winget is available.
winget --version

# If not, install it from <https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US>
# (Requires Windows 10 version 1709 (Build 16299) or later, or Windows 11)

# Install MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT

# Or search and choose a suitable version to install
winget search "mingw"

# Verify installation
gcc --version
```

> 💡 **Tip**: If any of the above is missing, please install the required version before continuing.

## Step 2: Install TEN Manager (tman)

TEN Manager (tman) is the command-line tool for TEN Framework, used to create projects, manage dependencies, and run applications.

Option 1: Install via Package Manager (Recommended)

**Linux (Ubuntu/Debian):**

```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```

**macOS:**

```bash
brew install TEN-framework/ten-framework/tman
```

**Windows:**

```powershell
winget install TEN-framework.tman
```

Option 2: Install via Script

**Linux / macOS:**

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```

**Windows:**

```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```

Option 3: Install from cloned repository

```bash
cd ten-framework

# Linux/macOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```

> 💡 **Note**: If tman is already installed on your system, the installation script will ask whether you want to reinstall/upgrade it. Press `y` to continue or `n` to cancel.
>
> **Non-interactive Installation** (for automation scripts or CI environments, not available on Windows):
>
> ```bash
> # Remote installation
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # Local installation
> yes y | bash tools/tman/install_tman.sh
> ```

**Verify Installation**:

```bash
tman --version
```

> 💡 **Tip**: If you see `tman: command not found` (Linux/macOS) or a similar unrecognized command error (Windows), make sure the tman directory is in your PATH:
>
> **Linux / macOS:**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc  # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc   # macOS
> source ~/.bashrc  # or source ~/.zshrc
> ```
>
> **Windows:**
>
> ```powershell
> # Check if tman is in PATH
> $env:Path -split ";" | Select-String "tman"
> # If not, either:
> # 1. Add it manually via "System Properties → Environment Variables", then restart your terminal
> # 2. Or temporarily add the path with:
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```

## Step 3: Create and Run the Demo App

<div className="fd-steps">
  <div className="fd-step">
    ### Create App

    ```bash
    # Create a new transcriber_demo app
    tman install app transcriber_demo
    cd transcriber_demo
    ```
  </div>

  <div className="fd-step">
    ### Install Dependencies

    ```bash
    # Install TEN package dependencies
    tman install

    # Install Python and npm package dependencies
    tman run install_deps
    ```

    > ⏱️ **Estimated Time**: 1-2 minutes
  </div>

  <div className="fd-step">
    ### Build the App

    ```bash
    tman run build
    ```

    > ⏱️ **Estimated Time**: 30 seconds
  </div>

  <div className="fd-step">
    ### Configure Environment Variables

    Before running the app, you need to configure the ASR (Automatic Speech Recognition) service credentials. The current example uses Azure ASR extension. You need to fill in the configuration in the `transcriber_demo/.env` file:

    **Linux / macOS:**

    ```bash
    # Create .env file
    cat > .env << EOF
    # Azure Speech Service Configuration
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region      # e.g., eastus
    AZURE_STT_LANGUAGE=en-US                # Set according to your audio language or real-time recording language, e.g., zh-CN, ja-JP, ko-KR, etc.
    EOF
    ```

    **Windows (PowerShell):**

    ```powershell
    # Create .env file
    @"
    # Azure Speech Service Configuration
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region
    AZURE_STT_LANGUAGE=en-US
    "@ | Out-File -Encoding utf8 .env
    ```

    > 💡 **Tip**: Windows users can also create the `.env` file directly with a text editor and fill in the configuration above.
    >
    > 💡 **Tip**: If you want to use other ASR extensions (such as OpenAI Whisper, Google Speech, etc.), you can download and replace them from the cloud store. Similarly, configure the corresponding API keys and environment variables in the `.env` file.
  </div>

  <div className="fd-step">
    ### Run the App

    ```bash
    tman run start
    ```

    If everything is working correctly, you should see output similar to:

    ```text
    [web_audio_control_go] Web server started on port 8001
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```
  </div>

  <div className="fd-step">
    ### Experience the Demo

    Open your browser and visit:

    ```text
    http://localhost:8001
    ```

    You should see the Transcriber Demo web interface. Try:

    * Click the microphone button for real-time voice transcription
    * Upload an audio file for transcription
    * View real-time transcription and subtitle results
  </div>
</div>

## Congratulations! 🎉

You've successfully run your first TEN application!

### Understanding the App Architecture

This `transcriber_demo` app showcases TEN Framework's multi-language extension capabilities, consisting of:

* **Go** - WebSocket server extension (`web_audio_control_go`)
* **Python** - ASR speech recognition extension (`azure_asr_python`)
* **TypeScript** - VTT subtitle generation and audio recording extension (`vtt_nodejs`)

🎯 **You can now run extensions in multiple languages!**

### Next Steps

Now you can:

1. **Explore and download more extensions from the cloud store, design and orchestrate your app**

   ```bash
   tman designer  # Launch TMAN Designer to explore extensions, download them, and design your app
   ```

2. **Choose a language and develop your own extension**
   * Supports Go, Python, TypeScript/JavaScript, C++, and more
   * Check out the [TEN Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) for details

## Advanced: Developing and Building C++ Extensions

If you want to develop and use C++ extensions, you'll need to install the TEN build toolchain (tgn). Here's the complete process:

<div className="fd-steps">
  <div className="fd-step">
    ### Install tgn Build Tool

    tgn is TEN Framework's C/C++ build system, based on Google's GN.

    Option 1: One-line Installation (Recommended)

    **Linux / macOS:**

    ```bash
    curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
    ```

    **Windows:**

    ```powershell
    irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
    ```

    Option 2: Install from Cloned Repository

    **Linux / macOS:**

    ```bash
    # If you've already cloned the TEN Framework repository
    cd ten-framework
    bash tools/tgn/install_tgn.sh
    ```

    **Windows:**

    ```powershell
    # If you've already cloned the TEN Framework repository
    & ".\tools\tgn\install_tgn.ps1"
    ```

    After installation, ensure tgn is added to PATH:

    **Linux / macOS:**

    ```bash
    # Temporarily add to current session
    export PATH="/usr/local/ten_gn:$PATH"

    # Or permanently add to shell configuration (recommended)
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc  # Linux
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc   # macOS
    source ~/.bashrc  # or source ~/.zshrc
    ```

    **Windows:**

    ```powershell
    # The install script adds to user PATH automatically. To add manually:
    $env:Path += ";$env:LOCALAPPDATA\ten_gn"
    ```

    Verify installation:

    ```bash
    tgn --help
    ```
  </div>

  <div className="fd-step">
    ### Install C++ Extension

    Using WebRTC VAD (Voice Activity Detection) extension as an example, install a C++ extension from the cloud store:

    ```bash
    cd transcriber_demo
    tman install extension webrtc_vad_cpp
    ```

    > 💡 **Note**: `webrtc_vad_cpp` is a voice activity detection extension implemented in C++. It can filter out non-speech segments in real-time speech recognition scenarios.
  </div>

  <div className="fd-step">
    ### Compile C++ Extension

    After installing the C++ extension, rebuild the app to compile the C++ code into a dynamic library:

    ```bash
    tman run build
    ```

    > ⏱️ **Expected Time**: First-time compilation of C++ extensions may take 1-3 minutes, depending on your machine's performance.
  </div>

  <div className="fd-step">
    ### Run App with VAD Functionality

    ```bash
    tman run start_with_vad
    ```

    If everything works correctly, you should see:

    ```text
    [web_audio_control_go] Web server started on port 8001
    [vad] WebRTC VAD initialized with mode 2
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```

    Now open your browser at `http://localhost:8001` and navigate to the microphone real-time transcription page. You'll see the silence state changes after VAD processing. When the silence state is true, it indicates there is no speech in the current audio.
  </div>
</div>

### C++ Development Environment Requirements

Developing and compiling C++ extensions requires installing a C++ compiler (gcc or clang):

**Linux:**

```bash
# Ubuntu/Debian
sudo apt-get install gcc g++

# Or use clang
sudo apt-get install clang
```

**macOS:**

```bash
# Install Xcode Command Line Tools (includes clang)
xcode-select --install
```

**Windows:**

```powershell
# Option 1: Install Visual Studio Build Tools (recommended)
# Download from https://visualstudio.microsoft.com/visual-cpp-build-tools/

# Option 2: Install via winget
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

# 💡 Note:
#  - Select the "Desktop development with C++" workload during installation
#  - Make sure to check "C++ Clang tools for Windows"
```

Verify compiler installation:

```bash
# Check gcc (Linux/macOS/MSYS2)
gcc --version
g++ --version

# Or check clang (Linux/macOS)
clang --version
```

**Windows (MSVC):**

```powershell
# Run in "Developer PowerShell for VS"
clang-cl --version
```

### Troubleshooting (C++ Extensions)

1. tgn command not found

   Ensure you've run the installation script and added tgn to PATH:

   ```bash
   export PATH="/usr/local/ten_gn:$PATH"
   ```

2. Compilation failed: Compiler not found

   Please refer to the "C++ Development Environment Requirements" section above to install the compiler.

### Learn More

* [ten\_gn Build System](https://github.com/TEN-framework/ten_gn) - TEN's C/C++ build tool
* [C++ Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) - Complete C++ extension development documentation

## Troubleshooting

<div className="fd-steps">
  <div className="fd-step">
    ### Python Library Loading Failure on macOS

    **Problem**: Error indicating `libpython3.10.dylib` cannot be found when running the app

    **Solution**:

    ```bash
    export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
    ```

    Consider adding this line to your `~/.zshrc` or `~/.bash_profile`.
  </div>

  <div className="fd-step">
    ### tman Download Failed or Slow

    **Problem**: Network connection to GitHub is restricted

    **Solution**:

    * Manual download: Visit the [Releases page](https://github.com/TEN-framework/ten-framework/releases) to download the `tman` binary for your platform
  </div>

  <div className="fd-step">
    ### Port 8001 Already in Use

    **Problem**: Port conflict error when starting the app

    **Solution**:

    * Find the process using the port:
      * Linux/macOS: `lsof -i :8001`
      * Windows: `netstat -ano | findstr :8001`
    * Kill the process:
      * Linux/macOS: `kill -9 <PID>`
      * Windows: `taskkill /PID <PID> /F`
    * Or modify the port number in the app configuration file (`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`)
  </div>

  <div className="fd-step">
    ### Go Build Failed

    **Problem**: Go module related errors during build

    **Solution**:

    ```
    # Set proxy
    # Linux/macOS
    export GOPROXY=https://goproxy.cn,direct
    # Windows PowerShell
    $env:GOPROXY = "https://goproxy.cn,direct"

    # Clean Go module cache
    go clean -modcache

    # Reinstall dependencies
    cd transcriber_demo
    tman run build
    ```
  </div>

  <div className="fd-step">
    ### Python Dependencies Installation Failed

    **Problem**: pip installation timeout or failure

    **Solution**: Use a mirror source (for users in China)

    ```bash
    pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
    ```
  </div>

  <div className="fd-step">
    ### Azure Speech Service Error

    **Problem**: Azure Speech Service related errors, such as authentication failure

    **Solution**: Check that the configuration in your `.env` file is correct, and ensure `AZURE_STT_KEY` and `AZURE_STT_REGION` are filled in accurately
  </div>

  <div className="fd-step">
    ### Permission Error on Windows

    **Problem**: PermissionError when accessing files on Windows

    **Solution**: Right-click PowerShell and select "Run as Administrator"
  </div>
</div>

## Get Help

* **GitHub Issues**: [https://github.com/TEN-framework/ten-framework/issues](https://github.com/TEN-framework/ten-framework/issues)
* **Documentation**: [https://theten.ai/docs](https://theten.ai/docs)
* **Contributing Guide**: [contributing.md](../code-of-conduct/contributing.md)


# WebSocket Voice Assistant Quickstart (/docs/ten_framework/0.11.59/getting-started/websocket-voice-assistant-quick-start)

## WebSocket Voice Assistant Quickstart

Build and run a real-time WebSocket voice assistant in under 10 minutes.

## Overview

This guide helps you run a complete voice assistant demo locally. It supports real-time voice conversations and tool calling. The demo includes:

| Component | Vendor     | Purpose                   |
| --------- | ---------- | ------------------------- |
| ASR       | Deepgram   | Speech-to-text            |
| LLM       | OpenAI     | Response generation       |
| TTS       | ElevenLabs | Text-to-speech            |
| Tool      | WeatherAPI | Weather lookup (optional) |

## Requirements

### Prerequisites

Follow the [TEN Framework Quick Start](./quick-start.md) to install the base environment, then make sure the check below passes:

```bash
tman check env
```

Expected output (at least Python and Go are Ready):

```text
✅ Operating System: Supported
✅ Python:   Ready
✅ Go:       Ready
```

### Extra tools

Install the following tools for build & run:

#### Task (task runner)

**Linux / macOS:**

```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version  # verify
```

**Windows:**

```powershell
# Option 1: Install via winget (recommended)
# Install winget from: https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# Requires Windows 10 version 1709 (Build 16299) or higher, or Windows 11
winget --version  # verify winget
winget install Task.Task
task --version  # verify

# Option 2: Install via scoop
irm get.scoop.sh | iex
scoop --version  # verify scoop
scoop install task
task --version  # verify
```

#### Bun (JavaScript package manager)

**Linux / macOS:**

```bash
curl -fsSL https://bun.sh/install | bash
bun --version  # verify
```

**Windows:**

```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version  # verify
```

#### uv (Python package manager)

**Linux / macOS:**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version  # verify
```

**Windows:**

```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version  # verify
```

> 💡 After installation, reload your shell config:
>
> * Linux/macOS: `source ~/.zshrc`
> * Windows: reopen your PowerShell terminal

## Quickstart

<div className="fd-steps">
  <div className="fd-step">
    ### Clone the repo

    ```bash
    git clone https://github.com/TEN-framework/ten-framework.git
    cd ten-framework/ai_agents/agents/examples/websocket-example
    ```
  </div>

  <div className="fd-step">
    ### Install dependencies

    ```bash
    task install
    ```

    > ⏱️ Estimated time: 2–3 minutes

    This command will:

    * Install TEN extension packages (45+)
    * Build the Go app and API server
    * Install Python and frontend dependencies
  </div>

  <div className="fd-step">
    ### Configure API keys

    Configure vendor keys in `ai_agents/.env`:

    **Linux / macOS:**

    ```bash
    cd ../../../  # back to ai_agents
    vim .env
    ```

    **Windows:**

    ```powershell
    cd ..\..\..   # back to ai_agents
    notepad .env
    ```

    Copy all content from .env.example, then fill in:

    ```bash
    # Deepgram - ASR
    DEEPGRAM_API_KEY=your_deepgram_api_key

    # OpenAI - LLM
    OPENAI_API_KEY=your_openai_api_key
    OPENAI_MODEL=your_openai_model

    # ElevenLabs - TTS
    ELEVENLABS_TTS_KEY=your_elevenlabs_api_key

    # WeatherAPI - Tool (optional)
    WEATHERAPI_API_KEY=your_weatherapi_key
    ```
  </div>

  <div className="fd-step">
    ### Start services

    ```bash
    cd agents/examples/websocket-example
    task run
    ```

    After startup, the following services are available:

    * **Frontend UI**: [http://localhost:3000](http://localhost:3000)
    * **API server**: [http://localhost:8080](http://localhost:8080)
    * **TMAN Designer**: [http://localhost:49483](http://localhost:49483)
  </div>

  <div className="fd-step">
    ### Talk to your assistant

    Open [http://localhost:3000](http://localhost:3000), click the microphone button, and start speaking.

    Try:

    * "Hello, who are you?"
    * "What's the weather in Beijing today?" (requires WeatherAPI)
    * "Tell me a joke."

    **Congratulations!** Your TEN Agent voice assistant is running.
  </div>
</div>

## Advanced

### Swap extensions

TEN Framework lets you swap ASR/LLM/TTS extensions flexibly.

#### Option 1: Use TMAN Designer (recommended)

Open [http://localhost:49483](http://localhost:49483), then in the visual editor:

1. Click an extension node to inspect it
2. Choose a replacement extension
3. Configure parameters
4. Save and restart

#### Option 2: Edit configuration file

Edit `tenapp/property.json` and modify the extension configs under `predefined_graphs`.

##### Example: switch ASR to Azure

```json
{
  "type": "extension",
  "name": "stt",
  "addon": "azure_asr_python",
  "extension_group": "stt",
  "property": {
    "params": {
      "key": "${env:AZURE_STT_KEY}",
      "region": "${env:AZURE_STT_REGION}",
      "language": "en-US"
    }
  }
}
```

##### Example: switch TTS to Minimax

```json
{
  "type": "extension",
  "name": "tts",
  "addon": "minimax_tts_websocket_python",
  "extension_group": "tts",
  "property": {
    "params": {
      "api_key": "${env:MINIMAX_TTS_API_KEY|}",
      "group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
      "model": "speech-02-turbo",
      "audio_setting": {
        "sample_rate": 16000
      },
      "voice_setting": {
        "voice_id": "female-shaonv"
      }
    }
  }
}
```

Restart after changes (`Ctrl+C` then `task run`).

### Monitoring (Grafana)

Use Grafana to observe application metrics and logs.

<div className="fd-steps">
  <div className="fd-step">
    #### Start monitoring stack

    **Linux / macOS:**

    ```bash
    cd tools/grafana-monitoring
    docker-compose -f docker-compose.push.yml up -d
    ```

    **Windows:**

    ```powershell
    cd tools\grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    # If Docker is not installed, install it from https://apps.microsoft.com/detail/xp8cbj40xlbwkx, restart your PC, then launch Docker Desktop
    ```

    Services:

    * **Grafana** (port 3001) - UI
    * **Prometheus** (port 9091) - metrics store
    * **Loki** (port 3100) - log store
    * **OpenTelemetry Collector** (ports 4317/4318) - OTLP receiver
  </div>

  <div className="fd-step">
    #### Configure app telemetry

    Edit `agents/examples/websocket-example/tenapp/property.json`:
  </div>
</div>

#### Logs (export)

Add to the `ten.log.handlers` array:

```json
{
  "matchers": [{ "level": "info" }],
  "formatter": { "type": "json", "colored": false },
  "emitter": {
    "type": "otlp",
    "config": {
      "endpoint": "http://localhost:4317",
      "service_name": "websocket-example"
    }
  }
}
```

#### Metrics (export)

Add to `ten.services`:

```json
{
  "telemetry": {
    "enabled": true,
    "metrics": {
      "enabled": true,
      "exporter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
  }
}
```

<div className="fd-steps">
  <div className="fd-step">
    #### Restart and open Grafana

    ```bash
    cd ../../../agents/examples/websocket-example
    task run
    ```

    Open [http://localhost:3001](http://localhost:3001) (username/password: `admin/admin`).
  </div>
</div>

#### Metrics (Dashboards)

Go to `Dashboards` → **"TEN Framework"** to view:

* Extension startup latency
* Cmd processing latency
* Message queue wait time

#### Logs (Explore/Loki)

Go to `Explore` → select `Loki`, then query:

* `{service_name="websocket-example"}` - all logs
* `{service_name="websocket-example", ten_extension_name="stt"}` - ASR logs
* `{service_name="websocket-example", ten_extension_name="llm"}` - LLM logs
* `{service_name="websocket-example", ten_extension_name="tts"}` - TTS logs

> 💡 Use the `ten_extension_name` label to filter logs per extension.

Stop the monitoring stack:

**Linux / macOS:**

```bash
cd tools/grafana-monitoring
docker-compose -f docker-compose.push.yml down
```

**Windows:**

```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```

## Troubleshooting

<details>
<summary><strong>Frontend dependency install fails</strong></summary>

If `bun install` fails due to missing versions, switch to the official npm registry:

```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```

</details>

<details>
<summary><strong>macOS Python library loading fails</strong></summary>

Set the Python dynamic library path:

```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# Add to ~/.zshrc for persistence
```

</details>

<details>
<summary><strong>Windows Python library loading fails</strong></summary>

Make sure the Python 3.10 installation path is in your system PATH:

```powershell
# Check Python path
python -c "import sys; print(sys.prefix)"

# Add Python DLL directory to PATH (adjust to your actual path)
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```

If you didn't check "Add Python to PATH" during installation, re-run the installer and choose "Modify" to add it.

</details>

<details>
<summary><strong>Port already in use</strong></summary>

Find and kill the process:

**Linux / macOS:**

```bash
lsof -i :3000  # or :8080
kill -9 <PID>
```

**Windows:**

```powershell
netstat -ano | findstr :3000   # or :8080
taskkill /PID <PID> /F
```

</details>

<details>
<summary><strong>Go module related errors</strong></summary>

```bash
# Set proxy
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"

# Clean Go module cache
go clean -modcache

# Reinstall dependencies
cd ai_agents/agents/examples/websocket-example
task install
```

</details>

## Next steps

* **[Extension development guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext)** - build custom extensions
* **[More examples](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - explore other agent demos
* **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - report bugs or request features
* **[Documentation hub](https://theten.ai/docs)** - full TEN Framework documentation


# Graph (/docs/ten_framework/0.11.59/graph/graph)

In the TEN framework, there are two types of graphs:

1. Dynamic Graph
2. Predefined Graph (`predefined_graph`)

|               | Dynamic Graph                                       | Predefined Graph                                                       |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------- |
| Startup Time  | When the TEN app receives the `start_graph` command | When the TEN app starts, or when it receives the `start_graph` command |
| Graph Content | Specified in the `start_graph` command              | Predefined in the TEN app's properties                                 |
| Graph ID      | Randomly generated UUID                             | Randomly generated UUID                                                |

![Two types of graphs](https://ten-framework-assets.s3.amazonaws.com/doc-assets/two_types_of_graph.png)

Predefined graphs have an `auto_start` property to determine whether the graph automatically starts when the TEN app launches.

Additionally, predefined graphs have a `singleton` property that indicates whether only *one* instance of the graph can be generated within the TEN app.

## Graph ID and Graph Name

For each graph instance, the TEN app internally uses a unique UUID4 string as an identifier, called the **Graph ID**.

For predefined graphs, you can assign a meaningful and easy-to-remember name, called the **Graph Name**. When you need to specify a particular predefined graph, you can directly use its graph name. If a predefined graph has the `singleton` property, it means that only one instance of this predefined graph can exist within the TEN app. In this case, the TEN runtime uses the graph name to uniquely identify the single instance generated from that predefined graph.

## Dynamic Graph

When a TEN app receives a `start_graph` command and creates a dynamic graph, the system assigns a random UUID as the graph's ID. If other clients obtain this graph ID, they can also connect to this graph.

Dynamic Graph ID example:

`123e4567-e89b-12d3-a456-426614174000`

## Predefined Graph

Predefined graphs work similarly to dynamic graphs, with the main difference being how the content is defined. The content of a dynamic graph is included in the `start_graph` command, while the content of a predefined graph is defined in advance by the TEN app. Clients only need to specify the name of the predefined graph in the `start_graph` command to start it.

The main advantage of predefined graphs is simplifying usage and protecting sensitive information. With predefined graphs, clients don't need to know the detailed structure of the graph, which both improves usability and avoids exposing potentially sensitive information contained in the graph.

Predefined Graph name example:

`http_server`

When a TEN app starts, all predefined graphs with the `auto_start` property set will automatically start.

## Graph Definition

Whether it's a dynamic graph or a predefined graph, the definition structure is the same:

```json
{
  "nodes": [
    // Definition of nodes
  ],
  "connections": [
    // Definition of communication links
  ]
}
```

Key points:

1. If there is only one TEN app, you can omit the `app` field. If there are multiple apps, you must explicitly specify the `app` field.

2. The `nodes` field defines the nodes in the graph, such as various extensions.

3. Each node can appear only once in the `nodes` field. If the same node appears multiple times, the TEN framework will report an error during validation.

4. The way to define an extension in the `nodes` field is as follows, where the `property` field is optional:

   ```json
   {
     "type": "extension",
     "name": "simple_http_server_cpp",
     "addon": "simple_http_server_cpp",
     "extension_group": "default_extension_group",
     "app": "msgpack://127.0.0.1:8001/",
     "property": {
       "root_key": "player",
       "extra_keys": ["playerName"]
     }
   }
   ```

   The `addon` field indicates that the extension is an instance generated by the corresponding plugin.

5. The `connections` field defines the communication links between nodes in the graph, where the `extension` value represents the name of the corresponding node.

Complete example:

```json
{
  "nodes": [
    {
      "type": "extension",
      "app": "msgpack://127.0.0.1:8001/",
      "name": "simple_http_server_cpp",
      "addon": "simple_http_server_cpp",
      "extension_group": "default_extension_group",
      "property": {
        "root_key": "player",
        "extra_keys": ["playerName"]
      }
    }
  ],
  "connections": [
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "simple_http_server_cpp",
      "cmd": [
        {
          "name": "start",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        },
        {
          "name": "stop",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        }
      ]
    },
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "gateway",
      "cmd": [
        {
          "name": "push_status_online",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "uap"
            }
          ]
        }
      ]
    }
  ]
}
```

## Predefined Graph Definition

Predefined graphs are defined under the `predefined_graphs` field in the TEN app's configuration file (`property.json`), including properties such as `name`, `auto_start`, and `singleton`:

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        // Complete graph definition
      }
    ]
  }
}
```

Complete example:

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        "nodes": [
          {
            "type": "extension",
            "name": "simple_http_server_cpp",
            "addon": "simple_http_server_cpp",
            "extension_group": "default_extension_group",
            "property": {
              "root_key": "player",
              "extra_keys": [
                "playerName"
              ]
            }
          }
        ],
        "connections": [
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "simple_http_server_cpp",
            "cmd": [
              {
                "name": "start",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              },
              {
                "name": "stop",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              }
            ]
          },
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "gateway",
            "cmd": [
              {
                "name": "push_status_online",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "uap"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## `start_graph` Command Definition

Dynamic graphs are created by sending a `start_graph` command to the TEN app, placing the graph definition in the command's `ten` field:

```json
{
  "ten": {
    "type": "start_graph",
    // Complete graph definition
  }
}
```

Complete example:

```json
{
  "ten": {
    "type": "start_graph",
    "nodes": [
      {
        "type": "extension",
        "name": "simple_http_server_cpp",
        "addon": "simple_http_server_cpp",
        "extension_group": "default_extension_group",
        "property": {
          "root_key": "player",
          "extra_keys": ["playerName"]
        }
      }
    ],
    "connections": [
      {
        "app": "msgpack://127.0.0.1:8001/",
        "extension": "simple_http_server_cpp",
        "cmd": [
          {
            "name": "start",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          },
          {
            "name": "stop",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          }
        ]
      },
      {
        "extension": "gateway",
        "cmd": [
          {
            "name": "push_status_online",
            "dest": [
              {
                "extension": "uap"
              }
            ]
          }
        ]
      }
    ]
  }
}
```

### Using `import_uri` to Reference Graph Definitions

When defining predefined graphs, you can use the `import_uri` field to reference external graph definition files:

```json
{
  "ten": {
    "uri": "http://localhost:8001",
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": false,
        "import_uri": "../graph.json"
      }
    ]
  }
}
```

`import_uri` can be a relative path, absolute path, or URL:

* Relative path: relative to the directory containing the `property.json` file
* Absolute path: relative to the root directory of the environment where the TEN app is located
* URL: treated directly as a URL path

## Graph Definition Specifications

* **`nodes` Field**: The `nodes` array must be provided in the graph definition. In contrast, while the `connections` array is optional, it is recommended to provide it to define communication between nodes.

* **Node `app` Field**: The `app` field cannot be set to `localhost`. In single-app graphs, the `app` URI should be omitted; in multi-app graphs, the `app` field value must match the `ten.uri` value in each app's `property.json`.

* **Node Uniqueness**: Each node in the `nodes` array represents a specific extension instance and must be uniquely identified by the combination of `app` and `name`. Duplicate definitions are not allowed, as in the following invalid example:

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_1",
        "extension_group": "test"
      },
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_2",
        "extension_group": "test"
      }
    ]
  }
  ```

* **Connection Definition Consistency**: All extension instances referenced in `connections` must be explicitly defined in `nodes`. The following example is invalid because `ext_2` is not defined in `nodes`:

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "ext_1",
        "addon": "addon_1",
        "extension_group": "some_group"
      }
    ],
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **Communication Link Consolidation**: All messages from the same source extension should be grouped in a single section, not scattered across multiple definitions. The following example is incorrect:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      },
      {
        "extension": "ext_1",
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  The correct approach is to consolidate all messages from the same source extension together:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ],
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **Message Destination Consolidation**: For each specific type of message, all destination extensions should be grouped under a single entry. The following example is incorrect:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          },
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  The correct approach is to consolidate all destinations for the same message together:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              },
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  Note that messages with the same name can coexist in different types (such as `cmd` and `data`) without causing conflicts.

## Subgraph

The core mechanism of the TEN framework is based on graph structures, which consist of nodes and connections. Subgraphs are a powerful reuse mechanism that allows complex graph structures to be split into reusable modules, thereby improving code organization and maintainability.

### Subgraph Design Philosophy

The essence of a graph is to define how data flows between extensions. Subgraphs do not change this fundamental principle in their implementation, but serve as syntactic sugar that is ultimately flattened into the parent graph and started using the same mechanisms as regular graphs. This design simplifies the development of complex systems while not adding runtime complexity.

#### Design Principles

Subgraph design follows these core principles:

1. **Independence**: Any graph is a complete graph that can run independently or be embedded as a component into other graphs.

2. **Tool-Friendly**: Subgraphs provide additional information to help development tools better understand graph structures, improving the development experience while not adding runtime complexity.

3. **Flattening Mechanism**: Subgraphs are ultimately flattened into standard graph structures to ensure performance and compatibility.

4. **Simplicity**: Subgraph design aims to simplify the development process rather than add complexity, avoiding increased JSON structure complexity.

The purpose of subgraph design is to allow developers to use a graph as a black box without needing to understand its internal complexity. Therefore, at the point of referencing subgraphs, no special patching mechanisms are provided to adjust the internal state of subgraphs, to reduce complexity. If subgraph definitions need to be modified, the original definitions should be modified directly rather than patching at the reference point. This is done to avoid complexity spread and increased maintenance costs. For example, when introducing subgraph A into large graph B, if you need to adjust the extension property schema definitions inside subgraph A, you should directly modify the schema definitions of that extension rather than providing modification mechanisms at the reference point, because this might lead to adverse chain reactions, only modifying the extension's schema definitions without patching other places, thus causing errors elsewhere.

In other words, subgraphs primarily provide a mapping mechanism to expose internal elements of subgraphs to the outside, rather than providing mechanisms to patch internal elements of subgraphs.

### Subgraph Implementation Mechanism

Next, we will detail the implementation mechanism of subgraphs, including subgraph definition, reference methods, and the flattening process.

#### Subgraph Definition Example

The following is a `subgraph.json` example that can be used as an independent graph or as a subgraph referenced by other graphs:

```json
{
  "nodes": [
    {
      // Define extension node named ext_c
      "type": "extension",
      "name": "ext_c",
      "addon": "extension_c"
    },
    {
      // Define extension node named ext_d
      "type": "extension",
      "name": "ext_d",
      "addon": "extension_d"
    }
  ],
  "connections": [
    {
      // ext_c sends B command to ext_d
      "extension": "ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_d"
            }
          ]
        }
      ]
    }
  ],
  "exposed_messages": [
    // Represents message interfaces exposed by the graph to the outside, mainly for development tools to provide intelligent hints
    {
      // ext_d's B command exposed to the outside
      "extension": "ext_d",
      "type": "cmd_in",
      "name": "B"
    }
  ],
  "exposed_properties": [
    // Represents properties exposed by the graph to the outside
    {
      "extension": "ext_c",
      "name": "a",
      "alias": "a1"
    }
  ]
}
```

#### Subgraph Reference Example

Example of `graph.json` referencing subgraphs:

```json
{
  "nodes": [
    {
      // Define extension node named ext_a
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // Define extension node named ext_b
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // Reference subgraph, named subgraph_1
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "./ten_packages/extension/aaa/subgraph.json",
      "property": {
        // Properties written in this property field must exist in the subgraph's exposed_properties field,
        // otherwise it will be an error condition
        "app_id": "${env:AGORA_APP_ID}",
        "token": "<agora_token>",
        "channel": "ten_agent_test"
      }
    },
    {
      // Reference subgraph, named subgraph_2
      "type": "subgraph",
      "name": "subgraph_2",
      "import_uri": "./ten_packages/extension/bbb/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // First target is ext_b
              "extension": "ext_b"
            },
            {
              // Second target is ext_d in the subgraph, this is advanced usage
              "extension": "subgraph_1:ext_d"
            },
            {
              // Third target is the subgraph, but this requires the subgraph to explicitly expose the B cmd_in interface,
              // otherwise it will be treated as an error condition
              "subgraph": "subgraph_2"
            }
          ]
        }
      ]
    },
    {
      // ext_c in the subgraph sends cmd H to ext_a, this is advanced usage
      "extension": "subgraph_1:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Subgraph subgraph_2 sends cmd H to ext_a, but this requires the subgraph to expose the H
      // cmd_out interface, otherwise it will be treated as an error condition
      "subgraph": "subgraph_2",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

#### Key Concepts

The subgraph mechanism introduces three key concepts:

1. **Message Exposure** (exposed\_messages):
   * Subgraphs declare externally exposed interfaces through the `exposed_messages` field
   * Mainly used by development tools to implement intelligent hints and checks
   * Hides internal details of subgraphs, improving development experience

2. **Property Exposure** (exposed\_properties):
   * Subgraphs declare externally exposed properties through the `exposed_properties` field
   * Mainly used by development tools to implement intelligent hints and checks
   * Hides internal details of subgraphs, improving development experience

3. **Subgraph Reference and Naming**:
   * Reference other graph files through `"type": "subgraph"`
   * Each subgraph has a unique identifier name serving as a namespace
   * Prevents conflicts between elements with the same name in different subgraphs

4. **Cross-Graph Connections**:
   * Reference elements within subgraphs through namespace syntax (such as `subgraph_1:ext_d`), or reference subgraphs through the `subgraph` field
   * Makes elements within subgraphs interact with the main graph
   * Builds complex cross-graph message flows

In summary, the `exposed_messages` and `exposed_properties` fields describe how messages flow into the graph from the graph boundary, and development tools use these fields to provide intelligent hints, making it convenient for developers to specify how messages flow to the graph boundary.

#### Flattening Mechanism

Ultimately, graphs referencing subgraphs are flattened into regular graph structures, ensuring runtime uniformity and efficiency. However, this flattening result is not seen by users, but is flattened into memory for use in the TEN runtime or TEN manager. Note that in the flattened graph definition, there are the original extensions and the connections between these extensions.

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // ext_c from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "extension_c"
    },
    {
      // ext_d from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "extension_d",
      "property": {
        // The property field defined when referencing the subgraph will be flattened into the corresponding extension's property field
        "app_id": "${env:AGORA_APP_ID}"
      }
    },
    {
      // ext_e from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "extension_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b"
            },
            {
              "extension": "subgraph_1_ext_d"
            },
            {
              "extension": "subgraph_2_ext_e"
            }
          ]
        }
      ]
    },
    {
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Internal connections within subgraphs are also flattened and included
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "subgraph_1_ext_d"
            }
          ]
        }
      ]
    }
  ]
  // exposed_messages and exposed_properties fields are discarded during flattening
}
```

The subgraph flattening mechanism follows these rules:

1. Before flattening, the colon (`:`) symbol indicates that an element is located within a subgraph (such as `subgraph_1:ext_d`).

2. After flattening, elements within subgraphs have the subgraph name added as a prefix to their names (such as `subgraph_1_ext_c`), ensuring global uniqueness.

3. The flattened graph definition no longer contains colon symbols, distinguishing between pre-flattening and post-flattening states.

4. Internal connections within subgraphs are preserved and included in the flattened graph, ensuring functional completeness.

### Advanced Features and Applications

As project complexity increases, advanced features of subgraphs can help better organize and manage systems.

#### Message Conversion and Subgraphs

Subgraphs fully support the message conversion (msg\_conversion) mechanism for handling message format conversion between different interfaces:

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "http://a.b.c.d/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1:ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "subgraph": "subgraph_2",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Development tools can use `exposed_messages` information to suggest compatibility and provide message conversion configuration interfaces. After configuration, conversion rules are automatically written into the graph definition, simplifying the development process.

After flattening, message conversion rules are correctly preserved, ensuring runtime behavior matches design intent:

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "addon_c"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "addon_d"
    },
    {
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "addon_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1_ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_2_ext_e",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Through this approach, developers can flexibly handle message format differences between different components without needing to understand underlying implementation details.

## Meaning of "subgraph\_1:ext\_c"

In the previous examples, `subgraph_1` in `subgraph_1:ext_c` does not represent the subgraph itself, but references an element in the `nodes` field. In subgraph applications, `subgraph_1` can be viewed as a namespace.

Through this mechanism, you can specify extensions at different locations as sources or targets in graph connections, enabling connections across multiple graphs.

The core concept is: graph connections are essentially connections between extensions. Through this mechanism, you can specify elements of different types in the `nodes` field as sources or targets of connections, enabling connections across multiple subgraphs or even multiple graphs.

## Connections Across Multiple Graphs

In addition to subgraphs, you can also specify connections to other graphs within a single graph through similar mechanisms.

### Connecting to Predefined Graphs in the Same TEN App

Example: How to connect to predefined graphs within the same TEN app in a graph:

```json
{
  "nodes": [
    {
      // Define extension named ext_a
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // Define extension named ext_b
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // Reference other graph, graph ID is default
      "type": "graph",
      "graph_id": "default"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // First target is ext_b
              "extension": "ext_b"
            },
            {
              // Second target is ext_d in another graph
              "extension": "default:ext_d"
            }
          ]
        }
      ]
    },
    {
      // ext_c in another graph sends cmd H to ext_a
      "extension": "default:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Another graph sends cmd X to ext_a, but this requires the other graph to expose the X cmd_out interface,
      // otherwise it will be treated as an error condition. This error condition can be reported during static
      // checking if it can be statically detected, or at runtime.
      "graph": "default",
      "cmd": [
        {
          "name": "X",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

### Types of Graph Nodes

1. Subgraph

   ```json
   {
     "type": "subgraph",
     "name": "subgraph_1", // Meaning of namespace
     "import_uri": "http://a.b.c.d/subgraph.json"
   }
   ```

2. Local graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": true
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": false
   }
   ```

3. Remote graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": true,
     "app": "msgpack://127.0.0.1:8002/"
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": false,
     "app": "msgpack://127.0.0.1:8002/"
   }
   ```


# Message Conversion (/docs/ten_framework/0.11.59/graph/msg_conversion)

## Message Conversion

TEN Framework provides powerful message conversion capabilities that allow you to transform and modify message content during message passing. By configuring conversion rules, you can achieve message field renaming, value replacement, and complex data structure transformations.

### Basic Configuration

Message conversion is configured through the `msg_conversion` configuration item:

```json
{
  "msg_conversion": {
    "type": "per_property",
    "keep_original": false,
    "rules": [
      {
        "path": "user.name",
        "conversion_mode": "fixed_value",
        "value": "default_user"
      },
      {
        "path": "data.items[0].id",
        "conversion_mode": "from_original",
        "original_path": "original_id"
      }
    ]
  }
}
```

### Configuration Parameters

* **type**: Conversion type, currently supports `per_property` (per-property conversion)
* **keep\_original**: Whether to retain original message content; when set to `false`, only converted fields are preserved
* **rules**: Array of conversion rules that define specific conversion logic

### Conversion Rules

Each conversion rule contains the following fields:

* **path**: Target field path, specifying the message field location to be set
* **conversion\_mode**: Conversion mode
  * `fixed_value`: Set a fixed value
  * `from_original`: Copy value from a specified path in the original message
* **value**: Fixed value used when conversion mode is `fixed_value`
* **original\_path**: When conversion mode is `from_original`, specifies the source field path in the original message

### Path Expressions

Path expressions in message conversion support multiple access methods:

* **Object Property Access**: `user.profile.name` - Access properties of nested objects
* **Array Index Access**: `items[0].name` - Access elements at specific array indices
* **Mixed Path Access**: `user.tags[1].value` - Combine object and array access

### Use Cases

Message conversion functionality is suitable for the following scenarios:

* **Field Renaming**: Convert field names in messages to formats required by target systems
* **Data Format Conversion**: Adjust message data structures to adapt to different processing modules
* **Default Value Setting**: Provide default values for missing fields
* **Data Cleansing**: Filter or transform data content that doesn't meet requirements


# Go (/docs/ten_framework/0.11.59/language_binding/go)

Let's build a Go application using the TEN framework.

## Creating the TEN App

First, we'll create a basic TEN Go app by integrating several pre-built TEN packages. Follow these steps:

```shell title="Terminal"
tman install app default_app_go
cd default_app_go

tman install protocol msgpack
tman install extension_group default_extension_group
```

### Installing a Default TEN Extension

Next, install a default TEN extension written in Go:

```shell title="Terminal"
tman install extension default_extension_go
```

### Declaring the Prebuilt Graph for Auto-Start

Now, we'll modify the `property.json` file of the TEN app to include a graph declaration. This will ensure the default extension starts automatically when the TEN app is launched.

```json title=".json"
"predefined_graphs": [
  {
    "name": "default",
    "auto_start": true,
    "nodes": [
      {
        "type": "extension_group",
        "name": "default_extension_group",
        "addon": "default_extension_group"
      },
      {
        "type": "extension",
        "name": "default_extension_go",
        "addon": "default_extension_go",
        "extension_group": "default_extension_group"
      }
    ]
  }
]
```

## Building the App

Unlike standard Go projects, the TEN Go app uses CGo, so you need to set up certain environment variables before building. A build script is already provided in the TEN runtime Go binding system package, so you can build the app with a single command:

```shell title="Terminal"
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```

The compiled binary, `main`, will be generated in the `/bin` folder.

## Starting the App

Since some environment variables need to be set, it is recommended to start the app using the provided script:

```shell title="Terminal"
./bin/start
```

## Debugging

If you are using Visual Studio Code (VSCode) as your development environment, you can use the following configurations to debug Go/C code.

### Debugging Go Code

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Go app (launch)",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "cwd": "${workspaceFolder}",
      "output": "${workspaceFolder}/bin/tmp",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

### Debugging C Code

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "C (launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/main",
      "cwd": "${workspaceFolder}",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

## CGO

### Generated Code

When interfacing Go with C, the cgo tool is crucial. When a C function is called from Go, cgo converts the Go source files into multiple output files, including both Go and C source files. These generated C source files are then compiled using compilers like gcc or clang.

Use the following command to generate these C/Go source files:

```bash title="Terminal"
mkdir out
go tool cgo -objdir out
```

Example:

```bash title="Terminal"
go tool cgo -objdir out cmd.go error.go
```

The generated files include:

```bash title="Terminal"
├── _cgo_export.c
├── _cgo_export.h
├── _cgo_flags
├── _cgo_gotypes.go
├── _cgo_main.c
├── _cgo_.o
├── cmd.cgo1.go
├── cmd.cgo2.c
├── error.cgo1.go
└── error.cgo2.c
```

* **\_cgo\_export.h** is key to the interoperability between Go and C within the cgo context. It contains necessary declarations for Go functions accessible from C.

  This file is automatically generated based on Go functions explicitly exported using the `//export` directive. It also includes definitions for C types corresponding to the Go types used in these functions, ensuring compatibility between the two languages.

  Example of type definitions:

```c title=".h"
typedef signed char GoInt8;
typedef unsigned char GoUint8;
typedef short GoInt16;
typedef unsigned short GoUint16;
typedef int GoInt32;
typedef unsigned int GoUint32;
typedef long long GoInt64;
typedef unsigned long long GoUint64;
typedef GoInt64 GoInt;
typedef GoUint64 GoUint;
typedef size_t GoUintptr;
typedef float GoFloat32;
typedef double GoFloat64;
```

* **\_cgo\_gotypes.go** contains the corresponding Go types defined in C and used in Go.

  For example, if you define a struct `ten_go_error_t` in a header file `common.h` and use `C.ten_go_error_t` in Go, there will be a corresponding Go type in `_cgo_gotypes.go`:

```go title=".go"
package ten

type _Ctype_ten_go_status_t = _Ctype_struct_ten_go_status_t

type _Ctype_struct_ten_go_status_t struct {
    err_no   _Ctype_int
    msg_size _Ctype_uint8_t
    err_msg  [256]_Ctype_char
    _        [3]byte
}
```

* **cmd.cgo1.go** is a cgo-generated counterpart to the original Go source file `cmd.go`. It replaces direct calls to C functions and types with calls to the generated Go functions and types provided by cgo.

  Example:

```go title=".go"
package ten

  func NewCmd(cmdName string) (Cmd, error) {
      // ...
      cStatus := func() _Ctype_struct_ten_go_status_t {
          var _cgo0 _cgo_unsafe.Pointer = unsafe.Pointer(unsafe.StringData(cmdName))
          var _cgo1 _Ctype_int = _Ctype_int(len(cmdName))
          _cgoBase2 := &bridge
          _cgo2 := _cgoBase2
          _cgoCheckPointer(_cgoBase2, 0 == 0)
          return _Cfunc_ten_go_cmd_create_custom_cmd(_cgo0, _cgo1, _cgo2)
      }()
      // ...
  }
```

* **cmd.cgo2.c** is a wrapper of the original C function called from Go.

  Example:

```c title=".c"
  CGO_NO_SANITIZE_THREAD void _cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd(void *v) {
      struct {
          const void* p0;
          int p1;
          char __pad12[4];
          ten_go_msg_t** p2;
          ten_go_error_t r;
      } __attribute__((__packed__, __gcc_struct__)) *_cgo_a = v;
      char *_cgo_stktop = _cgo_topofstack();
      __typeof__(_cgo_a->r) _cgo_r;
      _cgo_tsan_acquire();
      _cgo_r = ten_go_cmd_create_cmd(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2);
      _cgo_tsan_release();
      _cgo_a = (void*)((char*)_cgo_a + (_cgo_topofstack() - _cgo_stktop));
      _cgo_a->r = _cgo_r;
      _cgo_msan_write(&_cgo_a->r, sizeof(_cgo_a->r));
  }
```

So, the calling sequence of `C.ten_go_cmd_create_cmd()` from Go is:

```text
_Cfunc_ten_go_cmd_create_custom_cmd (auto-generated Go)
           |
           V
   _cgo_runtime_cgocall (Go)
           |
           V
      entrysyscall     // switch Go stack to C stack
           |
           V
_cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd (auto-generated C)
           |
           V
   ten_go_cmd_create_cmd (C)
           |
           V
      exitsyscall
```

### Incomplete Type

As mentioned earlier, the cgo tool generates corresponding Go types in `_cgo_gotypes.go` based on C header files imported via `import "C"`. If a struct is opaque in the C header, the cgo tool generates an incomplete type instead.

Example of an opaque C struct:

```c title=".h"
typedef struct ten_go_msg_t ten_go_msg_t;
```

The cgo tool will generate an incomplete type in Go:

```go title=".go"
type _Ctype_ten_go_msg_t = _Ctype_struct_ten_go_msg_t
type _Ctype_struct_ten_go_msg_t _cgopackage.Incomplete
```

#### What happens if you use the incomplete type in Go?

* **Incomplete types cannot be allocated.**

  Since `sys.NotInHeap` cannot be allocated on the Go heap, operations like `new` or `make` won't work. Attempting to create a new instance of an opaque struct in Go will result in a compiler error:

```go title=".go"
msg := new(C.ten_go_msg_t) // Error: can't be allocated in Go; it is incomplete (or unallocatable)
```

* **Pointers to incomplete types cannot be passed to C directly.**

  If you have a C function with a pointer to an opaque struct as a parameter, passing a Go pointer to this incomplete type directly to a C function will not work according to cgo rules. The Go compiler will require the pointer to be "pinned" to ensure it adheres to Go's garbage collector (GC) constraints.

### Rules for Using C.uintptr\_t Instead of a Pointer to an Opaque Struct

* **Benefits:**

  * `C.uintptr_t` and `uintptr` in Go are integers large enough to hold a C pointer, avoiding memory allocation or conversion when passing from Go to C.

* **Limitations:**
  * `uintptr` is an integer, not a pointer. It represents the bit pattern of an address with no pointer arithmetic.
  * `uintptr` cannot be dereferenced in Go. Converting it to `unsafe.Pointer` can cause issues with Go's GC.
  * Since `uintptr` is an integer, `nil` or `NULL` cannot be passed to C. Use `0` instead of `nil` to represent a null address.


# Manifest (/docs/ten_framework/0.11.59/metadata_system/manifest)

## What is a Manifest File

A Manifest file is a crucial configuration file in the TEN Framework used to store metadata for software packages. It defines essential information about Extensions or Apps, including basic information, dependencies, API interfaces, script configurations, and other key details.

## Purpose of Manifest Files

* **Package Identification**: Define the package type, name, and version information
* **Dependency Management**: Declare required dependencies and their version requirements
* **API Definition**: Describe data interfaces and command interfaces provided by the package
* **Package Configuration**: Specify files to be included in the release version
* **Script Management**: Define automation scripts for building, testing, and other tasks

## Extension Manifest Example

Here's a complete Manifest example for an AI chat extension:

```json
{
  "type": "extension",
  "name": "openai_chatgpt_python",
  "version": "0.1.0",
  "tags": ["ai", "llm", "python"],
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "^0.8.0"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "*.py",
      "README.md"
    ]
  },
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        }
      }
    },
    "data_in": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "data_out": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_in": [
      {
        "name": "flush"
      },
      {
        "name": "tool_register",
        "property": {
          "properties": {
            "tool": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "description": {
                  "type": "string"
                },
                "parameters": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                }
              },
              "required": ["name", "description", "parameters"]
            }
          }
        },
        "result": {
          "property": {
            "properties": {
              "response": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "tool_call",
        "property": {
          "properties": {
            "name": {
              "type": "string"
            },
            "args": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    ]
  },
  "scripts": {
    "test": "python -m pytest tests/",
    "build": "python setup.py build"
  }
}
```

## App Manifest Example

Here's a Manifest example for a voice assistant application:

```json
{
  "type": "app",
  "name": "voice_assistant_app",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "^0.8.0"
    }
  ],
  "api": {
    "property": {
      "properties": {
        "app_id": {
          "type": "string"
        },
        "log_level": {
          "type": "uint8"
        },
        "server_config": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "int32"
            },
            "ssl_enabled": {
              "type": "bool"
            }
          },
          "required": ["host", "port"]
        }
      }
    }
  }
}
```

## Key Field Descriptions

### Basic Information

* **type**: Package type, can be `extension` or `app`
* **name**: Unique identifier name for the package
* **version**: Version number following semantic versioning standards
* **tags**: Array of tags for categorization and search

### Dependency Configuration

* **dependencies**: List of dependencies, each containing type, name, and version requirements

### Package Configuration

* **package.include**: Specifies files and directories to include during packaging

### API Configuration

* **api.property**: Defines configuration properties for the extension
* **api.data\_in/data\_out**: Defines data input and output interfaces
* **api.cmd\_in/cmd\_out**: Defines command input and output interfaces

### Script Configuration

* **scripts**: Defines various automation scripts such as testing, building, etc.

## Best Practices

1. **Version Management**: Use semantic versioning to ensure version consistency
2. **Dependency Declaration**: Clearly declare all required dependencies and their version ranges
3. **API Design**: Clearly define input and output interfaces for easy integration with other components
4. **Complete Documentation**: Ensure that configurations in the Manifest are consistent with actual implementation


# Metadata System (/docs/ten_framework/0.11.59/metadata_system/metadata_system)

The TEN framework utilizes a consistent metadata system across various types of packages, including:

* App
* Extension Group
* Extension
* Protocol
* System

## Types of Metadata

In the TEN metadata system, there are two primary types of metadata:

### 1. **Manifest**

* **Location**: Stored in the root directory of the associated TEN package with the file name `manifest.json`.
* **Contents**:
  * **Package Name**: The name of the TEN package.
  * **Package Version**: The version of the TEN package, following semantic versioning.
  * **API Schema**: Defines schemas related to the package's properties and input/output messages.
    * **Property Schema**: These schemas are often defined in a `property.json` file in the root directory. The manifest can specify the schema for these properties.
    * **Message Schema**: Defines the schema for input/output messages handled by the package.

> ⚠️ **Note**: The API schema in `manifest.json` is *not* a JSON schema. Instead, it describes the metadata of TEN values, which are central to the TEN runtime, while JSON is merely a representation format.

### 2. **Property**

* **Location**: Typically stored in a `property.json` file in the root directory of the TEN package.
* **Purpose**: The `property.json` file stores initial property values, which are read-write during runtime. This means that properties can be modified while the TEN runtime is executing.

#### Example of `property.json`

```json
{
  "Darth Vader": "I am your father"
}
```

## Manifest File

The `manifest.json` file serves as the blueprint for the TEN package, defining its metadata, properties, and the input/output messages it handles.

### Example of `manifest.json`

```json
{
  "type": "app",
  "name": "default_app_cpp",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.0"
    }
  ],
  "api": {
    "property": {
      "exampleInt8": {
        "type": "int8"
      },
      "exampleString": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "detail": {
              "type": "string"
            },
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

### Purpose of TEN Schema in Manifest

The TEN schema in `manifest.json` provides the TEN runtime with metadata about the extension's external API, including the type information for properties and messages. This helps the runtime platform correctly handle the extension's properties and messages during operation.

### When TEN Schema is Used

1. **Property Validation**: When the TEN runtime gets/sets properties of a TEN package or message.
2. **Data Conversion**: When converting a JSON document to a TEN package or message property using the `from_json` API.
3. **Compatibility Check**: When the TEN manager checks if a message can be output from a source extension to a destination extension according to the TEN schema.

## Property Management

In the TEN framework, there are two types of properties:

1. **Message Properties**: These properties are associated with the messages exchanged between extensions within the framework. Message properties define the specific data or parameters that are carried within a message, such as command parameters, data payloads, or metadata related to audio/video frames.

2. **TEN Package Properties**: These properties are associated with the TEN packages themselves, such as extensions. TEN package properties define the configuration or settings specific to a package. For example, an extension might have properties that configure its behavior, such as runtime settings, initialization parameters, or other configuration data.

Both types of properties are managed within the TEN framework but serve different purposes, one focuses on the communication between components (message properties), while the other pertains to the configuration and operation of the components themselves (TEN package properties).

<figure><img src="..https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_system.png" alt=""><figcaption><p>Property system</p></figcaption></figure>

### Defining TEN Package Properties

The `property.json` file defines the properties of a TEN package. Here's an example:

```json
{
  "prop_1_name": 0,
  "prop_2_name": "prop_2_value",
  "prop_3_name": ["hello", "prop_3_sub_value"],
  "prop_4_name": {
    "prop_4_1_name": 1,
    "prop_4_2_name": "prop_4_sub_value"
  }
}
```

> ⚠️ **Note**: Each property name in the `property.json` file must be unique.

### TEN Schema for Properties

You can define a TEN schema for properties in the `manifest.json` file, enabling the TEN runtime to handle these properties more effectively. If a property does not have a corresponding TEN schema, the runtime platform will use the default JSON handling method (e.g., treating all JSON numbers as `float64`). If a TEN schema is provided, the runtime platform will use it to validate and process the property accordingly.

| Property | TEN Schema | Effect                                                                                          |
| -------- | ---------- | ----------------------------------------------------------------------------------------------- |
| Yes      | Yes        | The TEN runtime validates the property value based on the TEN schema (e.g., checking the type). |
| Yes      | No         | The TEN runtime uses default handling, treating all JSON numbers as `float64`.                  |

### Accessing Properties from an Extension

The TEN runtime provides APIs for extensions to access various properties.

### Accessing App Properties from an Extension

> ⚠️ **Note**: The behavior described in this section has not yet been implemented and is part of the future roadmap.

The TEN framework provides a powerful property access mechanism that allows extensions to read and modify app properties. Through the get/set property API, you can access all properties, including app properties. When the API parameter begins with the `app:` prefix, the system identifies this as a request to operate on app properties.

API design considerations for different language implementations:

* **Go**: Providing a synchronous version without callbacks is sufficient, making it concise and efficient.
* **Node.js**: Implemented as an async function without callbacks, aligning with Node.js's asynchronous programming model.
* **Python**: Ideally, versions both with and without callbacks should be provided. Currently, a synchronous version without callbacks can be implemented first, but accessing `app:` properties will use event wait, which may impact performance.
* **Async Python**: Implemented as an async function without callbacks, consistent with Python's asynchronous model.
* **C++**: Both versions with and without callbacks should be provided. Currently, a synchronous version without callbacks can be implemented first, but accessing `app:` properties will use event wait, which may impact performance.

This design enables CRUD operations on app properties through the unified get/set\_property API, maintaining interface consistency without affecting message property-related APIs.

Regarding the handling of `ten:` namespace fields, the following rules are adopted:

1. `get/set_property_to/from_json(nullptr)` operations do not process `ten:` fields
2. To access `ten:` fields, you must explicitly specify the complete path such as `get/set_property("ten:xxx")`
3. When using `get/set_property_to/from_json()`, `ten:` fields are only processed when a key in the form of `"ten:xxx"` is explicitly specified

This design maintains API simplicity while avoiding the complexity of creating additional commands for specific requirements.

### Specifying Property Values in the `start_graph` Command

Property values related to the TEN package can be specified in the `start_graph` command. The TEN runtime processes these properties according to the TEN schema (if defined) and stores them within the corresponding TEN package instance.

#### Example of Property Specification in `start_graph` Command

```json
{
  "nodes": [
    {
      "type": "extension_group",
      "name": "foo_extension_group",
      "addon": "foo_extension_group_addon"
    },
    {
      "type": "extension",
      "name": "bar_extension",
      "extension_group": "foo_extension_group",
      "property": {
        "prop_1_name": 0,
        "prop_2_name": "prop_2_value",
        "prop_3_name": ["hello", "prop_3_sub_value"],
        "prop_4_name": {
          "prop_4_1_name": 1,
          "prop_4_2_name": "prop_4_sub_value"
        }
      }
    }
  ]
}
```


# Property (/docs/ten_framework/0.11.59/metadata_system/property)

## Property File Overview

The Property file is a configuration file in the TEN Framework used to store attribute values for extensions. It defines runtime parameters and configuration options for extensions, serving as an essential component for their proper operation.

## Property File Format

Property files use JSON format and contain various configuration parameters required for extension operation, such as API keys, model configurations, runtime parameters, etc.

### Basic Property Example

Here's a simple Property file example:

```json
{
  "api_key": "your-api-key-here",
  "max_tokens": 1000,
  "temperature": 0.7,
  "model_config": {
    "name": "gpt-4",
    "version": "2024-01-01"
  },
  "allowed_languages": ["en", "zh", "es"]
}
```

### AI Model Configuration Example

Property file configuration example for AI model extensions:

```json
{
  "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxx",
  "model": "gpt-4",
  "max_tokens": 4096,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "prompt": "You are a helpful AI assistant.",
  "greeting": "Hello! How can I help you today?",
  "max_memory_length": 10,
  "vendor": "openai"
}
```

## Graph Configuration

Property files can also contain Graph Configuration, which defines connection relationships and data flow between extensions.

### Predefined Graph Configuration Example

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "nodes": [
          {
            "type": "extension",
            "name": "my_extension",
            "addon": "my_extension_addon",
            "property": {
              "setting1": "value1",
              "setting2": 42
            }
          }
        ],
        "connections": [
          {
            "extension": "source_extension",
            "cmd": [
              {
                "name": "process_cmd",
                "dest": [
                  {
                    "extension": "target_extension"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## Usage Guidelines

* Property files must be in valid JSON format
* Property values are loaded when the extension starts
* Extension behavior can be adjusted by modifying the Property file
* The graph configuration section defines connections and data flow between extensions


# Addon System (/docs/ten_framework/0.11.59/software_packages/addon_system)

The TEN framework is a software module system, where each software module is defined by the concept of an **addon**. Essentially, every software module in the TEN framework is an addon. For example, users can define the following types of TEN framework addons:

1. TEN **extension** addon
2. TEN **extension group** addon
3. TEN **protocol** addon

## Information Required to Load Addon

To load an addon into the TEN runtime, three pieces of information are needed:

1. **Addon name**

   It must be globally unique within the entire TEN app. Loading multiple addons with the same name into the TEN runtime is undefined behavior, and it may result in the second instance of the addon not being properly loaded into the TEN runtime.

2. **Addon folder location**

   In the TEN framework, when loading an addon, users must inform the TEN runtime where the addon's directory is located. This directory serves as the addon's `base_dir`. The TEN runtime will use this base directory to load other files related to the addon, such as the addon's `manifest.json` and `property.json` files.

   Basically, when designing the TEN framework's addon register API, it aims to automatically detect the directory where the addon is located, so the user does not need to explicitly specify the `base_dir`. However, in some special cases where the TEN framework cannot automatically detect the `base_dir`, the user will need to explicitly specify the folder where the addon is located for the TEN runtime.

3. **Addon instance itself**

## Timing for Registering Addon to TEN Runtime

There are two ways to register an addon into the TEN runtime:

1. Explicitly call the API to register the addon's `base_dir` to the TEN runtime.

   **Note** Currently, only the standalone testing framework provides such an API.

2. Place the addon in a designated location within the TEN app folder tree.

By placing the addon in the specified location, the TEN app will automatically load these addons when it starts. Currently, this designated location is under the `ten_packages/` directory in the TEN app. If the addons are placed in this format, they will be automatically loaded when the TEN app starts.

```text
.
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    └── protocol/
        └── <protocol_a>/
```


# App (/docs/ten_framework/0.11.59/software_packages/app)

App is the container in the TEN world, containing groups and extensions.
App can function as a process placed inside a Docker container (primarily used in the cloud).
App can also function as a thread embedded within an existing app (on client or server).
⇒ TEN can perform localized modifications (minor enhancements) on an existing project without requiring the entire project to be rewritten according to TEN.

![Run TEN as a Process or a Thread](https://ten-framework-assets.s3.amazonaws.com/doc-assets/run_ten_as_process_or_thread.png)

![Different mode of app running](https://ten-framework-assets.s3.amazonaws.com/doc-assets/different_mode_of_app_runnings.png)

## TEN App Folder Structure

In the TEN framework, developers can create TEN packages using various programming languages. Regardless of the language used, the overall folder structure of a TEN app remains consistent, ensuring uniformity across different implementations.

![TEN package folder structure](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_package_folder_structure.png)


# Extension (/docs/ten_framework/0.11.59/software_packages/extension)

## Life Cycle

The life cycle of an extension is divided into the following stages:

1. `on_configure`
2. `on_init`
3. `on_start`
4. `on_stop`
5. `on_deinit`

At all these stages, the extension can send messages, and if the sent message is a command type, the extension can also receive the result of the command it sent. In other words, there is no stage where the extension *cannot* send messages or receive the results of its own sent commands. So, in all these stages, the extension can send messages to other extensions, which helps implement dependencies between extensions in all stages.

Each life cycle stage corresponds to a callback function, and there is a corresponding `on_xxx_done()` function to mark the end of that life cycle stage.

| Life Cycle Callback | End of Life Cycle Function |
| ------------------- | -------------------------- |
| `on_configure`      | `on_configure_done`        |
| `on_init`           | `on_init_done`             |
| `on_start`          | `on_start_done`            |
| `on_stop`           | `on_stop_done`             |
| `on_deinit`         | `on_deinit_done`           |

### Stage Transition Graph

```mermaid
graph TD;
  on_configure((on_configure));
  on_init((on_init));
  on_start((on_start));
  on_stop((on_stop));
  on_deinit((on_deinit));

  on_configure-->on_configure_done
  on_configure_done-->on_init;
  on_configure-->on_stop;
  on_init-->on_init_done;
  on_init_done-->on_start;
  on_init-->on_stop;
  on_start-->on_stop;
  on_start-->on_start_done;
  on_stop-->on_stop_done;
  on_stop_done-->on_deinit;
  on_deinit-->on_deinit_done;
```

### on\_configure

Used to set the initial values of the extension's properties. This allows the `get_property` API from `ten_env` to be used in other lifecycle stages to retrieve properties set during the `on_configure()` stage. The `on_configure_done()` function is called to mark the end of the `on_configure` stage.

```c++
  void on_configure(ten::ten_env_t &ten_env) override {
    // Set the initial values of the extension's properties.
    ten_env.init_property_from_json(
      R"({
           "ten": {
             "uri": "msgpack://127.0.0.1:8001/",
             "log_level": 2
           }
         })", nullptr);
    ten_env.on_configure_done();
  }
```

### on\_init

Used to initialize the extension. Since the extension is not yet ready before `on_init_done()`, the TEN runtime will queue all messages sent to the extension until it's ready. Once ready, the queued messages are delivered to the extension. It's important to note that if the result is from a command sent by the extension itself, it is *not* subject to this restriction; the TEN runtime will directly pass the command's result back to the extension. This is because sending a command is a self-initiated action by the extension, so it must be prepared to receive and handle the result of its own commands.

### on\_start

This stage marks the runtime starting to send messages into the extension. The `on_start_done` function doesn't have a special purpose, but for consistency with other life cycle stages and to reduce learning complexity, the TEN framework includes it. Typically, `on_start_done` is invoked at the end of the `on_start` callback function. The TEN runtime will start delivering messages from other extensions to this extension after receiving its `on_start_done()`.

```c++
  void on_start(ten::ten_env_t &ten_env) override {
    // Some operations.
    ten_env.on_start_done();
  }
```

### on\_stop

There are many situations where the extension needs to stop, such as when the app or engine containing the extension is terminating. When the extension is about to stop, the TEN runtime uses this `on_stop` callback to notify the extension that it has entered the `on_stop` life cycle stage. The extension can then perform any necessary actions for termination.

### on\_deinit

After the extension calls `on_stop_done()`, it enters the `on_deinit` stage. During this stage, because the resources within the extension may no longer be fully available, the TEN runtime will not pass any messages from other extensions to this one.

## Relationship Between Extensions at Different Life Cycle Stages

Basically, there is no inherent relationship. Each extension operates independently, switching between its own life cycle stages. Extensions are independent of one another, and any dependencies between them must be explicitly implemented by the extensions themselves. The TEN runtime does not make any assumptions or guarantees. For example, if extension A needs to wait for extension B to complete its initialization before finishing its own, extension A can send a command to extension B during its `on_init`. Once extension B completes initialization and receives the command, it can reply with a result, and when extension A receives the result, it can call `on_init_done`.

```mermaid
sequenceDiagram
  participant A as Extension A
  participant B as Extension B

  Note over A,B: on_init
  activate A
  activate B
  deactivate B
  Note over B: on_init_done

  Note over B: on_start
  activate B
  deactivate B
  Note over B: on_start_done

  A->>B: cmd
  B-->>A: result

  deactivate A
  Note over A: on_init_done
```

Basically, in all other life cycle stages, if you want to implement the order of different extensions, you can achieve it using the same method.

## Interface with TEN Runtime

Extensions interact with the TEN runtime primarily through three interfaces:

1. **Lifecycle Callbacks**

   * These include callbacks like `on_init`, `on_deinit`, `on_start`, and `on_stop`.

2. **Callbacks for Receiving Messages**

   * These include callbacks such as `on_cmd`, `on_data`, `on_audio_frame`, and `on_video_frame`, which handle incoming messages to the extension.

3. **Functions for Sending Messages**

   * These include functions like `send_cmd`, `send_data`, `send_audio_frame`, and `send_video_frame`, which send messages out of the extension.

## Lifecycle Callbacks

The different stages of the extension's lifecycle and their connection to message handling are as follows:

* **on\_init \~ on\_init\_done**: Handles the extension's own initialization. At this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions.
* **on\_start \~ on\_start\_done**: At this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions. Since properties are initialized during `on_configure`, you can perform actions that depend on these properties being set up. However, as this is still part of the initialization phase, the extension will not receive messages initiated by others, avoiding the need for various checks. Active message sending is allowed.
* **After on\_start\_done \~ on\_stop\_done**: During this phase, the extension can normally send and receive all types of messages and their results.
* **on\_deinit \~ on\_deinit\_done**: Handles the extension's de-initialization. Similar to the `on_init` stage, at this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions.

## Implementing Extensions in Different Languages

Within the TEN framework, extensions can be implemented in various languages such as C++, Go, and Python. Developers can use the same conceptual approach to implement extensions in different languages. Learning how to develop an extension in one language makes it relatively easy to do so in other languages as well.

## Asynchronous Message Processing in Extensions

<figure><img src="../../..https://ten-framework-assets.s3.amazonaws.com/doc-assets/asynchronous_message_processing.png" alt=""><figcaption><p>Asynchronous Message Processing</p></figcaption></figure>

Extensions process messages asynchronously. When the TEN runtime delivers a message to an extension through callbacks like `on_cmd`, `on_data`, `on_audio_frame`, or `on_video_frame`, the extension is not required to process the message immediately within the callback. Instead, the extension can delegate the message to other threads, processes, or even machines for processing. This allows for full utilization of multi-core and distributed computing resources.

After processing is complete, the results can be sent back to the TEN runtime through callbacks such as `send_cmd`, `send_data`, `send_audio_frame`, or `send_video_frame`. The entire process is asynchronous, meaning the extension doesn't need to send the processed results back before the `on_cmd`, `on_data`, `on_audio_frame`, or `on_video_frame` callbacks return. The results can be transmitted only when they are actually ready, using the appropriate send functions.


# Overview (/docs/ten_framework/0.11.59/software_packages/overview)

The directory structure of a basic TEN framework software package is shown below.

```text
.
├── bin/
├── src/
├── manifest.json
├── manifest-lock.json
├── property.json
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    ├── protocol/
    │   └── <protocol_a>/
    └── system/
        ├── <system_package_1>/
        └── <system_package_2>/
```

Depending on the programming language, there may be additional required files. For example, in a C++ project, since the build system is `ten_gn`, a `BUILD.gn` file would be present. In a Go project, there would be files such as `go.mod` and `go.sum`. In a Python project, you might find a `requirements.txt` file.

## TEN Framework Software Package Types

| Type            | Description                     |
| --------------- | ------------------------------- |
| App             | Contains a TEN app.             |
| Extension group | Contains a TEN extension group. |
| Extension       | Contains a TEN extension.       |
| Protocol        | Contains a TEN protocol.        |
| System          | Contains a TEN system package.  |

## TEN Framework Software Package Kinds

There are two types of software packages:

* Release package
* Development package

The relationship between these two package types is as follows:

* A release package can be generated from a development package through a build process.
* If we add source-related content to a release package, it becomes a development package.
* If we remove source-related content from a development package, it becomes a release package.

In short, the directory structures of development and release packages are very similar, with the difference being that a development package includes source-related content, whereas a release package does not.

### Development package

The primary purpose of a development package is to develop TEN software packages. For instance, if you want to create a new TEN extension, you can modify an existing development package to achieve this. This is because the development package contains all the source-related content, enabling the creation of new TEN software packages from it.

### Release package

The primary purpose of a release package is to create and run graphs. Since no build tasks are required, a release package can be used to perform these actions. However, a development package can also be used for these tasks. Essentially, the application scope of a development package is broader than that of a release package.


# Python Async Extension (/docs/ten_framework/0.11.59/software_packages/python_async_extension)

A **Python async extension** is a modular component that using Python's `asyncio` framework. If you want to wrap existing Python codes that use `asyncio` into a TEN extension, using the Python async extension is the simplest and most convenient option.

## Example: The Default Python Async Extension

Here’s how a Python async extension is structured:

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    async def on_configure(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_json = cmd.to_json()
        ten_env.log_debug(f"DefaultAsyncExtension on_cmd: {cmd_json}")

        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

        # Send a new command to other extensions and wait for the result. The
        # result will be returned to the original sender.
        new_cmd = Cmd.create("hello")
        cmd_result = await ten_env.send_cmd(new_cmd)
        ten_env.return_result(cmd_result, cmd)
```

Each method simulates a delay using `await asyncio.sleep()`.

## FAQ

### Aysnc loop for event handling

* Create a queue: use `asyncio.Queue`.
* Create an async task for event handling.

Here is the sample code:

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    queue = asyncio.Queue()
    loop:asyncio.AbstractEventLoop = None

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        self.loop = asyncio.get_event_loop()

        self.loop.create_task(self._consume())

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        self.queue.put(None)

    async def _consume(self) -> None:
        while True
            try:
                value = await self.queue.get()
                if value is None:
                    self.ten_env.log_info("async loop exit")
                    break

                # Code for processing values retrieved from the queue.

            except Exception as e:
                self.ten_env.log_error(f"Failed to handle {e}")
```

## Conclusion

TEN's Python async extension provide a powerful way to handle long-running tasks asynchronously. By integrating Python’s `asyncio` framework, the extensions ensure that operations such as network calls or file handling are efficient and non-blocking. This makes TEN a great choice for building scalable, modular applications with asynchronous capabilities.


# Standard Interface (/docs/ten_framework/0.11.59/standard_extension/standard_interface)

In the TEN framework, we encourage developers to create their own extensions and publish them to the TEN extension marketplace for other developers to use. To enable developers to easily select and replace extensions, we need to define standardized interface specifications so that developers can quickly find extensions that meet their requirements based on these standard interfaces.

In short, standard interfaces are syntactic sugar that allows TEN tools (such as TEN Manager) to more conveniently assist developers in selecting and replacing extensions.

TEN's interface definitions are similar to TypeScript's `.d.ts` files, allowing the TypeScript compiler to perform type checking, but these interface definitions are not used at runtime. Therefore, the functionality and code enhancements brought by standard interface design are mainly reflected in development tools like TEN Manager, rather than the runtime TEN Runtime. For example, TEN Manager needs to provide relevant RESTful APIs to tman designer Frontend, allowing users to select and replace extensions through the UI interface based on standard interface definitions.

## What are Standard Interfaces

In the TEN framework, each extension's interface consists of the following core components:

1. **Properties** - Configuration parameters of the extension
2. **Command**
   * **Input command** - Input commands
   * **Output command** - Output commands
3. **Data**
   * **Input data** - Input data
   * **Output data** - Output data
4. **Audio frame**
   * **Input audio frame** - Input audio frames
   * **Output audio frame** - Output audio frames
5. **Video frame**
   * **Input video frame** - Input video frames
   * **Output video frame** - Output video frames

### Significance and Purpose of Standard Interfaces

Standard interfaces normalize these components into unified interface standards. Developers can refer to these interface standards to implement their own extensions, while extensions can also declare support for specific standard interfaces. This allows development tools to provide better development support based on the standard interfaces declared by extensions, enabling users to select or replace required extensions based on standard interfaces.

In standard interfaces, Command/Data/Audio frame/Video frame are used to standardize interactions between extensions, achieving interchangeability and compatibility of extensions. Properties are used to standardize extension configuration parameters, achieving configuration compatibility with tman designer.

It should be noted that standardizing interactions between extensions is not the function of the interface itself, but the function of Command/Data/Audio frame/Video frame. Interfaces are just syntactic sugar for aggregation purposes, helping developers quickly specify extension interface definitions.

### Interface Checking and Error Reporting

In an extension's Command/Data/Audio frame/Video frame definitions, if there are two messages with the same name and identical schema definitions, this is not considered an error, but will be treated as duplicate definitions and ignored. This rule also applies to standard interfaces, allowing different standard interfaces to define common functionalities themselves, forming self-contained standard interface definitions. When an extension implements these standard interfaces with identical definitions, it won't cause errors. This feature is very useful in standard interfaces because standard interface definitions can be referenced. So when an extension implements multiple standard interfaces, if these standard interfaces (or other standard interfaces they reference) have the same messages and identical schemas, it won't cause errors.

Since the TEN framework adopts a runtime-development separation architecture (similar to the relationship between JavaScript and TypeScript's .d.ts files), standard interface checking and error reporting mainly occur during the development tool phase, not at runtime in TEN Runtime. Therefore, TEN Runtime will not report errors due to interface errors when starting the graph.

In the tman designer development tool, if an extension's interface definition is incomplete, prompts will be displayed indicating that certain functions are temporarily unavailable for that extension. For example, when using one extension to replace another extension's functionality, if the interface definition is incomplete, a prompt will indicate that this function is temporarily unavailable for this extension and needs to wait for the interface definition to be complete before use.

Additionally, since interface definitions can be independent files that are referenced, incomplete interface definitions are not considered errors during TEN Manager's publish/install operations.

### Interface Compatibility Rules

When replacing an extension that implements interface A with an extension that implements interface B, interface A must be compatible with interface B for the replacement to be possible. There are two modes of compatibility rules between interfaces:

**Strict Mode:**

* All messages in interface A must appear in interface B, and the schema of messages in interface A must be compatible with the schema of corresponding messages in interface B.

**Relaxed Mode:**

* Only messages in interface A that have connections must appear in interface B, and the schemas of these messages in interface A must be compatible with the schemas of messages with the same names in interface B.

Strict mode and relaxed mode are user-selectable, suitable for different scenarios. For example, if it's certain that a specific message X in interface A implemented by an extension will never be used, then being unable to replace with another desired extension due to message X's incompatibility or non-existence might be too strict.

## Implementation of Standard Interfaces

The APIs supported by TEN extensions are defined in their `manifest.json` files. The implementation of standard interfaces involves writing the defined API specifications into independent JSON files and then referencing these files in `manifest.json`.

```json
{
  "type": "extension",
  "name": "foo_extension",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.1"
    }
  ],
  "api": {
    "interface": [
      {
        "import_uri": "https://xxx/asr_interface.json"  // <= Reference to standard interface definition file
      },
      {
        "import_uri": "https://xxx/tts_interface.json"  // <= Reference to standard interface definition file
      }
    ],
    "property": {
      "bar": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "query_vector",
        "property": {
          "collection_name": {
            "type": "string"
          },
          "top_k": {
            "type": "int64"
          },
          "embedding": {
            "type": "array",
            "items": {
              "type": "float64"
            }
          }
        },
        "required": [
          "collection_name",
          "top_k",
          "embedding"
        ],
        "result": {
          "property": {
            "response": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "content": {
                    "type": "string"
                  },
                  "score": {
                    "type": "float64"
                  }
                }
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "publish",
        "property": {
          "message": {
            "type": "buf"
          }
        }
      },
      {
        "name": "set_presence_state"
      }
    ],
    "data_in": [
      {
        "name": "data"
      }
    ]
  }
}
```

### Example of Standard Interface Definition File

When TEN systems (including TEN Runtime, TEN Manager, etc.) parse `manifest.json`, they download the corresponding JSON files based on `import_uri` and merge their contents into `manifest.json`. The example content of the referenced standard interface file is shown below. As you can see, the content is the same as the `api` field in manifest.json, just separated into an independent file for easy reference. This independent standard interface file can also reference other standard interface files.

```json
{
  "interface": [
    {
      "import_uri": "https://xxx/stt_interface.json"// <= Reference to other standard interface definition files
    }
  ],
  "property": {
    "bar": {
      "type": "string"
    }
  },
  "cmd_in": [
    {
      "name": "query_vector",
      "property": {
        "collection_name": {
          "type": "string"
        },
        "top_k": {
          "type": "int64"
        },
        "embedding": {
          "type": "array",
          "items": {
            "type": "float64"
          }
        }
      },
      "required": [
        "collection_name",
        "top_k",
        "embedding"
      ],
      "result": {
        "property": {
          "response": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string"
                },
                "score": {
                  "type": "float64"
                }
              }
            }
          }
        }
      }
    }
  ],
  "cmd_out": [
    {
      "name": "publish",
      "property": {
        "message": {
          "type": "buf"
        }
      }
    },
    {
      "name": "set_presence_state"
    }
  ],
  "data_in": [
    {
      "name": "data"
    }
  ]
}
```

### Interface Merging Rules

The system merges the content of JSON files pointed to by `import_uri` into the `api` field of `manifest.json`, specifically including:

* `property`
* `cmd_in`, `cmd_out`
* `data_in`, `data_out`
* `audio_frame_in`, `audio_frame_out`
* `video_frame_in`, `video_frame_out`

**Important Rules:**

1. **Conflict Detection**: If fields of the same name and type are found during merging (such as `cmd_in` or `property` with the same name), it's not considered an error if the schemas are identical, otherwise the system will report an error.
2. **Recursive Merging**: The merging process is recursive, and the system will report an error if circular references are detected.
3. **Multiple Interface Support**: An extension can support multiple standard interfaces by defining multiple `interface` fields in `manifest.json`.
4. **Duplicate Reference Detection**: If different `interface` fields reference the same `import_uri`, the system will report an error.

## Storage Methods for Standard Interfaces

Standard interface files support the following storage methods:

<div className="fd-steps">
  <div className="fd-step">
    ### TEN System Package

    **Overview**: This method packages standard interface definition files into TEN system packages and then references them in `manifest.json`. You can also depend on this system package in the dependencies field of manifest.json, so that when installing the extension, the dependent standard interface system package will be installed together, providing the best development experience. Through this method, you can use TEN package's version mechanism to implement version management of standard interfaces, and use TEN package's dependency mechanism to implement automatic installation of standard interfaces.

    Using TEN system packages, in addition to standard interface JSON definition files, can also include related resources such as model files, common utility function libraries, supporting tools, etc. Therefore, when any of these resources have incompatible changes, they can be better reflected through TEN system package version number changes. For example, changes in the storage location or file names of JSON files within the system package are also considered breaking changes to the system package, so the major version number of the system package should be upgraded.

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "0.1.0"
        }
      ],
      "api": {
        "interface": [
          {
            "import_uri": "../../../ten_packages/system/foo_interface/interface.json"
          }
        ]
      }
    }
    ```

    **Advantages of Using TEN System Packages:**

    1. **Easy Distribution**: System packages can be uploaded to the TEN extension marketplace for users to download directly
    2. **Rich Resources**: The package not only contains interface definition files but can also include related resources such as:
       * Model files
       * Common utility function libraries
       * Supporting tools, etc.
    3. **Dependency Management**: System packages can serve as dependencies of extensions. When extensions are downloaded, related system packages will also be automatically downloaded
    4. **Version Management**: Not limited to standard interface protocols themselves, any resources within the system package are part of the system package's version management

    It should be noted that the version number of TEN system packages does not only represent the version number of standard interfaces, but represents the version number of the entire system package. Therefore, if resources within the system package have breaking changes, the major version number of the system package should be upgraded. For example, changes in the storage location or file names of JSON files within the system package, even if the content of JSON files hasn't changed, are considered breaking changes to the system package because the usage method needs to change for users, so the major version number of the system package should be upgraded.

    Therefore, you cannot judge the compatibility of standard interfaces solely based on the version number of system packages. The compatibility of standard interfaces still needs to be judged through interface compatibility rules.

    #### Support for Multi-Version Coexistence

    > **Note**: The functionality described in this section will basically not be implemented. It only describes possible implementation methods and considerations if it were to be done in the future.

    **Overview**: In some scenarios, a TEN application might need to use different versions of the same standard interface simultaneously. To support this requirement, TEN allows installing multiple versions of the same system package within the same application.

    **System Package Configuration:**

    First, declare support for multi-version coexistence in the system package's `manifest.json`:

    ```json
    {
      "type": "system",
      "name": "foo_interface",
      "version": "0.1.0",
      "install_policy": {
        "coinstallable": "true"
      }
    }
    ```

    **Extension Dependency Configuration:**

    In the `manifest.json` of extensions that depend on this system package, you also need to declare the use of multi-version coexistence:

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "=0.1.0", // Note: Only equal syntax can be used here, not >= syntax
          "install_policy": {
            "coinstallable": "true"
          }
        }
      ],
    }
    ```

    And during the entire dependency tree construction process, if a system package's coinstallable is true, then all places that depend on this system package need to declare coinstallable, otherwise the system will report an error.

    **File System Path:**

    After enabling multi-version coexistence, the path of system packages in the file system becomes:

    ```text
    ten_packages/system/<hash_value>_foo_extension_0_1_0/
    ```

    > **Note**: A mapping mechanism might be introduced in the future to allow users not to explicitly specify hash\_value during usage, but to automatically find corresponding system packages through some mapping mechanism. However, for this mechanism to work well, it would also require modifying the module loading mechanisms in languages supported by TEN, so that developers don't need to specify hash\_value in the module loading mechanisms of those languages, but can automatically find corresponding system packages through some mapping mechanism. This kind of deep language-internal modification is also one of the reasons why this single-package multi-version coexistence mechanism will basically not be implemented.
    >
    > **Note**: If there's really a need to use multiple versions of a standard interface within a TEN application, you can use the several methods described below. If a standard interface has accompanying common resources such as model files, common utility function libraries, supporting tools, etc., then many of these resources cannot achieve multi-version coexistence within a TEN application, which is also one of the reasons why this single-package multi-version coexistence mechanism will basically not be implemented.
  </div>

  <div className="fd-step">
    ### Network Address

    **Overview**: Deploy standard interface definition files to network servers and reference them through HTTP/HTTPS addresses:

    ```json
    {
      "interface": [
        {
          "import_uri": "http://xxx/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    > **Note**: When TEN Manager handles this type of reference, it can download files to its own cache and then use some mapping mechanism to allow TEN Manager to find the definitions.

    **Applicable Scenarios:**

    * **Lightweight Interfaces**: Interface definitions without accompanying common resources
    * **Version Flexibility**: Need to reference different versions of the same standard interface in one `manifest.json`
  </div>

  <div className="fd-step">
    ### Local Path

    **Overview**: Reference standard interface definition files in the local file system:

    **Absolute Path:**

    ```json
    {
      "interface": [
        {
          "import_uri": "file://c:/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **Relative Path:**

    ```json
    {
      "interface": [
        {
          "import_uri": "../asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **Applicable Scenarios:**

    Similar to the network address method:

    * **Lightweight Interfaces**: Suitable for interface definitions without accompanying common resources
    * **Version Flexibility**: Need to reference different versions of the same standard interface in one `manifest.json`
  </div>
</div>

## How to Filter Desired Extensions from the Cloud Store

The relationship between standard interfaces and the cloud store is reflected in packaging standard interfaces into TEN system packages. There are two ways to search for extensions that meet specific conditions:

1. **Dependency Search**: Search for extensions that depend on a certain system package. Since the metadata of extensions in the cloud store contains dependency information, the cloud store can filter out all extensions that depend on a certain system package accordingly.

2. **Tag Search**: Quickly search for extensions that meet certain conditions through TEN package tags, select and install them, then perform replacement operations. If they are TEN officially certified extensions, they can even be tagged with `ten:` prefix to indicate that this is an officially certified extension that indeed conforms to the declared standard interface definition.


# Check-Graph (/docs/ten_framework/0.11.59/ten_manager/check_graph)

`tman` provides the `check graph` command to validate predefined graphs or a `start_graph` command for correctness. To see the usage details, use the following command:

```shell
$ tman check graph -h
Check whether the graph content of the predefined graph or start_graph command is correct. For more detailed usage, run 'graph -h'

Usage: tman check graph [OPTIONS] --app <APP>

Options:
      --app <APP>
          The absolute path of the app defined in the graph. By default, the predefined graph will be read from the first one in the list.
      --predefined-graph-name <PREDEFINED_GRAPH_NAME>
          Specify a predefined graph name to check, otherwise, all predefined graphs will be checked.
      --graph <GRAPH>
          Specify the JSON string of a 'start_graph' command to be checked. If not specified, the predefined graph in the first app will be checked.
  -h, --help
          Print help
```

The `check graph` command is designed to handle graphs that may span multiple apps, so it does not require being run from the root directory of a specific app. Instead, it can be executed from any directory, with the `--app` parameter used to specify the folders of the apps involved in the graph.

## Example Usages

* **Check all predefined graphs in `property.json`**:

  ```shell
  tman check graph --app /home/TEN-Agent/agents
  ```

* **Check a specific predefined graph**:

  ```shell
  tman check graph --predefined-graph-name va.openai.azure --app /home/TEN-Agent/agents
  ```

* **Check a `start_graph` command**:

  ```shell
  tman check graph --graph '{
    "type": "start_graph",
    "seq_id": "55",
    "nodes": [
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_2__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      },
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_1__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      }
    ]
  }' --app /home/TEN-Agent/agents
  ```

## Prerequisites

* **Predefined Graph Definition Requirement**: If a predefined graph name is specified, the definition will be extracted from the `property.json` file of the app specified by the first `--app` parameter.

* **Package Installation Requirement**: Before running the `check graph` command, all extensions that the app depends on must be installed using the `tman install` command. This is necessary because the validation process requires information about each extension in the graph, such as APIs defined in their `manifest.json` files.

* **Unique App URI Requirement**: In a multi-app graph, each app's `property.json` must define a unique `ten::uri`. Additionally, the `uri` value cannot be set to `"localhost"`.

## Validation Rules

<div className="fd-steps">
  <div className="fd-step">
    ### Presence of Nodes

    The `nodes` array is required in any graph definition. If absent, an error will be thrown.

    * **Example (No nodes defined)**:

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": []
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        No extension node is defined in graph.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### Uniqueness of Nodes

    Each node in the `nodes` array represents a specific extension instance within a group of an app, created by a specified addon. Therefore, each extension instance should be uniquely represented by a single node. A node must be uniquely identified by the combination of `app`, `extension_group`, and `name`. Multiple entries for the same extension instance are not allowed.

    * **Example (Duplicate nodes)**:

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": [
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_2__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          },
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_1__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        Duplicated extension was found in nodes[1], addon: basic_hello_world_1__test_extension, name: test_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### Extensions used in connections should be defined in nodes

    All extension instances referenced in the `connections` field, whether as a source or destination, must be explicitly defined in the `nodes` field. Any instance not defined in the `nodes` array will cause validation errors.

    * **Example (Source extension not defined)**:

      Imagine that the content of `property.json` of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0] is not defined in nodes, extension_group: producer, extension: some_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

    * **Example (Destination extension not defined)**:

      Imagine that the content of `property.json` of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                },
                {
                  "type": "extension",
                  "name": "some_extension_1",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        }
                      ]
                    },
                    {
                      "name": "world",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        },
                        {
                          "extension": "consumer"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0].cmd[1] is not defined in nodes extension_group: some_group, extension: consumer.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The addons declared in the `nodes` must be installed in the app

    * **Example (The `ten::uri` in property.json is not equal to the `app` field in nodes)**:

      Imagine that the content of property.json of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ],
          "uri": "http://localhost:8001"
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

      The problem is all packages in the app will be stored in a map which key is the `uri` of the app, and each node in the graph is retrieved by the `app` field (which is `localhost` by default). The `app` in node (i.e., localhost) is mismatch with the `uri` of app (i.e., [http://localhost:8001](http://localhost:8001)).

    * **Example (the ten\_packages does not exist as the `tman install` has not executed)**:

      Imagine that the content of property.json of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ]
        }
      }
      ```

      And the `ten_packages` directory does **\_NOT** exist.

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### In connections, messages sent from one extension should be defined in the same section

    * **Example**:

      Imagine that all the packages have been installed.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          },
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello_2",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        extension 'some_extension' is defined in connection[0] and connection[1], merge them into one section.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### In connections, the messages sent out from one extension should have a unique name in each type

    * **Example**:

      Imagine that all the packages have been installed.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              },
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "some_extension"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        - connection[0]:
         - Merge the following cmd into one section:
            'hello' is defined in flow[0] and flow[1].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The messages declared in the connections should be compatible

    The message declared in each message flow in the connections will be checked if the schema is compatible, according to the schema definition in the manifest.json of extensions. The rules of message compatible are as follows.

    * If the message schema is not found neither in the source extension nor the target extension, the message is compatible.

    * If the message schema is only found in one of the extensions, the message is incompatible.

    * The message is compatible only if the following conditions are met.

      * The property type is compatible.
        * If the property is an object, the fields both in the source and target schema must have a compatible type.
        * If the `required` keyword is defined in the target schema, there must be a `required` keyword in the source schema, which is the superset of the `required` in the target.

    * **Example**:

      Imagine that all the packages have been installed.

      The content of manifest.json in `addon_a` is as follows.

      ```json
      {
        "type": "extension",
        "name": "addon_a",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_out": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "string"
                }
              }
            }
          ]
        }
      }
      ```

      And, the content of manifest.json in `addon_b` is as follows.

      ```json
      {
        "type": "extension",
        "name": "addon_b",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_in": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "int8"
                }
              }
            }
          ]
        }
      }
      ```

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        - connections[0]:
          - cmd[0]:  Schema incompatible to [extension_group: some_group, extension: another_ext], { .foo: type is incompatible, source is [string], but target is [int8] }

      All is done.
      ❌ Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The `app` in node must be unambiguous

    The `app` field in each node must met the following rules.

    * The `app` field must be equal to the `ten::uri` of the corresponding TEN app.

    * Either all nodes should have `app` declared, or none should.

    * The `app` field can not be `localhost`.

    * The `app` field can not be an empty string.

    * **Example (some of the nodes specified the `app` field)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        Either all nodes should have 'app' declared, or none should, but not a mix of both.
      ```

    * **Example (No `app` specified in all nodes, but some source extension specified `app` in the connections)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
         {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].the 'app' should not be declared, as not any node has declared it
      ```

    * **Example (No `app` specified in all nodes, but some target extension specified `app` in the connections)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext",
                    "app": "http://localhost:8000"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].cmd[0].dest[0]: the 'app' should not be declared, as not any node has declared it
      ```

    * **Example (The `app` field in nodes is not equal to the `ten::uri` of app)**:

      Same as [Rule 4](#id-4.-the-addons-declared-in-the-nodes-must-be-installed-in-the-app).

    * **Example (The `app` field is `localhost` in a single-app graph)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, and the graph seems to be a single-app graph, just remove the 'app' field
      ```

    * **Example (The `app` field is `localhost` in a multi-app graph)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, change the content of 'app' field to be consistent with 'ten::uri'
      ```
  </div>
</div>


# Overview (/docs/ten_framework/0.11.59/ten_manager/overview)

In the TEN ecosystem, almost everything is a component. For example, a TEN app is a type of component, a TEN extension is another type, and a TEN protocol is yet another. To assist TEN developers in managing all operations related to these components, there is a crucial tool called the TEN Manager.

<Callout type="info">
  `tman`

   is the abbreviation for TEN Manager.
</Callout>

The TEN Manager includes a TEN package manager, designed to simplify the management of TEN packages. This package manager handles tasks such as determining which dependency packages are necessary, fetching those dependencies, and installing them. In short, the TEN Manager streamlines the process of creating new TEN packages from scratch and makes it easier to install TEN packages from the TEN Cloud Store.


# Debugging (/docs/ten_framework/0.11.59/testing_and_debugging/debugging)

## For Developers of the TEN Framework

If you're using VS Code to develop the TEN framework, there is a `launch.json` file in the TEN framework source tree that provides default debug targets.

## For Users of the TEN Framework

If you're using VS Code to develop your own application based on the TEN framework, you can add configurations to your `.vscode/launch.json` file so that you can debug your application with breakpoints and variable inspection whichever programming language (C++/Golang/Python) you are using.

### Debugging in C++ Applications

If the application is written in C++, it means the extensions can be written
in C++ or Python.

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

* **Debugging C++ code with lldb**

  ```json
  {
    "name": "app (C++) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    }
  }
  ```

* **Debugging C++ code with gdb**

  ```json
  {
    "name": "app (C++) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### Debugging Python code with debugpy

You can use `debugpy` to debug Python code. However, since the program is not started directly through the Python interpreter, the Python code is executed by the embedded Python interpreter. Therefore, you can only attach a debugger to a running process.

First, you need to install `debugpy` in the Python environment:

```shell
pip install debugpy
```

Then, you need to start the application with environment variable `TEN_ENABLE_PYTHON_DEBUG` and `TEN_PYTHON_DEBUG_PORT` set. For example:

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

If the environment variable `TEN_ENABLE_PYTHON_DEBUG` is set to `true`, then the application will block until the debugger is attached. If the environment variable `TEN_PYTHON_DEBUG_PORT` is set, then the debugger server will listen on the specified port for incoming connections.

Then, you can attach the debugger to the running process. Here is an example of the configuration:

**Debugging Python code with debugpy**

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### Debugging C++ and Python code at the same time

If you want to start debugging C++ and Python code at the same time with one click in VSCode, you can do the following:

1. Define a task for delaying several seconds before attaching the debugger:

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. Add the following configurations to the `launch.json` file:

   ```json
   "configurations": [
        {
            "name": "app (C++) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch", // "launch" or "attach"
            "program": "${workspaceFolder}/bin/worker", // The executable path
            "cwd": "${workspaceFolder}/",
            "env": {
                "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
                "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (C++) (debugpy, attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "justMyCode": false,
            "preLaunchTask": "delay 3 seconds"
        }
   ],
   "compounds": [
        {
            "name": "app (C++) (lldb, launch) and app (C++) (debugpy, attach)",
            "configurations": [
                "app (C++) (lldb, launch with debugpy)",
                "app (C++) (debugpy, attach with delay)"
            ]
        }
   ]
   ```

Then, you can start debugging C++ and Python code with the compound configuration `app (C++) (lldb, launch) and app (C++) (debugpy, attach)`.

### Debugging in Go applications

If the application is written in Go, it means the extensions can be written in Go or C++ or Python.

#### Debugging Go code with delve

You can use the following configuration to debug in your Go extensions:

```json
{
  "name": "app (golang) (go, launch)",
  "type": "go",
  "request": "launch",
  "mode": "exec",
  "cwd": "${workspaceFolder}/",
  "program": "${workspaceFolder}/bin/worker", // The executable path
  "env": {
    "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
    "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
    "TEN_APP_BASE_DIR": "${workspaceFolder}"
  }
}
```

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

* **debugging C++ code with lldb**

  ```json
  {
    "name": "app (Go) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    },
    "initCommands": ["process handle SIGURG --stop false --pass true"]
  }
  ```

* **debugging C++ code with gdb**

  ```json
  {
    "name": "app (Go) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### Debugging Python code with debugpy

Refer to the section above for debugging Python code with debugpy. Start the application with environment variable `TEN_ENABLE_PYTHON_DEBUG` and `TEN_PYTHON_DEBUG_PORT` set.

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

Then, you can attach the debugger to the running process. Here is an example of the configuration:

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### Debugging C++, Go and Python code at the same time

If you want to start debugging C++, Go and Python code at the same time with one click in VSCode, you can do the following:

1. Define a task for delaying several seconds before attaching the debugger:

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. Add the following configurations to the `launch.json` file:

   ```json
   "configurations": [
        {
            "name": "app (golang) (debugpy, remote attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "delay 3 seconds",
            "justMyCode": false
        },
        {
            "name": "app (golang) (go, launch with debugpy)",
            "type": "go",
            "request": "launch",
            "mode": "exec",
            "cwd": "${workspaceFolder}/",
            "program": "${workspaceFolder}/bin/worker",
            "env": {
                "TEN_APP_BASE_DIR": "${workspaceFolder}",
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (golang) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/bin/worker",
            "cwd": "${workspaceFolder}/",
            "env": {
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            },
            "initCommands": [
                "process handle SIGURG --stop false --pass true"
            ]
        },
   ],
   "compounds": [
        {
            "name": "Mixed Go/Python",
            "configurations": [
                "app (golang) (go, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ],
        },
        {
            "name": "Mixed Go/Python/C++ (lldb)",
            "configurations": [
                "app (golang) (lldb, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ]
        }
   ]
   ```

If you want to debug C++ and Python code at the same time, you can use the compound configuration `Mixed Go/Python/C++ (lldb)`. With this configuration, you can perform breakpoint debugging for C++, Go and Python code simultaneously. However, the variable inspection is only supported for C++ and Python but not for Go code.

If you want to debug Go and Python code at the same time, you can use the compound configuration `Mixed Go/Python`. With this configuration, you can perform breakpoint debugging for Go and Python code simultaneously and inspect variables for both Go and Python code.

### Debugging in Python applications

If the application is written in Python, it means the extensions can be written in Python or C++.

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

```json
{
  "name": "app (Python) (cpp, launch)",
  "type": "cppdbg", //
  "request": "launch", // "launch" or "attach"
  "program": "/usr/bin/python3", // The Python interpreter path
  "args": ["main.py"],
  "cwd": "${workspaceFolder}/",
  "environment": [
    {
      "name": "PYTHONPATH",
      "value": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface"
    },
    {
      "name": "TEN_APP_BASE_DIR",
      "value": "${workspaceFolder}/"
    }
  ],
  "MIMode": "gdb" // "gdb" or "lldb"
}
```

#### Debugging Python code with debugpy

Refer to the section above for debugging Python code with debugpy. You can use
the following configurations to debug Python code:

```json
{
  "name": "app (Python) (debugpy, launch)",
  "type": "debugpy",
  "python": "/usr/bin/python3",
  "request": "launch",
  "program": "${workspaceFolder}/main.py",
  "console": "integratedTerminal",
  "cwd": "${workspaceFolder}/",
  "env": {
    "PYTHONPATH": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface",
    "TEN_APP_BASE_DIR": "${workspaceFolder}/",
    "TEN_ENABLE_PYTHON_DEBUG": "true"
  }
}
```

#### Debugging C++ and Python code at the same time

If you want to start debugging C++ and Python code at the same time, you can launch the application with the configuration `app (Python) (debugpy, launch)` and attach the debugger to the running process with the following configuration:

```json
{
  "name": "app (Python) (cpp, attach)",
  "type": "cppdbg",
  "request": "attach", // "launch" or "attach"
  "program": "${workspaceFolder}/bin/worker", // The executable path
  "cwd": "${workspaceFolder}/",
  "MIMode": "gdb",
  "environment": [
    {
      "name": "LD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    },
    {
      "name": "DYLD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    }
  ]
}
```


# For Developers of TEN framework (/docs/ten_framework/0.11.59/testing_and_debugging/for_developers_of_ten_framework)

The TEN framework includes three types of test suites:

* Unit Tests
* Smoke Tests
* Integration Tests

The TEN framework uses `gtest` as the testing framework for unit tests and smoke tests, and `pytest` as the testing framework for integration tests.

## Unit Tests

The source directory for unit tests is `tests/ten_runtime/unit`. If you need to add new unit test cases, please place them in the `tests/ten_runtime/unit directory`.

For C++ projects, after successful compilation, you can navigate to the output directory (e.g., `out/linux/x64/tests/standalone/`, depending on the target OS and CPU) and execute the following command:

```shell
./ten_runtime_unit_test
```

## Smoke Tests

The source directory for smoke tests is `tests/ten_runtime/smoke`. If you need to add new smoke test cases, please place them in the `tests/ten_runtime/smoke` directory.

Navigate to the output directory (e.g., `out/linux/x64/tests/standalone`, depending on the target OS and CPU) and execute the following command:

```shell
./ten_runtime_smoke_test
```

## Loop Testing

You can perform multiple rounds of testing using the following commands.

To run only unit tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;
```

To run only smoke tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

To run both unit tests and smoke tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

## Integration Tests

Integration tests are black-box tests for the TEN framework. The source directory for integration tests is `tests/ten_runtime/integration`. These tests are used to validate real-world execution scenarios.

The file structure for integration tests is as follows:

```text
tests/ten_runtime/integration/
  ├── test_1/
  │    ├── test_case.py
  │    └── ...
  ├── test_2/
  │    ├── test_case.py
  │    └── ...
  └── ...
```

To execute the integration tests, use the following command:

```shell
cd path/to/out/
pytest tests/ten_runtime/integration/
```

## Testing Tricks

### Linux

On Linux, you can use the following technique to slow down the execution of TEN, which can help in triggering timing-related bugs.

```shell
sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...
```

* `taskset 0x1`: Use only 1 CPU core to run the program.
* `taskset 0x3`: Use only 2 CPU cores to run the program.


# For Users of the TEN Framework (/docs/ten_framework/0.11.59/testing_and_debugging/for_users_of_ten_framework)

## Standalone Testing of Extension

The TEN framework provides an independent extension testing mechanism. This mechanism allows developers to test extensions without relying on other TEN concepts, such as other TEN extensions, TEN graphs, or TEN apps. When developers need to test the behavior of an extension without running the entire TEN app, this independent extension testing mechanism is especially useful.

The standalone testing framework for TEN extensions follows two key principles:

1. Compatibility with any native testing framework of the used language.

   For example, if a TEN extension is developed using C++, the Google gtest/gmock testing framework can be used alongside the TEN extension standalone testing framework to achieve independent testing of the C++ TEN extension.

2. Users don't need to modify any code in the extension under test.

   The exact same extension code used in runtime can be fully tested using this standalone testing framework.

3. The standalone testing flow for extensions developed in different languages follows the same design principles and usage methods.

   This ensures that once you learn the standalone testing concepts and methods for one language, you can quickly grasp the standalone testing concepts and methods for TEN extensions in other languages.

For users, the TEN extension standalone testing framework introduces two main concepts:

1. **extension\_tester**
2. **ten\_env\_tester**

The role of **extension\_tester** is similar to a testing driver, responsible for setting up and executing the entire testing process. **ten\_env\_tester**, on the other hand, behaves like a typical TEN extension's `ten_env` instance, enabling users to invoke functionalities within the standalone testing framework from the callback of `extension_tester`, such as sending messages to the extension under test and receiving messages from the extension.

From the API design of **extension\_tester** and **ten\_env\_tester**, you can see that they are very similar to **TEN extension** and **ten\_env**, and this is by design. The main purpose is to allow users familiar with extension development to quickly get accustomed to the standalone testing framework and then develop standalone test cases for the extension itself.

However, for testing purposes, there are inevitably APIs and features dedicated specifically to testing. To prevent these test-specific functionalities and APIs, which are not needed during actual runtime, from polluting the runtime API set, the standalone testing framework is designed not to directly use or extend the API sets of **extension** and **ten\_env**. Instead, it introduces types and API sets that are exclusive to standalone testing. This separation ensures that the types and API sets used in actual runtime and those used during testing do not interfere with each other, avoiding any negative side effects.

## Standalone Testing Framework Internal

Internally, the TEN extension standalone testing framework implicitly starts a test app, loads the extension addon to be tested (let’s call it extension A), and initializes a graph containing both the extension A to be tested and another extension (let’s call it extension B) dedicated to testing other extensions. All input and output messages of extension A are redirected to extension B, allowing users to customize inputs and outputs during the testing process and complete the actual testing work.

```mermaid
sequenceDiagram
  participant tester as Extension Tester
  participant testing as Extension Testing
  participant tested as Extension Tested

  tester->>testing: send messages
  testing->>tested: send messages

  tested-->>testing: send results
  testing-->>tester: send results
```

Basically, you can think of this testing extension hidden within the testing framework as a proxy between the extension being tested and the tester. It acts as a proxy extension within the TEN graph, facilitating the exchange of messages between the tested extension and the tester using the language of the TEN environment.

## Basic Testing Process

The basic testing process and logic are as follows:

1. Create an extension tester to manage the entire standalone testing process.
2. Inform the standalone testing framework of the folder containing the extension to be tested.
3. Set a testing mode, such as a mode for testing a single extension.
4. Start the testing.

## C++

Here is an example of TEN extension standalone testing using Google gtest:

```c++
class extension_tester_basic : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    auto new_cmd = ten::cmd_t::create("hello_world");
    ten_env.send_cmd(std::move(new_cmd),
                     [](ten::ten_env_tester_t &ten_env,
                        std::unique_ptr<ten::cmd_result_t> result) {
                       if (result->get_status_code() == TEN_STATUS_CODE_OK) {
                         ten_env.stop_test();
                       }
                     });
  }
};

TEST(Test, Basic) {
  // 1. Create an extension tester to manage the entire standalone testing process.
  auto *tester = new extension_tester_basic();

  // 2. Set a testing mode, such as a mode for testing a single extension.
  tester->set_test_mode_single("default_extension_cpp");

  // 3. Start the testing.
  tester->run();

  delete tester;
}
```

## Golang

TODO: To be added.

## Python

```python
class ExtensionTesterBasic(ExtensionTester):
    def check_hello(self, ten_env: TenEnvTester, result: CmdResult):
        statusCode = result.get_status_code()
        print("receive hello_world, status:" + str(statusCode))

        if statusCode == StatusCode.OK:
            ten_env.stop_test()

    def on_start(self, ten_env: TenEnvTester) -> None:
        new_cmd = Cmd.create("hello_world")

        print("send hello_world")
        ten_env.send_cmd(
            new_cmd,
            lambda ten_env, result: self.check_hello(ten_env, result),
        )

        print("tester on_start_done")
        ten_env.on_start_done()


def test_basic():
    # 1. Create an extension tester to manage the entire standalone testing process.
    tester = ExtensionTesterBasic()

    # 2. Set a testing mode, such as a mode for testing a single extension.
    tester.set_test_mode_single("default_extension_python")

    # 3. Start the testing.
    tester.run()
```


# How to Debug with Logs (/docs/ten_framework/0.11.59/testing_and_debugging/how_to_debug_with_logs)

We will introduce how to view logs, as they are now ported to stdout, making them very easy to view.

For example, if a TEN Agent is running on localhost:3000, you might see output similar to the following in the logs:

```bash
...
[GIN] 2024/09/29 - 09:29:42 | 200 |     634.458µs |    192.168.65.1 | POST     "/ping"
[agora_5bnxv1] 09-29 09:29:42.895  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ How about you?], text_recv_ts: 1727602181374572, read_bytes: 37200, sent_frames: 117, first_frame_latency: 306ms, finish_latency: 428ms, synthesis_first_byte_latency: 300ms, synthesis_finish_latency: 332ms
[agora_5bnxv1] 09-29 09:29:43.350  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Anything exciting happening?], text_recv_ts: 1727602181383864, read_bytes: 63600, sent_frames: 199, first_frame_latency: 300ms, finish_latency: 452ms, synthesis_first_byte_latency: 295ms, synthesis_finish_latency: 332ms
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_5bnxv1 requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
2024/09/29 09:29:45 INFO handlerPing end worker="&{ChannelName:agora_5bnxv1 HttpServerPort:10024 LogFile:/tmp/astra/app-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.json Pid:7455 QuitTimeoutSeconds:60 CreateTs:1727602125 UpdateTs:1727602185}" requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |    1.038208ms |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
2024/09/29 09:29:45 ERROR handlerPing channel not existed channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |     442.375µs |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:48 INFO handlerPing start channelName=agora_5bnxv1 requestId=6fb07d85-
...
```

Once you interact with the TEN Agent, you should see entries like the following:

```bash title="Bash"
[agora_5bnxv1] 09-29 09:35:25.049  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ and the World Cup in Russia.], text_recv_ts: 1727602522313774, read_bytes: 64000, sent_frames: 200, first_frame_latency: 418ms, finish_latency: 568ms, synthesis_first_byte_latency: 413ms, synthesis_finish_latency: 458ms
[agora_5bnxv1] 09-29 09:35:25.570  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ In tech,], text_recv_ts: 1727602522347918, read_bytes: 29200, sent_frames: 92, first_frame_latency: 427ms, finish_latency: 515ms, synthesis_first_byte_latency: 419ms, synthesis_finish_latency: 460ms
[agora_5bnxv1] 09-29 09:35:26.502  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ we saw some big advancements like the rise of AI and electric cars becoming more mainstream.], text_recv_ts: 1727602522461866, read_bytes: 178800, sent_frames: 559, first_frame_latency: 510ms, finish_latency: 928ms, synthesis_first_byte_latency: 503ms, synthesis_finish_latency: 800ms
[agora_5bnxv1] 09-29 09:35:27.029  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Personally,], text_recv_ts: 1727602522468652, read_bytes: 34400, sent_frames: 108, first_frame_latency: 442ms, finish_latency: 524ms, synthesis_first_byte_latency: 436ms, synthesis_finish_latency: 463ms
```

When you see logs like these, it means the system is working properly and is logging every sentence you speak.


# Doc Window (/docs/ten_framework/0.11.59/tman_designer/doc_window)

TMAN Designer has a built-in documentation window that allows you to view TEN Framework concepts and terminology. For example, you can select `About App` from the App dropdown menu to open the documentation window and display the App's description.

![TMAN Designer Doc Window](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_doc_window.png)


# Overview (/docs/ten_framework/0.11.59/tman_designer/overview)

TEN Framework offers high flexibility and provides rich functionalities, including TEN Cloud Store, package management, multi-language support, and multi-platform support. However, these features may require a longer learning period and higher technical threshold to master TEN Framework for AI Agent development and customization. To facilitate developers' use, TEN Framework provides a command-line tool called TEN Manager, which enables developers to easily modify and customize AI Agents built with TEN Framework.

Since command-line tools are not as user-friendly as graphical user interfaces, TEN Manager includes a web UI-based visual development interface called TMAN Designer, in addition to the command-line interface. It provides rich functionality to help developers develop and debug TEN Apps and TEN Extensions. This means that developers only need to obtain the TEN Manager tool to access a complete TEN Framework development environment with both command-line and graphical user interfaces.

You can get TMAN Designer from the following link:

```text
https://github.com/TEN-framework/ten-framework/releases
```

TMAN Designer supports the following operating systems and architectures. Please obtain the appropriate version for your platform:

* Windows
  * x86\_64
* Linux
  * x86\_64
* MacOS
  * x86\_64
  * arm64

After downloading and extracting, use the following command to start TMAN Designer:

```shell
tman designer
```

You can execute this command in the root directory of any TEN App, or in any location. If executed in the root directory of a TEN App, TMAN Designer will load that TEN App by default. You can also manually load any TEN App in TMAN Designer. Therefore, starting TMAN Designer in the root directory of a specific TEN App is not mandatory, but it simplifies the initial step of manually loading a TEN App.

After startup, you will see information like the following, indicating that TMAN Designer starts on port 49483 by default:

```text
🏆  Starting server at http://0.0.0.0:49483
```

You can interact with TMAN Designer using the following URL:

```text
http://127.0.0.1:49483/
```

> Note: If your TMAN Designer is running on a remote machine, please use the IP address of the remote machine to access TMAN Designer.

## Overview

After opening the TMAN Designer page in your browser, you will see the following interface:

![TMAN Designer Overview](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_overview.png)

In the upper right corner of TMAN Designer, you can see several function buttons:

* The rightmost button displays the current version of TMAN Designer and automatically checks for new version releases. If an update is available, an upward arrow will be displayed. Clicking it will open the download page for the new version.
* The first button on the left is the [TEN Agent](https://agent.theten.ai/) button. Clicking it will open the TEN Agent page. TEN Agent is a complete AI Agent implementation built using TEN Framework. One of the main goals of TMAN Designer is to enable developers to easily develop, debug, and customize TEN Agent through TMAN Designer.
* The second button on the left is the [TEN Framework](https://github.com/TEN-framework/ten-framework) GitHub page. Clicking it will open the GitHub repository for TEN Framework. You can visit this page to learn more about TEN Framework, view source code, or submit issues.
* On the far left, there are two buttons for switching between light and dark modes and selecting languages.

In the upper left corner of TMAN Designer, there are several dropdown menus that developers can use to access various functions of TMAN Designer.

## Working with TEN Apps

### Loading an Existing TEN App

You can load an existing TEN App using the Load App button in the App menu. After clicking the button, you will see the following interface:

![Loading an Existing TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_load_app.png)

Through this file browser dialog, developers can specify the root directory of an existing TEN App to load it.

### Managing Loaded TEN Apps

You can manage loaded TEN Apps using the Manage Loaded App(s) button in the App menu. After clicking the button, you will see the following Apps Manager interface:

![Managing Loaded TEN Apps](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_manage_loaded_apps.png)

Developers can use this dialog to:

* Unload loaded TEN Apps
* Reload specified TEN Apps
* Reload all loaded TEN Apps
* Install all dependencies for specified Apps
* Run functions for specified Apps

### Installing All Dependencies for an App

Developers can install various TEN packages in a TEN App. All TEN packages that a TEN App depends on are recorded in the `manifest.json` file. Here is an example of a `manifest.json` file:

```json
{
  "type": "app",
  "name": "aaa",
  "version": "0.8.18",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.8.18"
    },
    {
      "type": "extension",
      "name": "ffmpeg_muxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_demuxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_client",
      "version": "^0.1.0"
    },
    {
      "type": "protocol",
      "name": "msgpack",
      "version": "^0.8.18"
    }
  ],
  "scripts": {
    "start": "python3 tools/run_script.py start",
    "build": "python3 tools/run_script.py build"
  }
}
```

From the above configuration, we can see that this TEN App depends on the following TEN packages:

* `ten_runtime`: The system package of TEN Framework
* `ffmpeg_demuxer`: A TEN Extension package for decoding audio and video
* `ffmpeg_muxer`: A TEN Extension package for encoding audio and video
* `ffmpeg_client`: A TEN Extension package for processing audio and video using the other two ffmpeg extension packages
* `msgpack`: A TEN Protocol package for message serialization and deserialization using the MessagePack protocol

After completing the installation of all dependencies for the TEN App, the TEN package file structure in the system looks like this:

```text
├── ten_packages
│   ├── extension
│   │   ├── ffmpeg_client
│   │   ├── ffmpeg_demuxer
│   │   └── ffmpeg_muxer
│   ├── protocol
│   │   └── msgpack
│   └── system
│       ├── ffmpeg
│       └── ten_runtime
```

It's worth noting that even though it's not explicitly declared in the `manifest.json`, there is an additional `ffmpeg` system package in the system. This is because `ffmpeg_demuxer` and `ffmpeg_muxer` depend on the `ffmpeg` system package, and TEN Manager automatically identifies and installs these transitive dependencies.

Through the Apps Manager interface in TMAN Designer, developers can perform the Install All operation on a specified App to easily complete the installation of all dependencies. After clicking the button, you will see the following interface showing the installation progress:

![Installing All Dependencies for an App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_install_all.png)

### Installing TEN Extensions from the Cloud Store

You can open the TEN Cloud Store and install TEN Extensions to the current TEN App using the Open Extension Store button in the Graph menu. After clicking the button, you will see the following interface:

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_store.png)

In the TEN Cloud Store, developers can search for the required TEN Extensions and click the Install button to install them to the current TEN App. After installation, you can view the installed TEN Extensions in the current TEN App.

### Running a TEN App

In the Apps Manager, you can perform preset runtime operations on the specified App. After clicking the Run button in the Apps Manager, you will see the following interface:

![Running a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_run.png)

In the dialog, developers can select the preset operation to execute from the dropdown menu. These typically include the following operations:

* Run App: Start the App and display the App's output information in the TMAN Designer message window.
* Build App: Compile the App. Since TEN Framework supports multiple programming languages, some languages need to be compiled while others do not. For example, TEN Apps developed in C++ need to be compiled, while TEN Apps developed in Python do not.

Below is the build interface for a TEN App developed in C++:

![Building a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_build.png)

After the build is complete, you can continue with the startup operation for the TEN App. The system will display a window showing the TEN App's runtime logs in real-time:

![Starting a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_start.png)

## Working with TEN Graphs

### Loading an Existing TEN Graph

You can load an existing TEN Graph using the Open Existing Graph button in the Graph menu. After successful loading, you will see an interface similar to the following:

![TEN Graph](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_graph_overview.png)

Using this Graph as an example, it contains 3 TEN Extensions, namely:

* `ffmpeg_demuxer`
* `ffmpeg_muxer`
* `ffmpeg_client`

Each TEN Extension has its corresponding inputs and outputs. For example, `ffmpeg_demuxer` has the following inputs and outputs:

* 2 cmd inputs
* 1 cmd output
* 1 audio\_frame output
* 1 video\_frame output

Clicking on the number buttons for inputs and outputs opens a dialog like the following, which is used to manage input and output connections:

![TEN Message Input/Output Dialog](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_msg_info_dialog.png)

### Auto-Layout Graph

After manually adjusting the positions of TEN Extensions in the TEN Graph, developers can click the Auto Layout button in the Graph menu to automatically arrange the TEN Extensions in the TEN Graph.

### Opening the Context Menu for TEN Extensions

In the Graph, right-clicking on each TEN Extension opens the TEN Extension's context menu:

![TEN Extension Context Menu](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_context_menu.png)

In the context menu, you can open the TEN Extension's `manifest.json` file, `property.json` file, or directly start a terminal in the root directory of the TEN Extension, making it convenient for developers to modify and customize the TEN Extension:

![Opening Manifest](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_context_menu_effect.png)


# How to run local AI model in Python extension (/docs/ten_framework/0.11.60/ai_model/how_to_run_local_model_in_python_extensions)

In the TEN framework, extensions can utilize third-party AI services or run AI models locally to improve performance and reduce costs. This tutorial explains how to run a local AI model in a Python extension and how to interact with it within the extension.

## Step 1: Check Hardware Requirements

Before running an AI model locally, ensure that your hardware meets the necessary requirements. Key components to verify include:

* **CPU/GPU**: Check if the model requires specific processing power.
* **Memory**: Ensure sufficient memory to load and run the model.

Verify that your system can support the model’s demands to ensure smooth operation.

## Step 2: Install Necessary Software and Dependencies

Once your hardware is ready, install the required software and dependencies. Follow these steps:

1. **Operating System**: Ensure compatibility with your model. Most AI frameworks support Windows, macOS, and Linux, though specific versions may be required.

2. **Python Version**: Ensure compatibility with the TEN Python runtime and the model.

3. **Required Libraries**: Install necessary libraries such as:

   * TensorFlow
   * PyTorch
   * Numpy
   * vllm

   You can list the required dependencies in a `requirements.txt` file for easy installation.

4. **Download the Model**: Obtain the local version of the AI model you plan to run.

## Step 3: Implement Your Python Extension

Below is an example of how to implement a basic text generation feature using the `vllm` inference engine in a Python extension.

First, initialize the local model within the extension:

```python
from ten_runtime import (
    Extension,
    TenEnv,
    Cmd,
    CmdResult,
)
from vllm import LLM

class TextGenerationExtension(Extension):
    def on_init(self, ten_env: TenEnv) -> None:
        self.llm = LLM(model="<model_path>")
        ten_env.on_init_done()
```

Next, implement the `on_cmd` method to handle text generation based on the provided input:

```python
    def on_cmd(self, ten_env: TenEnv, cmd: Cmd) -> None:
        prompt = cmd.get_property_string("prompt")

        outputs = self.llm.generate(prompt)
        generated_text = outputs[0].outputs[0].text

        cmd_result = CmdResult.create(StatusCode.OK)
        cmd_result.set_property_string("result", generated_text)
        ten_env.return_result(cmd_result, cmd)
```

In this code, the `on_cmd` method retrieves the `prompt`, generates text using the model, and returns the generated text as the command result.

You can adapt this approach to implement other functionalities such as image recognition or speech-to-text by processing the relevant input types.

## Step 4: Unload the Model

It’s important to unload the model during extension cleanup to free resources:

```python
import gc
import torch

class TextGenerationExtension(Extension):
    ...

    def on_deinit(self, ten_env: TenEnv) -> None:
        del self.llm
        gc.collect()
        torch.cuda.empty_cache()
        torch.distributed.destroy_process_group()
        print("Successfully deleted the LLM pipeline and freed GPU memory!")
        ten_env.on_deinit_done()
```

This ensures efficient memory management, especially when working with GPU resources.

## Summary

Running a local model in a TEN Python extension is similar to native Python development. By loading and unloading the model in the appropriate extension lifecycle methods, you can easily integrate local AI models and interact with them effectively.


# API (/docs/ten_framework/0.11.60/api/api)

In the TEN Framework, the API definition of an extension is contained in the `manifest.json` file. When defining APIs, the TEN Framework does not use JSON Schema, but instead uses a JSON-like syntax to define API Schema.

## Why Not Use JSON Schema

The TEN Framework chooses not to use standard JSON Schema for the following reasons:

* **Type System Differences**: The TEN Framework's type system includes many types that JSON Schema does not have, such as `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float32`, `float64`, `buf`, `ptr`, etc.

* **Reduce Redundancy**: In the TEN Framework's API definitions, many places do not need flexible settings. For example, a message's schema must be an object type and cannot be other types. Requiring developers to explicitly write `"type": "object"` would be redundant and confusing.

Based on these reasons, the TEN Framework adopts a more natural definition approach, using JSON-like syntax to define API Schema.

## Top-level API Definition

The API Schema definition is placed in the `api` field of the `manifest.json` file:

```json
{
  "api": {
    "property": {
      "properties": {...},
      "required": [...]
    },
    "cmd_in": [{
      "name": "foo",
      "property": {
        "properties": {...},
        "required": [...]
      },
      "result": {
        "property": {
          "properties": {...},
          "required": [...]
        }
      }
    }]
  }
}
```

## API Schema Definition Patterns

### Expressing Object Type Fields

Use the `properties` field to describe the type of each property within an object type field:

```json
{
  "my_object": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "age": {
        "type": "int32"
      }
    }
  }
}
```

### Expressing Required and Optional Fields

Use a `required` field at the same level as the object type field to describe required and optional fields:

```json
{
  "some_concept": {
    "properties": {
      "foo": {},
      "bar": {}
    },
    "required": ["foo"]
  }
}
```

### Expressing Array Type Fields

Use the `items` field to describe the type of each element within an array type field:

```json
{
  "my_array": {
    "type": "array",
    "items": {
      "type": "string"
    }
  }
}
```

### Usage Principles for type Field

If a field can only be a specific type (constrained by the TEN Framework itself), there is no need to explicitly write the `type` field. If a field can be multiple types, then the `type` field needs to be explicitly written.

For example, if the `some_concept` field below can only be an object type, there is no need to explicitly write the `type` field. However, the `foo` field within it can be multiple types, so the `type` field needs to be explicitly written:

```json
{
  "some_concept": {
    "properties": {
      "foo": {
        "type": "string"
      },
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "string"
          }
        },
        "required": ["baz"]
      }
    },
    "required": ["foo"]
  }
}
```

Use the `required` field appropriately to ensure the existence of critical data:

```json
{
  "name": "critical_cmd",
  "property": {
    "properties": {
      "essential_param": {
        "type": "string"
      },
      "optional_param": {
        "type": "int32"
      }
    },
    "required": ["essential_param"]
  }
}
```

## Expressing Enum Type Fields

Use the `enum` field to describe all possible values of an enumeration type:

```json
{
  "api": {
    "property": {
      "properties": {
        "status": {
          "type": "string",
          "enum": ["CREATING", "ACTIVE", "DELETING", "FAILED"]
        }
      }
    }
  }
}
```

## Extension Property Schema

Complete example of defining extension properties:

```json
{
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        },
        "config": {
          "type": "object",
          "properties": {
            "enabled": {
              "type": "bool"
            },
            "settings": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          },
          "required": ["enabled"]
        }
      },
      "required": ["api_key", "max_tokens", "temperature", "config"]
    }
  }
}
```

## Command-like Message Schema

Commands support input (`cmd_in`) and output (`cmd_out`):

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "process_text",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            },
            "options": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "max_length": {
                  "type": "int32"
                }
              }
            }
          },
          "required": ["text"]
        },
        "result": {
          "property": {
            "properties": {
              "processed_text": {
                "type": "string"
              },
              "confidence": {
                "type": "float32"
              }
            },
            "required": ["processed_text"]
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "notify_completion",
        "property": {
          "properties": {
            "task_id": {
              "type": "string"
            },
            "status": {
              "type": "string"
            }
          },
          "required": ["task_id", "status"]
        }
      }
    ]
  }
}
```

## Data-like Message Schema

Schema definitions for data-type messages:

```json
{
  "api": {
    "data_in": [
      {
        "name": "input_data",
        "property": {
          "properties": {
            "content": {
              "type": "string"
            },
            "metadata": {
              "type": "object",
              "properties": {
                "timestamp": {
                  "type": "int64"
                }
              }
            }
          },
          "required": ["content"]
        }
      }
    ],
    "data_out": [
      {
        "name": "output_data",
        "property": {
          "properties": {
            "result": {
              "type": "string"
            }
          }
        }
      }
    ],
    "audio_frame_in": [
      {
        "name": "audio_input"
      }
    ],
    "audio_frame_out": [
      {
        "name": "audio_output"
      }
    ],
    "video_frame_in": [
      {
        "name": "video_input"
      }
    ],
    "video_frame_out": [
      {
        "name": "video_output"
      }
    ]
  }
}
```

## Common Errors and Solutions

<div className="fd-steps">
  <div className="fd-step">
    ### Property Names Not Following Conventions

    **Error Example**: Property names containing hyphens or starting with numbers

    ```json
    {
      "api-key": {  // Error: contains hyphens
        "type": "string"
      },
      "2nd_param": {  // Error: starts with a number
        "type": "int32"
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "api_key": {
        "type": "string"
      },
      "second_param": {
        "type": "int32"
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### Array Type Missing items Definition

    **Error Example**:

    ```json
    {
      "my_array": {
        "type": "array"
        // Missing items definition
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "my_array": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### Object Type Missing properties Definition

    **Error Example**:

    ```json
    {
      "my_object": {
        "type": "object"
        // Missing properties definition
      }
    }
    ```

    **Correct Example**:

    ```json
    {
      "my_object": {
        "type": "object",
        "properties": {
          "field1": {
            "type": "string"
          }
        }
      }
    }
    ```
  </div>
</div>

## Summary

The TEN Framework's Schema system provides a powerful and structured way to define and validate data structures, ensuring consistency and type safety between components. By following design principles and best practices, developers can create robust and maintainable TEN applications and extensions.

The system supports rich type definitions, complex API descriptions, and flexible message transformation capabilities, providing a solid foundation for building large-scale real-time multimodal applications.


# Log (/docs/ten_framework/0.11.60/api/log)

The TEN framework allows extensions developed in different languages to run within the same process. This creates a need to view logs from all these extensions in a unified log with a consistent format and information, making debugging easier.

To address this, the `ten_env` provides a logging API. Within each callback of an extension, an instance of `ten_env` can be accessed. Using this instance, you can call the logging API to output logs from extensions developed in various languages to a unified log output.

## Log Levels

The TEN framework supports the following log levels in increasing order of severity:

* **DEBUG** (1): Debug information
* **INFO** (2): General information
* **WARN** (3): Warning information
* **ERROR** (4): Error information

## API Interface

### C++ Language

```cpp
#include "ten_runtime/binding/cpp/ten.h"

class Extension {
public:
    void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr<ten::cmd_t> cmd) {
        // Using convenience macros
        TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
        TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
        TEN_ENV_LOG_WARN(ten_env, "Warning message");
        TEN_ENV_LOG_ERROR(ten_env, "Error occurred");

        // Direct method call with category and fields support
        ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
                   "Direct log call", nullptr, nullptr);
    }
};
```

C++ provides the following convenience macros:

* `TEN_ENV_LOG_DEBUG(ten_env, msg)`
* `TEN_ENV_LOG_INFO(ten_env, msg)`
* `TEN_ENV_LOG_WARN(ten_env, msg)`
* `TEN_ENV_LOG_ERROR(ten_env, msg)`

### Python Language

```python
from ten_runtime import Extension, TenEnv

class MyExtension(Extension):
    def on_cmd(self, ten_env: TenEnv, cmd):
        # Using different log level methods
        ten_env.log_debug("Debug information")
        ten_env.log_info("General information")
        ten_env.log_warn("Warning information")
        ten_env.log_error("Error information")

        # Supports formatted strings and additional parameters
        test_value = "example"
        ten_env.log_info(f"Processing command, test value: {test_value}")

        # Supports category and fields parameters
        ten_env.log_info("Log with category", category="my_extension")
```

Python provides the following methods:

* `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`

### Go Language

```go
package main

import "ten_runtime/ten"

type MyExtension struct {
    ten.DefaultExtension
}

func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    // Using different log level methods
    tenEnv.LogDebug("Debug information")
    tenEnv.LogInfo("General information")
    tenEnv.LogWarn("Warning information")
    tenEnv.LogError("Error information")

    // Supports formatting
    cmdName := cmd.GetName()
    tenEnv.LogInfo("Processing command: " + cmdName)

    // Using the full Log method for additional parameters
    category := "my_extension"
    tenEnv.Log(ten.LogLevelInfo, "Log with category", &category, nil, nil)
}
```

Go provides the following methods:

* `LogDebug(msg string) error`
* `LogInfo(msg string) error`
* `LogWarn(msg string) error`
* `LogError(msg string) error`
* `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`

### Node.js/TypeScript Language

```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";

class MyExtension extends Extension {
    async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
        // Using different log level methods
        tenEnv.logDebug("Debug information");
        tenEnv.logInfo("General information");
        tenEnv.logWarn("Warning information");
        tenEnv.logError("Error information");

        // Supports string concatenation
        const cmdName = cmd.getName();
        tenEnv.logInfo("Processing command: " + cmdName);

        // Supports category and fields parameters
        tenEnv.logInfo("Log with category", "my_extension");
    }
}
```

Node.js provides the following methods:

* `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
* `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
* `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
* `logError(message: string, category?: string, fields?: Value): TenError | undefined`

## Usage Examples

### Basic Usage

```python
# Python example
def on_start(self, ten_env: TenEnv):
    ten_env.log_info("Extension starting")
    ten_env.on_start_done()

def on_cmd(self, ten_env: TenEnv, cmd):
    cmd_name = cmd.get_name()
    ten_env.log_debug(f"Received command: {cmd_name}")

    # Command processing logic
    if cmd_name == "hello":
        ten_env.log_info("Processing hello command", category="command_handler")
        # ... processing logic
    else:
        ten_env.log_warn(f"Unknown command: {cmd_name}")
```

### Using in Threads

```python
import threading
import time

def log_routine(self, ten_env: TenEnv):
    """Log messages in a separate thread"""
    while not self.stop_flag:
        self.log_count += 1
        ten_env.log_info(f"Log message {self.log_count}")
        time.sleep(0.05)

def on_start(self, ten_env: TenEnv):
    # Start logging thread
    self.log_thread = threading.Thread(
        target=self.log_routine, args=(ten_env,)
    )
    self.log_thread.start()
    ten_env.on_start_done()
```

### Error Handling

```python
def on_cmd(self, ten_env: TenEnv, cmd):
    try:
        # Business logic
        result = self.process_command(cmd)
        ten_env.log_info("Command processed successfully")
    except ValueError as e:
        ten_env.log_error(f"Parameter error: {str(e)}", category="error_handling")
    except Exception as e:
        ten_env.log_error(f"Fatal error: {str(e)}", category="error_handling")
        raise
```

## Log Format and Output

The TEN framework's logging system automatically includes the following information:

* **Timestamp**: Time when the log was recorded
* **Log Level**: DEBUG, INFO, WARN, ERROR
* **Function Name**: Name of the function that called the log (if available)
* **File Name**: Source file name where the log was called (if available)
* **Line Number**: Code line number where the log was called (if available)
* **Category**: Optional log category information
* **Message Content**: User-provided log message

The output format is similar to:

```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] User log message
```

## Advanced Log Configuration

The TEN framework supports advanced logging functionality through the `property.json` file configuration. This allows you to customize the log output format, destination, and filtering rules.

### Configuration Structure

In the `property.json` file, you can define advanced log configuration in the `ten.log` section:

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "my_extension"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

### Configuration Components

<div className="fd-steps">
  <div className="fd-step">
    #### Log Handlers

    Each handler contains three main components:

    * **matchers**: Define which log messages will be processed by this handler
    * **formatter**: Define the log output format
    * **emitter**: Define the log output destination
  </div>

  <div className="fd-step">
    #### Matchers

    Matchers are used to filter log messages:

    ```json
    {
      "level": "info",      // Log level: "off", "debug", "info", "warn", "error"
      "category": "auth"    // Optional: log category (string)
    }
    ```

    Supported log levels:

    * `"off"`: Disable logging
    * `"debug"`: Debug level
    * `"info"`: Info level
    * `"warn"`: Warning level
    * `"error"`: Error level
  </div>

  <div className="fd-step">
    #### Formatter

    Controls the log output format:

    ```json
    {
      "type": "plain",      // Format type: "plain" or "json"
      "colored": true       // Optional: enable colored output (only for plain format)
    }
    ```

    * **plain**: Human-readable plain text format
    * **json**: JSON structured format, convenient for log analysis tools
  </div>

  <div className="fd-step">
    #### Emitter

    Defines the log output destination:

    ##### Console Output

    ```json
    {
      "type": "console",
      "config": {
        "stream": "stdout",  // "stdout" or "stderr"
        "encryption": {      // Optional: encryption configuration
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### File Output

    ```json
    {
      "type": "file",
      "config": {
        "path": "/path/to/logfile.log",  // Log file path
        "encryption": {                  // Optional: encryption configuration
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### OTLP Output

    OTLP (OpenTelemetry Protocol) output allows you to send logs to OpenTelemetry-compatible backend systems, such as Jaeger, Prometheus, Grafana, and more.

    ```json
    {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4317",  // OTLP receiver endpoint URL
        "protocol": "grpc",                   // Optional: transport protocol, "grpc" (default) or "http"
        "headers": {                          // Optional: custom HTTP headers
          "Authorization": "Bearer token",
          "X-Custom-Header": "value"
        },
        "service_name": "my-service"          // Optional: service name for identifying log source
      }
    }
    ```

    Configuration parameters:

    * **endpoint**: OTLP receiver endpoint URL (required)
    * **protocol**: Transport protocol, supports `"grpc"` or `"http"`, defaults to `"grpc"`
    * **headers**: Custom HTTP request headers for authentication or metadata (optional)
    * **service\_name**: Service name for identifying different services in log aggregation systems (optional)
  </div>
</div>

### Configuration Examples

#### Example 1: Multi-level Log Separation

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/errors.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### Example 2: Category-based Log Separation

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "database"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/database.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info",
              "category": "api"
            }
          ],
          "formatter": {
            "type": "plain"
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### Example 3: Encrypted Log Output

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/secure/logs/app.log",
              "encryption": {
                "enabled": true,
                "algorithm": "AES-CTR",
                "params": {
                  "key": "your-32-character-encryption-key",
                  "nonce": "your-16-character-nonce"
                }
              }
            }
          }
        }
      ]
    }
  }
}
```

#### Example 4: OTLP Log Output to OpenTelemetry Collector

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "http://localhost:4317",
              "protocol": "grpc",
              "service_name": "ten-framework-app"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "https://api.monitoring.example.com/v1/logs",
              "protocol": "http",
              "headers": {
                "Authorization": "Bearer your-api-token",
                "X-Environment": "production"
              },
              "service_name": "ten-framework-app"
            }
          }
        }
      ]
    }
  }
}
```

### Configuration Notes

1. **Multiple Handlers**: You can define multiple handlers to implement complex log routing strategies
2. **Matching Priority**: Handlers are matched in the order they appear in the configuration file
3. **Encryption Security**: Encryption keys and nonces should be stored securely, not hardcoded in configuration files
4. **File Permissions**: Ensure log file paths have appropriate write permissions
5. **Performance Considerations**: JSON format may produce larger log files than plain format
6. **OTLP Endpoints**:
   * When using gRPC protocol, the default port is typically 4317
   * When using HTTP protocol, the default port is typically 4318
   * Ensure the OTLP collector endpoint is accessible and properly configured
7. **Authentication**: When using OTLP, if the backend requires authentication, add appropriate credentials in the headers

## Best Practices

1. **Choose appropriate log levels**:
   * Use `DEBUG` for detailed debugging information
   * Use `INFO` for normal business processes
   * Use `WARN` for potential issues
   * Use `ERROR` for error information

2. **Provide meaningful log messages**:
   * Include contextual information (such as command names, parameter values, etc.)
   * Use descriptive messages rather than simple identifiers
   * Use the category parameter appropriately to classify logs

3. **Avoid excessive logging**:
   * Don't use too many logs in high-frequency loops
   * Appropriately adjust log levels in production environments
   * Use advanced log configuration matchers to filter unnecessary logs

4. **Thread safety**:
   * The TEN framework's logging API is thread-safe
   * Log methods can be safely called from any thread

5. **Advanced configuration recommendations**:
   * Use different categories for different modules to facilitate log analysis
   * Consider using file output in production environments for log persistence
   * Use encryption functionality for sensitive logs
   * Regularly rotate log files to control disk usage
   * Use JSON format for easier integration with log analysis tools
   * Consider using OTLP output to integrate logs into modern observability platforms (e.g., Grafana, Jaeger, Datadog)
   * OTLP output seamlessly integrates with the OpenTelemetry ecosystem, enabling distributed tracing and log correlation

The overall effect is shown in the image below:

![Unified log output for multi-language extensions](https://ten-framework-assets.s3.amazonaws.com/doc-assets/log.png)


# required (/docs/ten_framework/0.11.60/api/required)

## Principles of the `required` Field

1. **In a communication link, the source's `required` field must be a superset of the target's `required` field**

   This means that if a target extension requires a field to be mandatory, then the source extension must also mark that field as required. This ensures that messages will not lack necessary fields during transmission.

2. **Fields with the same name in both source and target `required` fields must have compatible types**

   The framework validates field type compatibility to ensure data can be correctly transmitted and parsed. Type compatibility checks include basic type matching, array item type matching, and object property type matching.

3. **Graph Check validates the integrity of the entire message transmission path**

   * Checks whether the output message schema of the source extension is compatible with the input message schema of the target extension
   * Verifies the consistency of `required` fields throughout the transmission path
   * Ensures that message transformation rules (if any) correctly handle all required fields

Graph Check is performed before deployment, helping developers identify potential message transmission issues early and avoid runtime errors.

## Usage of the `required` Field

### When Extensions Send Messages

When an extension calls `send_<foo>(msg_X)` or `return_result(result_Y)`, the framework checks `msg_X` or `result_Y` against the corresponding schema in the extension. If `msg_X` or `result_Y` lacks any field marked as `required` in the schema, the schema check will fail.

The following three scenarios are handled in the same way:

1. **When sending a command and schema check fails**

   `send_<foo>` will immediately return `false`, and the error message will contain the schema check failure details.

2. **When `return_result` fails schema check**

   `return_result` will return `false`, and the error message will contain the schema check failure details, typically showing the missing required fields, such as: "the required properties are absent: 'foo'".

3. **When sending data-like messages (such as data, audio frames, or video frames)**

   `send_<foo>` will return `false`, and the error message will contain the schema check failure details.

**Note:** The framework's schema system only takes effect when fields are present. If a field is defined in the schema but does not exist in the message, and that field is not listed in `required`, no type checking will be performed. Type validation only occurs when fields exist both in the message and in the schema definition.

### When Extensions Receive Messages

Before the TEN runtime passes `msg_X` or `result_Y` to the extension's `on_<foo>()` or result handler, it checks whether all `required` fields defined in the schema of `msg_X` or `result_Y` are present. If any `required` fields are missing, the schema check will fail.

1. **When the incoming message is a command**

   The TEN runtime will return an error to the previous extension, with an error message indicating the missing required fields.

2. **When the incoming message is a TEN command result**

   The TEN runtime will change the result's `status_code` to error, add the missing `required` fields, and set the values of these fields to default values based on their types.

3. **When the incoming message is a data-type TEN message**

   The TEN runtime will directly discard the data-type message.

### Error Message Format

When `required` field validation fails, the framework provides detailed error information:

* Single missing field: `"the required properties are absent: 'field_name'"`
* Multiple missing fields: `"the required properties are absent: 'field1', 'field2'"`
* Missing fields in nested objects: `".nested_object: the required properties are absent: 'field_name'"`


# Type System (/docs/ten_framework/0.11.60/api/type_system)

## Types

The TEN framework type system is the system used within the TEN framework to define the data types of values. Developers can declare the types of messages or extension properties through TEN schema.

The TEN framework type system includes basic types and composite types. The basic types include the following:

| Type    | Description                                               |
| ------- | --------------------------------------------------------- |
| null    | Null value type.                                          |
| bool    | Boolean value, true or false.                             |
| int8    | 8-bit signed integer.                                     |
| int16   | 16-bit signed integer.                                    |
| int32   | 32-bit signed integer.                                    |
| int64   | 64-bit signed integer.                                    |
| uint8   | 8-bit unsigned integer.                                   |
| uint16  | 16-bit unsigned integer.                                  |
| uint32  | 32-bit unsigned integer.                                  |
| uint64  | 64-bit unsigned integer.                                  |
| float32 | Single precision (32-bit) IEEE 754 floating-point number. |
| float64 | Double precision (64-bit) IEEE 754 floating-point number. |
| string  | Unicode (UTF-8) character sequence.                       |
| buf     | Sequence of 8-bit unsigned bytes.                         |
| ptr     | Pointer to a memory address.                              |

Composite types include the following:

| Type   | Description                                                    |
| ------ | -------------------------------------------------------------- |
| array  | Collection of elements of the same type.                       |
| object | Represents complex key/value pairs. Key type is always string. |

For basic types, properties can be accessed or set using methods such as `get_property()` / `set_property()`. For example:

```cpp
// Get property value
int32_t value = cmd.get_property_int32("property_name");

// Set property value
cmd.set_property("property_name", 100);
```

For composite types, it is typically necessary to use related serialization methods. For example:

```go
type MyProp struct {
    Name string `json:"name"`
}

var prop MyProp
bytes, _ := json.Marshal(&prop)
cmd.SetPropertyFromJSONBytes("property_name", bytes)
```

## JSON Type Mapping

When creating TEN values from JSON, the TEN framework uses the following mapping rules:

| JSON Type | TEN Type Mapping Rules                                                          |
| --------- | ------------------------------------------------------------------------------- |
| Integer   | Maps to int32 if the value is within int32 range; otherwise maps to int64       |
| Float     | Maps to float32 if the value is within float32 range; otherwise maps to float64 |
| String    | Maps to string                                                                  |
| Boolean   | Maps to bool                                                                    |
| null      | Maps to null                                                                    |
| Array     | Maps to array                                                                   |
| Object    | Maps to object                                                                  |

It's important to note that JSON numeric types, when no explicit schema is defined, will automatically select the appropriate TEN type based on the numeric value size. This ensures data precision and efficiency.

## Types and Schema

If a TEN schema is specified, the property type will be determined according to the corresponding TEN schema. If no TEN schema is specified, the property type will be determined based on the type of the initial value assignment. For example, if the initial assignment is C++'s `int32_t`, the property type will be `int32`; if the initial assignment is done using JSON, the type will be determined according to JSON processing rules.

When a TEN schema is defined, JSON data will be forcibly converted to the type specified by the schema. If the conversion fails, an error will be reported.

## Conversion Rules

The TEN framework supports flexible automatic conversion between different types of values. Conversions are divided into two types: safe conversions and unsafe conversions.

### Type Compatibility

The TEN framework uses type compatibility rules to determine whether conversion between two types is possible:

* **Integer type compatibility**: All integer types (int8, int16, int32, int64, uint8, uint16, uint32, uint64) are mutually compatible
* **Float compatible with integer**: Floating-point types can accept values of integer types (when the integer value is within the floating-point representation range)
* **Other types**: bool, ptr, string, array, object, buf, null types are only compatible with themselves

### Safe Conversion (Must Succeed)

Converting a lower precision type to a higher precision type is always safe and guaranteed to succeed, as the higher precision type can fully accommodate the value of the lower precision type without data loss.

In the TEN type system, safe conversion rules are as follows:

1. Within signed integer types, from lower to higher precision
2. Within unsigned integer types, from lower to higher precision
3. From smaller range unsigned integers to larger range signed integers (when the value is within the target type range)
4. Within floating-point types, from float32 to float64

| From    | To (Allowed Safe Conversions)                    |
| ------- | ------------------------------------------------ |
| int8    | int16 / int32 / int64                            |
| int16   | int32 / int64                                    |
| int32   | int64                                            |
| uint8   | uint16 / uint32 / uint64 / int16 / int32 / int64 |
| uint16  | uint32 / uint64 / int32 / int64                  |
| uint32  | uint64 / int64                                   |
| float32 | float64                                          |

For example:

```cpp
// Set property value. The type of `property_name` in TEN runtime is `int32`.
cmd.set_property("property_name", 100);

// Get property value. Correct.
int32_t value = cmd.get_property_int32("property_name");

// Get property value. Correct. TEN type system will automatically convert the type to `int64`.
int64_t value2 = cmd.get_property_int64("property_name");

// Get property value. Incorrect, an error will be thrown because this is an unsafe conversion.
int16_t error_type = cmd.get_property_int16("property_name");
```

### Unsafe Conversion (May Fail)

Converting a higher precision type to a lower precision type is unsafe because the value of the higher precision type may exceed the range of the lower precision type, leading to data loss. This conversion is called unsafe conversion. When performing unsafe conversion, the TEN runtime performs strict range checking. If an overflow or precision loss occurs, the TEN type system will throw an error.

In the TEN framework type system, unsafe conversions include:

1. Converting higher precision integers to lower precision integers
2. Converting signed integers to unsigned integers (when the value is negative)
3. Converting unsigned integers to signed integers (when the value exceeds the signed integer range)
4. Converting `float64` to `float32`
5. Converting floating-point numbers to integers (when there are decimal parts or the value exceeds integer range)
6. Converting integers to floating-point numbers (when precision loss may occur)

| Conversion Type                | Success Condition                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------ |
| High precision → Low precision | Value must be within target type range                                               |
| Signed → Unsigned              | Value must be non-negative and within target type range                              |
| Unsigned → Signed              | Value must be within target type range                                               |
| float64 → float32              | Value must be within float32 range without precision loss                            |
| Floating-point → Integer       | Value must be within target integer type range and have no decimal part              |
| Integer → Floating-point       | Integer value must be exactly representable in floating-point without precision loss |

#### Range Check Details

TEN framework implements precise range checking algorithms:

* **Integer range checking**: Uses standard MIN/MAX constants for comparison
* **Floating-point range checking**: Uses `ldexp` function to calculate precise boundary values
* **Precision checking**: For integer to floating-point conversion, performs round-trip conversion verification to ensure no precision loss

### When Conversion Is Triggered

The TEN runtime performs type conversion in the following situations:

1. **When loading `property.json`**
   * JSON integers are parsed to appropriate integer types based on numeric value size
   * JSON floating-point numbers are parsed to appropriate floating-point types based on numeric value size
   * If the property has a defined TEN schema, forced type conversion will be performed according to the schema (may be unsafe conversion)

2. **When calling methods like `set_property_from_json()`**
   * When passing serialized JSON strings, type conversion will be performed according to the schema

3. **During property access**
   * When using `get_property_xxx()` methods to get properties, the system will attempt to convert the stored type to the requested type
   * This may trigger safe or unsafe conversions

4. **During message passing**
   * When message properties need to be converted between different types

### Error Handling

When type conversion fails, the TEN framework will:

* Return error status codes and detailed error information
* Provide source type, target type, and failure reason in the error object
* For out-of-range conversions, the error message format is: "out of range of \[target\_type]"
* For incompatible type conversions, the error message format is: "unsupported conversion from `[source_type]` to `[target_type]`"

#### Common Error Examples

```cpp
// Range overflow error
int8_t small_value = cmd.get_property_int8("large_number");  // If large_number = 200
// Error: out of range of int8

// Type incompatibility error
std::string str_value = cmd.get_property_string("integer_prop");
// Error: unsupported conversion from `int32` to `string`

// Precision loss error
float float_value = cmd.get_property_float32("precise_double");  // If precise_double exceeds float32 precision
// Error: out of range of float32
```

## Best Practices

1. **Use appropriate types**: Choose suitable integer and floating-point types based on the actual data range
2. **Explicitly define schema**: Clearly define property types in manifest.json to avoid type conversion ambiguity
3. **Handle conversion errors**: Properly handle potential errors from type conversions in your code
4. **Understand JSON mapping**: Be aware of JSON data to TEN type mapping rules to avoid unexpected type conversions


# Development Process (/docs/ten_framework/0.11.60/development/development_process)

Basically, by using the TEN framework to complete the desired scenarios, there are two development approaches or perspectives:

1. From small extensions to large scenarios
2. From large scenarios to small extensions

**Note:** Regardless of which of these two development approaches you take, you can either write your own extensions from scratch or download pre-built extensions from the TEN cloud store.

In the first approach, **from small extensions to large scenarios**, the development process typically follows these steps:

1. Develop extensions.
2. Assemble the extensions into a graph, and further create an agent, for example, [TEN-Agent](https://github.com/TEN-framework/TEN-Agent).

In the second approach, **from large scenarios to small extensions**, the development process typically follows these steps:

1. Select the desired agent template, for example, [TEN-Agent](https://github.com/TEN-framework/TEN-Agent).
2. Replace some of the extensions within the chosen agent template with your own.

In this second development approach, you can ensure that the behavior of the replaced extension matches the original one by using the framework for **standalone testing** of extensions. In this process, the developer only needs to understand concepts related to TEN extensions, including how to independently develop and test a TEN extension. All other TEN-related concepts outside of extension development and testing do not need to be understood, as these concepts are encapsulated within the selected TEN agent.


# Complete Guide to TEN Extension Development (/docs/ten_framework/0.11.60/development/how_to_develop_with_ext)

TEN Framework provides rich extension templates to help developers quickly create extensions and complete the entire process from development to testing. This guide will demonstrate in detail how to use C++, Go, Python, and Node.js for the complete extension development workflow through practical operations.

## Pre-development Preparation

### Environment Requirements

Before starting extension development, please ensure your development environment is properly configured. Verify the installation with the following command:

```bash
tman --version
```

Under normal circumstances, you should see version information output similar to the following:

```bash
TEN Framework version: <version>
```

> **Important Note**: Please ensure you are using `tman` version >= 0.10.12. If your version is too low, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to install the latest version.

### Development Workflow Overview

Regardless of which programming language you use, TEN extension development follows the following standardized workflow:

1. **Project Creation** - Generate extension project skeleton using official templates
2. **Dependency Installation** - Configure runtime environment and install necessary dependency packages
3. **Core Development** - Implement extension business logic and functional code
4. **Build and Test** - Compile project (if needed) and execute unit tests
5. **Debug and Optimize** - Use professional debugging tools to locate and solve problems

***

## C++ Extension Development

C++ extensions are suitable for application scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency data computation, low-level system operations, etc.

### Create Project

Use the official C++ extension template provided by TEN to quickly create a new project:

```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```

After successful project creation, you will get the following complete project structure:

```bash
my_example_ext_cpp/
├── BUILD.gn              # GN build system configuration file
├── manifest.json         # Extension metadata and configuration information
├── property.json         # Extension properties and parameter configuration
├── src/
│   └── main.cc           # Extension core implementation code
├── tests/
│   ├── basic.cc          # Basic functionality test cases
│   └── gtest_main.cc     # Test framework program entry
├── include/              # Header file storage directory
├── tools/                # Auxiliary tools and scripts
└── .vscode/
    └── launch.json       # VSCode debugging configuration file
```

### Environment Configuration Verification

Enter the project directory and verify that the build tools are working properly:

```bash
cd my_example_ext_cpp
tgn --version
```

> **Expected Output**:

```bash
0.1.0
```

If the command cannot be executed, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to check whether the TEN Framework development environment is correctly installed.

### Dependency Package Installation

Install all dependency packages required for extension runtime:

```bash
tman install --standalone
```

After successful installation, you will see detailed installation logs similar to the following:

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done

🏆  Install successfully in 1 second
```

### Project Build

TEN framework provides two convenient build methods for you to choose from:

#### Method 1: Manual Step-by-step Build

```bash
# Step 1: Generate build configuration files (enable standalone test mode)
tgn gen linux x64 debug -- ten_enable_standalone_test=true

# Step 2: Execute project build
tgn build linux x64 debug
```

#### Method 2: Use tman Shortcut Command

```bash
tman run build
```

After the build is complete, check the generated executable test file:

```bash
ls -la bin/
# You should see: my_example_ext_cpp_test
```

### Run Tests

Verify that the extension functionality is working properly:

#### Method 1: Execute Test File Directly

```bash
./bin/my_example_ext_cpp_test
```

#### Method 2: Use tman Unified Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN      ] Test.Basic
[       OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[  PASSED  ] 1 test.
```

> **Success Indicator**: When you see `[  PASSED  ] 1 test.` it means all tests have passed.

### Core Code Structure Explanation

#### Extension Main Implementation (src/main.cc)

The core class of C++ extensions needs to inherit from the `ten::extension_t` base class and implement complete lifecycle management methods:

```cpp
class my_example_ext_cpp_t : public ten::extension_t {
 public:
  // Extension initialization phase - basic configuration and resource pre-allocation
  void on_init(ten::ten_env_t &ten_env) override;

  // Extension startup phase - officially start processing business logic
  void on_start(ten::ten_env_t &ten_env) override;

  // Command handler - handle command requests from other extensions or applications
  void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr<ten::cmd_t> cmd) override;

  // Data handler - handle general data flow
  void on_data(ten::ten_env_t &ten_env, std::unique_ptr<ten::data_t> data) override;

  // Audio frame handler - handle real-time audio stream data
  void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::audio_frame_t> frame) override;

  // Video frame handler - handle real-time video stream data
  void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::video_frame_t> frame) override;

  // Extension stop phase - clean up resources and graceful shutdown
  void on_stop(ten::ten_env_t &ten_env) override;
};
```

#### Test Framework Implementation (tests/basic.cc)

Use TEN-specific test framework to write complete unit tests:

```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    // Create command object for testing
    auto new_cmd = ten::cmd_t::create("foo");

    // Send command to extension and verify response results
    ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
      // Verify the correctness of test results here
    });
  }
};
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

1. Install the **CodeLLDB** extension plugin for VSCode
2. Set breakpoints in source code
3. Select the "standalone test (lldb, launch)" debug configuration
4. Press F5 to start debugging session

Debug configuration file `.vscode/launch.json` content:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (lldb, launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

#### Command Line Debugging

Use the classic GDB debugger for command-line debugging:

```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt  # View complete function call stack
```

***

## Go Extension Development

Go extensions provide a good balance between high performance and development efficiency, particularly suitable for building network services, concurrent processing, microservice architecture and other application scenarios.

### Create Project

Create a new project using the Go extension template:

```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```

After successful project creation, it will display:

```bash
🏆  Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```

The complete project structure is as follows:

```bash
my_example_ext_go/
├── extension.go             # Extension core implementation code
├── go.mod                   # Go module dependency management configuration
├── manifest.json            # Extension metadata information
├── property.json            # Extension property configuration
├── README.md                # Project documentation
├── tests/                   # Test-related files
│   ├── basic_tester.go      # Test logic implementation
│   ├── basic_tester_test.go # Test case definition
│   ├── main_test.go         # Test program entry
│   └── bin/start            # Test startup script
└── .vscode/launch.json      # VSCode debug configuration
```

### Dependency Package Installation

Install dependency packages required for project runtime:

```bash
tman install --standalone
```

### Run Tests

Verify the correctness of extension functionality:

#### Method 1: Use Startup Script

```bash
./tests/bin/start
```

#### Method 2: Use tman Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
=== RUN   TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```

### Core Code Structure Explanation

#### Extension Implementation (extension.go)

Core implementation structure of Go extensions:

```go
import (
    ten "ten_framework/ten_runtime"
)

type ExampleExtension struct {
    ten.DefaultExtension
}

// Extension startup lifecycle
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStart")
    tenEnv.OnStartDone()
}

// Command handling logic
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    tenEnv.LogDebug("OnCmd")
    cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
    tenEnv.ReturnResult(cmdResult, nil)
}

// Extension stop lifecycle
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStop")
    tenEnv.OnStopDone()
}
```

#### Test Framework (tests/basic\_tester.go)

Go extension test framework implementation:

```go
type BasicExtensionTester struct {
    ten.DefaultExtensionTester
}

func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
    // Create command object for testing
    cmdTest, _ := ten.NewCmd("test")

    // Send command and handle response results
    tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
        if err != nil {
            panic(err)
        }
        statusCode, _ := cr.GetStatusCode()
        if statusCode != ten.StatusCodeOk {
            panic(statusCode)
        }
        tenEnvTester.StopTest(nil)
    })

    tenEnvTester.OnStartDone()
}
```

### Development Environment Optimization

To improve development experience and debugging convenience, it is recommended to create a `go.work` workspace file in the extension root directory:

```go
go 1.18

use (
    .
    .ten/app/ten_packages/system/ten_runtime_go/interface
)
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure the official Go extension is installed, then use the following debug configuration:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (go, launch)",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}/tests/",
      "env": {
        "CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
        "LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
      },
      "args": ["-test.v"]
    }
  ]
}
```

***

## Python Extension Development

Python extensions have the highest development efficiency, particularly suitable for rapid prototyping, AI/ML application integration, complex business logic implementation and other scenarios.

### Create Project

Create a project using the Python async extension template:

```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```

Complete project structure:

```bash
my_example_ext_python/
├── extension.py         # Extension core implementation code
├── addon.py             # Extension plugin registration entry
├── __init__.py          # Python package initialization file
├── requirements.txt     # Python dependency package list
├── manifest.json        # Extension metadata configuration
├── property.json        # Extension property configuration
├── tests/               # Test-related files
│   ├── test_basic.py    # Basic test cases
│   ├── conftest.py      # pytest configuration file
│   └── bin/start        # Test startup script
└── .vscode/launch.json  # VSCode debug configuration
```

### Dependency Package Installation

Install Python dependency packages required for the project:

```bash
tman install --standalone
```

### Run Tests

Verify Python extension functionality:

#### Method 1: Use Startup Script

```bash
./tests/bin/start
```

#### Method 2: Use tman Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py .                                                                                [100%]
============================================ 1 passed in 0.04s =======================================
```

### Core Code Structure Explanation

#### Extension Implementation (extension.py)

Python extensions recommend using modern async programming patterns for better performance and concurrent processing capabilities:

```python
from ten_runtime import (
    AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
    Cmd, StatusCode, CmdResult, Data
)

class ExampleExtension(AsyncExtension):
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_init")

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_start")
        # TODO: Read configuration files here, initialize necessary resources

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_name = cmd.get_name()
        ten_env.log_debug(f"on_cmd name {cmd_name}")

        # TODO: Implement specific business logic processing here
        cmd_result = CmdResult.create(StatusCode.OK, cmd)
        await ten_env.return_result(cmd_result)

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_stop")
        # TODO: Clean up resources here, perform graceful shutdown
```

#### Plugin Registration Entry (addon.py)

Extension plugin registration and creation logic:

```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension

@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
    def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
        ten_env.log_info("on_create_instance")
        ten_env.on_create_instance_done(ExampleExtension(name), context)
```

#### Test Implementation (tests/test\_basic.py)

Complete async test framework implementation:

```python
from ten_runtime import (
    AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
    TenError, TenErrorCode
)

class ExtensionTesterBasic(AsyncExtensionTester):
    async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
        # Create command object for testing
        new_cmd = Cmd.create("hello_world")

        ten_env.log_debug("send hello_world")
        result, err = await ten_env.send_cmd(new_cmd)

        # Verify the correctness of test results
        if (err is not None or result is None
            or result.get_status_code() != StatusCode.OK):
            ten_env.stop_test(TenError.create(
                TenErrorCode.ErrorCodeGeneric,
                "Failed to send hello_world"
            ))
        else:
            ten_env.stop_test()

def test_basic():
    tester = ExtensionTesterBasic()
    tester.set_test_mode_single("my_example_ext_python")
    err = tester.run()
    if err is not None:
        assert False, err.error_message()
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure you have installed the Python extension and debugpy debugger, use the following configuration for debugging:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (debugpy, launch)",
      "type": "debugpy",
      "request": "launch",
      "python": "/usr/bin/python3",
      "module": "pytest",
      "args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
      "env": {
        "TEN_ENABLE_PYTHON_DEBUG": "true",
        "PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
      },
      "console": "integratedTerminal"
    }
  ]
}
```

***

## Node.js Extension Development

Node.js extensions provide a modern JavaScript/TypeScript development experience, particularly suitable for Web application integration, rapid prototyping, frontend technology stack extensions and other scenarios. Thanks to Node.js's async characteristics and rich ecosystem, developers can easily build efficient real-time applications.

### Create Project

Use the official Node.js extension template provided by TEN to quickly create a new project:

```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```

After successful project creation, you will get the following complete project structure:

```bash
my_example_ext_nodejs/
├── manifest.json         # Extension metadata and configuration information
├── property.json         # Extension properties and parameter configuration
├── package.json          # Node.js dependency package management configuration
├── tsconfig.json         # TypeScript compiler configuration
├── src/
│   └── index.ts          # Extension core implementation code
├── tests/                # Test-related files
│   ├── src/
│   │   ├── index.ts      # Tester implementation
│   │   ├── index.spec.ts # Test case definition
│   │   └── main.spec.ts  # Test framework configuration
│   ├── bin/start         # Test startup script
│   ├── package.json      # Test dependency configuration
│   └── tsconfig.json     # Test TypeScript configuration
├── tools/                # Auxiliary tools and scripts
└── .vscode/
    └── launch.json       # VSCode debug configuration file
```

### Environment Configuration Verification

Enter the project directory and verify the development environment:

```bash
cd my_example_ext_nodejs
node --version
npm --version
```

> **Expected Output**:

```bash
v18.0.0 or higher
8.0.0 or higher
```

### Dependency Package Installation

Install all dependency packages required for extension runtime:

```bash
tman install --standalone
```

After successful installation, you will see detailed installation logs similar to the following:

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       2/2       Done

🏆  Install successfully in 1 second
```

> **Important Note**: `tman install --standalone` will create a `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory in the project directory. Subsequent build and test operations need to be performed in this directory.

### Project Build

Node.js extensions are developed using TypeScript and need to install standalone mode dependencies first, then compile to JavaScript:

#### Method 1: Manual Build

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Install standalone mode Node.js dependencies
npm run standalone-install

# Compile TypeScript code
npm run build
```

#### Method 2: Use tman Shortcut Command

```bash
tman run build
```

After the build is complete, check the generated compilation results:

```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# You should see: index.js and related mapping files
```

### Run Tests

Verify that the extension functionality is working properly:

#### Method 1: Use Startup Script

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Run tests
./tests/bin/start
```

#### Method 2: Use tman Unified Command

```bash
tman run test
```

Example output when tests execute successfully:

```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case2 (1010ms)

  2 passing (2s)
```

> **Success Indicator**: When you see all test cases showing `✓` and finally displaying `passing`, it means all tests have passed.

### Core Code Structure Explanation

#### Extension Main Implementation (src/index.ts)

The core class of Node.js extensions needs to inherit from the `Extension` base class and implement complete lifecycle management methods:

```typescript
import {
  Addon,
  RegisterAddonAsExtension,
  Extension,
  TenEnv,
  Cmd,
  CmdResult,
  StatusCode,
} from "ten-runtime-nodejs";

class DefaultExtension extends Extension {
  constructor(name: string) {
    super(name);
  }

  // Extension configuration phase - read and validate configuration parameters
  async onConfigure(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onConfigure");
  }

  // Extension initialization phase - basic configuration and resource pre-allocation
  async onInit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onInit");
  }

  // Extension startup phase - officially start processing business logic
  async onStart(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStart");
  }

  // Command handler - handle command requests from other extensions or applications
  async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
    console.log("DefaultExtension onCmd", cmd.getName());

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("detail", "This is a demo");
    tenEnv.returnResult(cmdResult);
  }

  // Extension stop phase - clean up resources and graceful shutdown
  async onStop(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStop");
  }

  // Extension destruction phase - final cleanup and resource release
  async onDeinit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onDeinit");
  }
}
```

#### Plugin Registration Entry

Extension plugin registration and creation logic:

```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
  async onCreateInstance(
    _tenEnv: TenEnv,
    instanceName: string,
  ): Promise<Extension> {
    return new DefaultExtension(instanceName);
  }
}
```

#### Test Framework Implementation (tests/src/index.ts)

Use TEN-specific test framework to write complete unit tests:

```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";

export class MyExtensionTester extends ExtensionTester {
  async onStart(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStart");

    // Simulate async operations and test logic
    new Promise((resolve) => {
      setTimeout(() => {
        resolve(true);
      }, 1000);
    }).then(() => {
      // Stop test and return results
      tenEnvTester.stopTest();
    });
  }

  async onStop(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStop");
  }

  async onDeinit(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onDeinit");
  }
}
```

#### Test Case Definition (tests/src/index.spec.ts)

Use Mocha test framework to write specific test cases:

```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";

const test_addon_name = "default_extension_nodejs";

describe("MyExtensionTester", () => {
  it("case1", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });

  it("case2", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });
});
```

### TypeScript Configuration

Node.js extensions use modern TypeScript configuration, supporting the latest language features:

```json
{
  "compilerOptions": {
    "allowJs": false,
    "composite": true,
    "module": "NodeNext",
    "target": "es6",
    "moduleResolution": "NodeNext",
    "outDir": "build",
    "removeComments": false,
    "sourceMap": true,
    "noImplicitAny": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "strictNullChecks": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "rootDir": "src",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
  },
  "exclude": [
    "node_modules"
  ],
  "include": [
    "src/**/*"
  ]
}
```

### Debug Environment Configuration

#### VSCode Integrated Debugging

Make sure you have installed the official Node.js extension, then use the following debug configuration:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (nodejs) (mocha, launch)",
      "type": "node",
      "request": "launch",
      "program": "node_modules/mocha/bin/_mocha",
      "stopOnEntry": true,
      "args": [
        "--no-timeouts",
        "--package",
        "package.json",
      ],
      "cwd": "${workspaceFolder}/tests",
      "env": {
        "NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
      },
      "runtimeArgs": [
        "--expose-gc",
        "--loader",
        "ts-node/esm",
        "--no-warnings",
      ]
    }
  ]
}
```

#### Command Line Debugging

Use Node.js built-in debugger for command-line debugging:

```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# Start debug mode
node --inspect-brk ./tests/bin/start

# Or use Chrome DevTools
node --inspect ./tests/bin/start
```

### Complete Development Process Summary

To help developers get started quickly, here is a complete Node.js extension development process summary:

```bash
# 1. Create extension project
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example

# 2. Enter project directory
cd my_example_ext_nodejs

# 3. Install dependencies
tman install --standalone

# 4. Enter extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 5. Install standalone mode dependencies
npm run standalone-install

# 6. Build project
npm run build

# 7. Run tests
./tests/bin/start
```

> **Working Directory Note**:
>
> * Extension source code is located in the `src/` folder at the project root directory
> * Actual build, test, and run operations are performed in the `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory
> * This design ensures extension independence and proper dependency management

### Development Best Practices

#### Async Programming

Node.js extensions fully utilize async programming patterns for better performance:

```typescript
// Recommended: Use async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    const result = await processCommand(cmd);
    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("result", result);
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

#### Error Handling

Implement comprehensive error handling mechanisms:

```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    // Business logic processing
    const result = await this.handleBusinessLogic(cmd);

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("data", JSON.stringify(result));
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    console.error("Command processing error:", error);

    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

***

## Development Summary

By following the complete development process provided in this guide, you can efficiently develop, test, and debug TEN extensions. Whether you choose C++, Go, Python, or Node.js, TEN Framework provides you with a complete toolchain and best practices to help you fully leverage the powerful features of TEN Framework and build high-performance, high-reliability extension applications.

Each language has its unique advantages and applicable scenarios:

* **C++**: Suitable for scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency computation
* **Go**: Provides a balance between high performance and development efficiency, suitable for network services, concurrent processing
* **Python**: Has the highest development efficiency, particularly suitable for AI/ML applications, rapid prototyping
* **Node.js**: Provides modern Web development experience, suitable for frontend technology stack extensions, real-time applications

Please choose the most suitable development solution based on your specific needs and team technology stack. During the development process, it is recommended to fully utilize the debugging tools and testing frameworks provided by TEN Framework to ensure the quality and stability of your extensions.


# TEN Framework Quick Start Guide (/docs/ten_framework/0.11.60/getting-started/quick-start)

## TEN Framework Quick Start Guide

> 🎯 **Goal**: Set up your development environment and run your first TEN app in 5 minutes

## System Requirements

**Supported Operating Systems**:

* Linux (x64)
* Linux (arm64)
* macOS Intel (x64)
* macOS Apple Silicon (arm64)
* Windows (x64)

**Required Software**:

* Python 3.10
* Go 1.20+
* Node.js / npm (for managing JavaScript dependencies)

> 💡 **Windows users**: It's recommended to use PowerShell to run the commands in this guide. Some commands may not be compatible with CMD.

## Step 1: Check Your Environment

Before you begin, make sure the following software is installed on your system:

### Python 3.10

```bash
python3 --version
# Should display: Python 3.10.x
```

> 💡 **Important**: TEN Framework currently only supports Python 3.10. It's recommended to use `pyenv` or `venv` to manage your Python environment:
>
> **Linux / macOS:**
>
> ```bash
> # Install and manage Python 3.10 using pyenv (recommended)
> pyenv install 3.10.14
> pyenv local 3.10.14
>
> # Or create virtual environment using venv
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows:**
>
> Download the Windows installer from the bottom table at [https://www.python.org/downloads/release/python-31011/](https://www.python.org/downloads/release/python-31011/) and run it to install Python 3.10.
>
> Make sure to check "Add Python to PATH" before clicking Install Now.
>
> ```bash
> # On Windows, configure the python3 command:
>
> # First, find where python is installed:
> where.exe python
> # Example output: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # Then, open PowerShell as Administrator and create a symlink in the same directory:
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # Replace the paths above with the actual output from where.exe python
>
> # Verify:
> python3 --version
> ```
>
> ```powershell
> # It's recommended to create a venv after installation
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # Activate the environment
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # If you encounter a permission error, close the terminal/IDE, right-click and select "Run as Administrator" to reopen
> # Or change the execution policy to allow ps1 scripts
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```

### Go 1.20+

```bash
go version
# Should display: go version go1.20 or higher

$env:CGO_ENABLED = "1"
# CGO must be explicitly enabled on Windows
```

### Node.js / npm

```bash
node --version
npm --version
# Ensure node and npm commands are available
```

### GCC (MinGW)

💡 Only required on Windows

```bash
# First, make sure winget is available.
winget --version

# If not, install it from <https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US>
# (Requires Windows 10 version 1709 (Build 16299) or later, or Windows 11)

# Install MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT

# Or search and choose a suitable version to install
winget search "mingw"

# Verify installation
gcc --version
```

> 💡 **Tip**: If any of the above is missing, please install the required version before continuing.

## Step 2: Install TEN Manager (tman)

TEN Manager (tman) is the command-line tool for TEN Framework, used to create projects, manage dependencies, and run applications.

Option 1: Install via Package Manager (Recommended)

**Linux (Ubuntu/Debian):**

```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```

**macOS:**

```bash
brew install TEN-framework/ten-framework/tman
```

**Windows:**

```powershell
winget install TEN-framework.tman
```

Option 2: Install via Script

**Linux / macOS:**

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```

**Windows:**

```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```

Option 3: Install from cloned repository

```bash
cd ten-framework

# Linux/macOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```

> 💡 **Note**: If tman is already installed on your system, the installation script will ask whether you want to reinstall/upgrade it. Press `y` to continue or `n` to cancel.
>
> **Non-interactive Installation** (for automation scripts or CI environments, not available on Windows):
>
> ```bash
> # Remote installation
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # Local installation
> yes y | bash tools/tman/install_tman.sh
> ```

**Verify Installation**:

```bash
tman --version
```

> 💡 **Tip**: If you see `tman: command not found` (Linux/macOS) or a similar unrecognized command error (Windows), make sure the tman directory is in your PATH:
>
> **Linux / macOS:**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc  # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc   # macOS
> source ~/.bashrc  # or source ~/.zshrc
> ```
>
> **Windows:**
>
> ```powershell
> # Check if tman is in PATH
> $env:Path -split ";" | Select-String "tman"
> # If not, either:
> # 1. Add it manually via "System Properties → Environment Variables", then restart your terminal
> # 2. Or temporarily add the path with:
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```

## Step 3: Create and Run the Demo App

<div className="fd-steps">
  <div className="fd-step">
    ### Create App

    ```bash
    # Create a new transcriber_demo app
    tman install app transcriber_demo
    cd transcriber_demo
    ```
  </div>

  <div className="fd-step">
    ### Install Dependencies

    ```bash
    # Install TEN package dependencies
    tman install

    # Install Python and npm package dependencies
    tman run install_deps
    ```

    > ⏱️ **Estimated Time**: 1-2 minutes
  </div>

  <div className="fd-step">
    ### Build the App

    ```bash
    tman run build
    ```

    > ⏱️ **Estimated Time**: 30 seconds
  </div>

  <div className="fd-step">
    ### Configure Environment Variables

    Before running the app, you need to configure the ASR (Automatic Speech Recognition) service credentials. The current example uses Azure ASR extension. You need to fill in the configuration in the `transcriber_demo/.env` file:

    **Linux / macOS:**

    ```bash
    # Create .env file
    cat > .env << EOF
    # Azure Speech Service Configuration
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region      # e.g., eastus
    AZURE_STT_LANGUAGE=en-US                # Set according to your audio language or real-time recording language, e.g., zh-CN, ja-JP, ko-KR, etc.
    EOF
    ```

    **Windows (PowerShell):**

    ```powershell
    # Create .env file
    @"
    # Azure Speech Service Configuration
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region
    AZURE_STT_LANGUAGE=en-US
    "@ | Out-File -Encoding utf8 .env
    ```

    > 💡 **Tip**: Windows users can also create the `.env` file directly with a text editor and fill in the configuration above.
    >
    > 💡 **Tip**: If you want to use other ASR extensions (such as OpenAI Whisper, Google Speech, etc.), you can download and replace them from the cloud store. Similarly, configure the corresponding API keys and environment variables in the `.env` file.
  </div>

  <div className="fd-step">
    ### Run the App

    ```bash
    tman run start
    ```

    If everything is working correctly, you should see output similar to:

    ```text
    [web_audio_control_go] Web server started on port 8001
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```
  </div>

  <div className="fd-step">
    ### Experience the Demo

    Open your browser and visit:

    ```text
    http://localhost:8001
    ```

    You should see the Transcriber Demo web interface. Try:

    * Click the microphone button for real-time voice transcription
    * Upload an audio file for transcription
    * View real-time transcription and subtitle results
  </div>
</div>

## Congratulations! 🎉

You've successfully run your first TEN application!

### Understanding the App Architecture

This `transcriber_demo` app showcases TEN Framework's multi-language extension capabilities, consisting of:

* **Go** - WebSocket server extension (`web_audio_control_go`)
* **Python** - ASR speech recognition extension (`azure_asr_python`)
* **TypeScript** - VTT subtitle generation and audio recording extension (`vtt_nodejs`)

🎯 **You can now run extensions in multiple languages!**

### Next Steps

Now you can:

1. **Explore and download more extensions from the cloud store, design and orchestrate your app**

   ```bash
   tman designer  # Launch TMAN Designer to explore extensions, download them, and design your app
   ```

2. **Choose a language and develop your own extension**
   * Supports Go, Python, TypeScript/JavaScript, C++, and more
   * Check out the [TEN Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) for details

## Advanced: Developing and Building C++ Extensions

If you want to develop and use C++ extensions, you'll need to install the TEN build toolchain (tgn). Here's the complete process:

<div className="fd-steps">
  <div className="fd-step">
    ### Install tgn Build Tool

    tgn is TEN Framework's C/C++ build system, based on Google's GN.

    Option 1: One-line Installation (Recommended)

    **Linux / macOS:**

    ```bash
    curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
    ```

    **Windows:**

    ```powershell
    irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
    ```

    Option 2: Install from Cloned Repository

    **Linux / macOS:**

    ```bash
    # If you've already cloned the TEN Framework repository
    cd ten-framework
    bash tools/tgn/install_tgn.sh
    ```

    **Windows:**

    ```powershell
    # If you've already cloned the TEN Framework repository
    & ".\tools\tgn\install_tgn.ps1"
    ```

    After installation, ensure tgn is added to PATH:

    **Linux / macOS:**

    ```bash
    # Temporarily add to current session
    export PATH="/usr/local/ten_gn:$PATH"

    # Or permanently add to shell configuration (recommended)
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc  # Linux
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc   # macOS
    source ~/.bashrc  # or source ~/.zshrc
    ```

    **Windows:**

    ```powershell
    # The install script adds to user PATH automatically. To add manually:
    $env:Path += ";$env:LOCALAPPDATA\ten_gn"
    ```

    Verify installation:

    ```bash
    tgn --help
    ```
  </div>

  <div className="fd-step">
    ### Install C++ Extension

    Using WebRTC VAD (Voice Activity Detection) extension as an example, install a C++ extension from the cloud store:

    ```bash
    cd transcriber_demo
    tman install extension webrtc_vad_cpp
    ```

    > 💡 **Note**: `webrtc_vad_cpp` is a voice activity detection extension implemented in C++. It can filter out non-speech segments in real-time speech recognition scenarios.
  </div>

  <div className="fd-step">
    ### Compile C++ Extension

    After installing the C++ extension, rebuild the app to compile the C++ code into a dynamic library:

    ```bash
    tman run build
    ```

    > ⏱️ **Expected Time**: First-time compilation of C++ extensions may take 1-3 minutes, depending on your machine's performance.
  </div>

  <div className="fd-step">
    ### Run App with VAD Functionality

    ```bash
    tman run start_with_vad
    ```

    If everything works correctly, you should see:

    ```text
    [web_audio_control_go] Web server started on port 8001
    [vad] WebRTC VAD initialized with mode 2
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```

    Now open your browser at `http://localhost:8001` and navigate to the microphone real-time transcription page. You'll see the silence state changes after VAD processing. When the silence state is true, it indicates there is no speech in the current audio.
  </div>
</div>

### C++ Development Environment Requirements

Developing and compiling C++ extensions requires installing a C++ compiler (gcc or clang):

**Linux:**

```bash
# Ubuntu/Debian
sudo apt-get install gcc g++

# Or use clang
sudo apt-get install clang
```

**macOS:**

```bash
# Install Xcode Command Line Tools (includes clang)
xcode-select --install
```

**Windows:**

```powershell
# Option 1: Install Visual Studio Build Tools (recommended)
# Download from https://visualstudio.microsoft.com/visual-cpp-build-tools/

# Option 2: Install via winget
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

# 💡 Note:
#  - Select the "Desktop development with C++" workload during installation
#  - Make sure to check "C++ Clang tools for Windows"
```

Verify compiler installation:

```bash
# Check gcc (Linux/macOS/MSYS2)
gcc --version
g++ --version

# Or check clang (Linux/macOS)
clang --version
```

**Windows (MSVC):**

```powershell
# Run in "Developer PowerShell for VS"
clang-cl --version
```

### Troubleshooting (C++ Extensions)

1. tgn command not found

   Ensure you've run the installation script and added tgn to PATH:

   ```bash
   export PATH="/usr/local/ten_gn:$PATH"
   ```

2. Compilation failed: Compiler not found

   Please refer to the "C++ Development Environment Requirements" section above to install the compiler.

### Learn More

* [ten\_gn Build System](https://github.com/TEN-framework/ten_gn) - TEN's C/C++ build tool
* [C++ Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) - Complete C++ extension development documentation

## Troubleshooting

<div className="fd-steps">
  <div className="fd-step">
    ### Python Library Loading Failure on macOS

    **Problem**: Error indicating `libpython3.10.dylib` cannot be found when running the app

    **Solution**:

    ```bash
    export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
    ```

    Consider adding this line to your `~/.zshrc` or `~/.bash_profile`.
  </div>

  <div className="fd-step">
    ### tman Download Failed or Slow

    **Problem**: Network connection to GitHub is restricted

    **Solution**:

    * Manual download: Visit the [Releases page](https://github.com/TEN-framework/ten-framework/releases) to download the `tman` binary for your platform
  </div>

  <div className="fd-step">
    ### Port 8001 Already in Use

    **Problem**: Port conflict error when starting the app

    **Solution**:

    * Find the process using the port:
      * Linux/macOS: `lsof -i :8001`
      * Windows: `netstat -ano | findstr :8001`
    * Kill the process:
      * Linux/macOS: `kill -9 <PID>`
      * Windows: `taskkill /PID <PID> /F`
    * Or modify the port number in the app configuration file (`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`)
  </div>

  <div className="fd-step">
    ### Go Build Failed

    **Problem**: Go module related errors during build

    **Solution**:

    ```
    # Set proxy
    # Linux/macOS
    export GOPROXY=https://goproxy.cn,direct
    # Windows PowerShell
    $env:GOPROXY = "https://goproxy.cn,direct"

    # Clean Go module cache
    go clean -modcache

    # Reinstall dependencies
    cd transcriber_demo
    tman run build
    ```
  </div>

  <div className="fd-step">
    ### Python Dependencies Installation Failed

    **Problem**: pip installation timeout or failure

    **Solution**: Use a mirror source (for users in China)

    ```bash
    pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
    ```
  </div>

  <div className="fd-step">
    ### Azure Speech Service Error

    **Problem**: Azure Speech Service related errors, such as authentication failure

    **Solution**: Check that the configuration in your `.env` file is correct, and ensure `AZURE_STT_KEY` and `AZURE_STT_REGION` are filled in accurately
  </div>

  <div className="fd-step">
    ### Permission Error on Windows

    **Problem**: PermissionError when accessing files on Windows

    **Solution**: Right-click PowerShell and select "Run as Administrator"
  </div>
</div>

## Get Help

* **GitHub Issues**: [https://github.com/TEN-framework/ten-framework/issues](https://github.com/TEN-framework/ten-framework/issues)
* **Documentation**: [https://theten.ai/docs](https://theten.ai/docs)
* **Contributing Guide**: [contributing.md](../code-of-conduct/contributing.md)


# WebSocket Voice Assistant Quickstart (/docs/ten_framework/0.11.60/getting-started/websocket-voice-assistant-quick-start)

## WebSocket Voice Assistant Quickstart

Build and run a real-time WebSocket voice assistant in under 10 minutes.

## Overview

This guide helps you run a complete voice assistant demo locally. It supports real-time voice conversations and tool calling. The demo includes:

| Component | Vendor     | Purpose                   |
| --------- | ---------- | ------------------------- |
| ASR       | Deepgram   | Speech-to-text            |
| LLM       | OpenAI     | Response generation       |
| TTS       | ElevenLabs | Text-to-speech            |
| Tool      | WeatherAPI | Weather lookup (optional) |

## Requirements

### Prerequisites

Follow the [TEN Framework Quick Start](./quick-start.md) to install the base environment, then make sure the check below passes:

```bash
tman check env
```

Expected output (at least Python and Go are Ready):

```text
✅ Operating System: Supported
✅ Python:   Ready
✅ Go:       Ready
```

### Extra tools

Install the following tools for build & run:

#### Task (task runner)

**Linux / macOS:**

```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version  # verify
```

**Windows:**

```powershell
# Option 1: Install via winget (recommended)
# Install winget from: https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# Requires Windows 10 version 1709 (Build 16299) or higher, or Windows 11
winget --version  # verify winget
winget install Task.Task
task --version  # verify

# Option 2: Install via scoop
irm get.scoop.sh | iex
scoop --version  # verify scoop
scoop install task
task --version  # verify
```

#### Bun (JavaScript package manager)

**Linux / macOS:**

```bash
curl -fsSL https://bun.sh/install | bash
bun --version  # verify
```

**Windows:**

```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version  # verify
```

#### uv (Python package manager)

**Linux / macOS:**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version  # verify
```

**Windows:**

```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version  # verify
```

> 💡 After installation, reload your shell config:
>
> * Linux/macOS: `source ~/.zshrc`
> * Windows: reopen your PowerShell terminal

## Quickstart

<div className="fd-steps">
  <div className="fd-step">
    ### Clone the repo

    ```bash
    git clone https://github.com/TEN-framework/ten-framework.git
    cd ten-framework/ai_agents/agents/examples/websocket-example
    ```
  </div>

  <div className="fd-step">
    ### Install dependencies

    ```bash
    task install
    ```

    > ⏱️ Estimated time: 2–3 minutes

    This command will:

    * Install TEN extension packages (45+)
    * Build the Go app and API server
    * Install Python and frontend dependencies
  </div>

  <div className="fd-step">
    ### Configure API keys

    Configure vendor keys in `ai_agents/.env`:

    **Linux / macOS:**

    ```bash
    cd ../../../  # back to ai_agents
    vim .env
    ```

    **Windows:**

    ```powershell
    cd ..\..\..   # back to ai_agents
    notepad .env
    ```

    Copy all content from .env.example, then fill in:

    ```bash
    # Deepgram - ASR
    DEEPGRAM_API_KEY=your_deepgram_api_key

    # OpenAI - LLM
    OPENAI_API_KEY=your_openai_api_key
    OPENAI_MODEL=your_openai_model

    # ElevenLabs - TTS
    ELEVENLABS_TTS_KEY=your_elevenlabs_api_key

    # WeatherAPI - Tool (optional)
    WEATHERAPI_API_KEY=your_weatherapi_key
    ```
  </div>

  <div className="fd-step">
    ### Start services

    ```bash
    cd agents/examples/websocket-example
    task run
    ```

    After startup, the following services are available:

    * **Frontend UI**: [http://localhost:3000](http://localhost:3000)
    * **API server**: [http://localhost:8080](http://localhost:8080)
    * **TMAN Designer**: [http://localhost:49483](http://localhost:49483)
  </div>

  <div className="fd-step">
    ### Talk to your assistant

    Open [http://localhost:3000](http://localhost:3000), click the microphone button, and start speaking.

    Try:

    * "Hello, who are you?"
    * "What's the weather in Beijing today?" (requires WeatherAPI)
    * "Tell me a joke."

    **Congratulations!** Your TEN Agent voice assistant is running.
  </div>
</div>

## Advanced

### Swap extensions

TEN Framework lets you swap ASR/LLM/TTS extensions flexibly.

#### Option 1: Use TMAN Designer (recommended)

Open [http://localhost:49483](http://localhost:49483), then in the visual editor:

1. Click an extension node to inspect it
2. Choose a replacement extension
3. Configure parameters
4. Save and restart

#### Option 2: Edit configuration file

Edit `tenapp/property.json` and modify the extension configs under `predefined_graphs`.

##### Example: switch ASR to Azure

```json
{
  "type": "extension",
  "name": "stt",
  "addon": "azure_asr_python",
  "extension_group": "stt",
  "property": {
    "params": {
      "key": "${env:AZURE_STT_KEY}",
      "region": "${env:AZURE_STT_REGION}",
      "language": "en-US"
    }
  }
}
```

##### Example: switch TTS to Minimax

```json
{
  "type": "extension",
  "name": "tts",
  "addon": "minimax_tts_websocket_python",
  "extension_group": "tts",
  "property": {
    "params": {
      "api_key": "${env:MINIMAX_TTS_API_KEY|}",
      "group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
      "model": "speech-02-turbo",
      "audio_setting": {
        "sample_rate": 16000
      },
      "voice_setting": {
        "voice_id": "female-shaonv"
      }
    }
  }
}
```

Restart after changes (`Ctrl+C` then `task run`).

### Monitoring (Grafana)

Use Grafana to observe application metrics and logs.

<div className="fd-steps">
  <div className="fd-step">
    #### Start monitoring stack

    **Linux / macOS:**

    ```bash
    cd tools/grafana-monitoring
    docker-compose -f docker-compose.push.yml up -d
    ```

    **Windows:**

    ```powershell
    cd tools\grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    # If Docker is not installed, install it from https://apps.microsoft.com/detail/xp8cbj40xlbwkx, restart your PC, then launch Docker Desktop
    ```

    Services:

    * **Grafana** (port 3001) - UI
    * **Prometheus** (port 9091) - metrics store
    * **Loki** (port 3100) - log store
    * **OpenTelemetry Collector** (ports 4317/4318) - OTLP receiver
  </div>

  <div className="fd-step">
    #### Configure app telemetry

    Edit `agents/examples/websocket-example/tenapp/property.json`:
  </div>
</div>

#### Logs (export)

Add to the `ten.log.handlers` array:

```json
{
  "matchers": [{ "level": "info" }],
  "formatter": { "type": "json", "colored": false },
  "emitter": {
    "type": "otlp",
    "config": {
      "endpoint": "http://localhost:4317",
      "service_name": "websocket-example"
    }
  }
}
```

#### Metrics (export)

Add to `ten.services`:

```json
{
  "telemetry": {
    "enabled": true,
    "metrics": {
      "enabled": true,
      "exporter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
  }
}
```

<div className="fd-steps">
  <div className="fd-step">
    #### Restart and open Grafana

    ```bash
    cd ../../../agents/examples/websocket-example
    task run
    ```

    Open [http://localhost:3001](http://localhost:3001) (username/password: `admin/admin`).
  </div>
</div>

#### Metrics (Dashboards)

Go to `Dashboards` → **"TEN Framework"** to view:

* Extension startup latency
* Cmd processing latency
* Message queue wait time

#### Logs (Explore/Loki)

Go to `Explore` → select `Loki`, then query:

* `{service_name="websocket-example"}` - all logs
* `{service_name="websocket-example", ten_extension_name="stt"}` - ASR logs
* `{service_name="websocket-example", ten_extension_name="llm"}` - LLM logs
* `{service_name="websocket-example", ten_extension_name="tts"}` - TTS logs

> 💡 Use the `ten_extension_name` label to filter logs per extension.

Stop the monitoring stack:

**Linux / macOS:**

```bash
cd tools/grafana-monitoring
docker-compose -f docker-compose.push.yml down
```

**Windows:**

```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```

## Troubleshooting

<details>
<summary><strong>Frontend dependency install fails</strong></summary>

If `bun install` fails due to missing versions, switch to the official npm registry:

```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```

</details>

<details>
<summary><strong>macOS Python library loading fails</strong></summary>

Set the Python dynamic library path:

```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# Add to ~/.zshrc for persistence
```

</details>

<details>
<summary><strong>Windows Python library loading fails</strong></summary>

Make sure the Python 3.10 installation path is in your system PATH:

```powershell
# Check Python path
python -c "import sys; print(sys.prefix)"

# Add Python DLL directory to PATH (adjust to your actual path)
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```

If you didn't check "Add Python to PATH" during installation, re-run the installer and choose "Modify" to add it.

</details>

<details>
<summary><strong>Port already in use</strong></summary>

Find and kill the process:

**Linux / macOS:**

```bash
lsof -i :3000  # or :8080
kill -9 <PID>
```

**Windows:**

```powershell
netstat -ano | findstr :3000   # or :8080
taskkill /PID <PID> /F
```

</details>

<details>
<summary><strong>Go module related errors</strong></summary>

```bash
# Set proxy
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"

# Clean Go module cache
go clean -modcache

# Reinstall dependencies
cd ai_agents/agents/examples/websocket-example
task install
```

</details>

## Next steps

* **[Extension development guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext)** - build custom extensions
* **[More examples](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - explore other agent demos
* **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - report bugs or request features
* **[Documentation hub](https://theten.ai/docs)** - full TEN Framework documentation


# Graph (/docs/ten_framework/0.11.60/graph/graph)

In the TEN framework, there are two types of graphs:

1. Dynamic Graph
2. Predefined Graph (`predefined_graph`)

|               | Dynamic Graph                                       | Predefined Graph                                                       |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------- |
| Startup Time  | When the TEN app receives the `start_graph` command | When the TEN app starts, or when it receives the `start_graph` command |
| Graph Content | Specified in the `start_graph` command              | Predefined in the TEN app's properties                                 |
| Graph ID      | Randomly generated UUID                             | Randomly generated UUID                                                |

![Two types of graphs](https://ten-framework-assets.s3.amazonaws.com/doc-assets/two_types_of_graph.png)

Predefined graphs have an `auto_start` property to determine whether the graph automatically starts when the TEN app launches.

Additionally, predefined graphs have a `singleton` property that indicates whether only *one* instance of the graph can be generated within the TEN app.

## Graph ID and Graph Name

For each graph instance, the TEN app internally uses a unique UUID4 string as an identifier, called the **Graph ID**.

For predefined graphs, you can assign a meaningful and easy-to-remember name, called the **Graph Name**. When you need to specify a particular predefined graph, you can directly use its graph name. If a predefined graph has the `singleton` property, it means that only one instance of this predefined graph can exist within the TEN app. In this case, the TEN runtime uses the graph name to uniquely identify the single instance generated from that predefined graph.

## Dynamic Graph

When a TEN app receives a `start_graph` command and creates a dynamic graph, the system assigns a random UUID as the graph's ID. If other clients obtain this graph ID, they can also connect to this graph.

Dynamic Graph ID example:

`123e4567-e89b-12d3-a456-426614174000`

## Predefined Graph

Predefined graphs work similarly to dynamic graphs, with the main difference being how the content is defined. The content of a dynamic graph is included in the `start_graph` command, while the content of a predefined graph is defined in advance by the TEN app. Clients only need to specify the name of the predefined graph in the `start_graph` command to start it.

The main advantage of predefined graphs is simplifying usage and protecting sensitive information. With predefined graphs, clients don't need to know the detailed structure of the graph, which both improves usability and avoids exposing potentially sensitive information contained in the graph.

Predefined Graph name example:

`http_server`

When a TEN app starts, all predefined graphs with the `auto_start` property set will automatically start.

## Graph Definition

Whether it's a dynamic graph or a predefined graph, the definition structure is the same:

```json
{
  "nodes": [
    // Definition of nodes
  ],
  "connections": [
    // Definition of communication links
  ]
}
```

Key points:

1. If there is only one TEN app, you can omit the `app` field. If there are multiple apps, you must explicitly specify the `app` field.

2. The `nodes` field defines the nodes in the graph, such as various extensions.

3. Each node can appear only once in the `nodes` field. If the same node appears multiple times, the TEN framework will report an error during validation.

4. The way to define an extension in the `nodes` field is as follows, where the `property` field is optional:

   ```json
   {
     "type": "extension",
     "name": "simple_http_server_cpp",
     "addon": "simple_http_server_cpp",
     "extension_group": "default_extension_group",
     "app": "msgpack://127.0.0.1:8001/",
     "property": {
       "root_key": "player",
       "extra_keys": ["playerName"]
     }
   }
   ```

   The `addon` field indicates that the extension is an instance generated by the corresponding plugin.

5. The `connections` field defines the communication links between nodes in the graph, where the `extension` value represents the name of the corresponding node.

Complete example:

```json
{
  "nodes": [
    {
      "type": "extension",
      "app": "msgpack://127.0.0.1:8001/",
      "name": "simple_http_server_cpp",
      "addon": "simple_http_server_cpp",
      "extension_group": "default_extension_group",
      "property": {
        "root_key": "player",
        "extra_keys": ["playerName"]
      }
    }
  ],
  "connections": [
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "simple_http_server_cpp",
      "cmd": [
        {
          "name": "start",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        },
        {
          "name": "stop",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        }
      ]
    },
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "gateway",
      "cmd": [
        {
          "name": "push_status_online",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "uap"
            }
          ]
        }
      ]
    }
  ]
}
```

## Predefined Graph Definition

Predefined graphs are defined under the `predefined_graphs` field in the TEN app's configuration file (`property.json`), including properties such as `name`, `auto_start`, and `singleton`:

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        // Complete graph definition
      }
    ]
  }
}
```

Complete example:

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        "nodes": [
          {
            "type": "extension",
            "name": "simple_http_server_cpp",
            "addon": "simple_http_server_cpp",
            "extension_group": "default_extension_group",
            "property": {
              "root_key": "player",
              "extra_keys": [
                "playerName"
              ]
            }
          }
        ],
        "connections": [
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "simple_http_server_cpp",
            "cmd": [
              {
                "name": "start",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              },
              {
                "name": "stop",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              }
            ]
          },
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "gateway",
            "cmd": [
              {
                "name": "push_status_online",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "uap"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## `start_graph` Command Definition

Dynamic graphs are created by sending a `start_graph` command to the TEN app, placing the graph definition in the command's `ten` field:

```json
{
  "ten": {
    "type": "start_graph",
    // Complete graph definition
  }
}
```

Complete example:

```json
{
  "ten": {
    "type": "start_graph",
    "nodes": [
      {
        "type": "extension",
        "name": "simple_http_server_cpp",
        "addon": "simple_http_server_cpp",
        "extension_group": "default_extension_group",
        "property": {
          "root_key": "player",
          "extra_keys": ["playerName"]
        }
      }
    ],
    "connections": [
      {
        "app": "msgpack://127.0.0.1:8001/",
        "extension": "simple_http_server_cpp",
        "cmd": [
          {
            "name": "start",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          },
          {
            "name": "stop",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          }
        ]
      },
      {
        "extension": "gateway",
        "cmd": [
          {
            "name": "push_status_online",
            "dest": [
              {
                "extension": "uap"
              }
            ]
          }
        ]
      }
    ]
  }
}
```

### Using `import_uri` to Reference Graph Definitions

When defining predefined graphs, you can use the `import_uri` field to reference external graph definition files:

```json
{
  "ten": {
    "uri": "http://localhost:8001",
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": false,
        "import_uri": "../graph.json"
      }
    ]
  }
}
```

`import_uri` can be a relative path, absolute path, or URL:

* Relative path: relative to the directory containing the `property.json` file
* Absolute path: relative to the root directory of the environment where the TEN app is located
* URL: treated directly as a URL path

## Graph Definition Specifications

* **`nodes` Field**: The `nodes` array must be provided in the graph definition. In contrast, while the `connections` array is optional, it is recommended to provide it to define communication between nodes.

* **Node `app` Field**: The `app` field cannot be set to `localhost`. In single-app graphs, the `app` URI should be omitted; in multi-app graphs, the `app` field value must match the `ten.uri` value in each app's `property.json`.

* **Node Uniqueness**: Each node in the `nodes` array represents a specific extension instance and must be uniquely identified by the combination of `app` and `name`. Duplicate definitions are not allowed, as in the following invalid example:

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_1",
        "extension_group": "test"
      },
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_2",
        "extension_group": "test"
      }
    ]
  }
  ```

* **Connection Definition Consistency**: All extension instances referenced in `connections` must be explicitly defined in `nodes`. The following example is invalid because `ext_2` is not defined in `nodes`:

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "ext_1",
        "addon": "addon_1",
        "extension_group": "some_group"
      }
    ],
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **Communication Link Consolidation**: All messages from the same source extension should be grouped in a single section, not scattered across multiple definitions. The following example is incorrect:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      },
      {
        "extension": "ext_1",
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  The correct approach is to consolidate all messages from the same source extension together:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ],
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **Message Destination Consolidation**: For each specific type of message, all destination extensions should be grouped under a single entry. The following example is incorrect:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          },
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  The correct approach is to consolidate all destinations for the same message together:

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              },
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  Note that messages with the same name can coexist in different types (such as `cmd` and `data`) without causing conflicts.

## Subgraph

The core mechanism of the TEN framework is based on graph structures, which consist of nodes and connections. Subgraphs are a powerful reuse mechanism that allows complex graph structures to be split into reusable modules, thereby improving code organization and maintainability.

### Subgraph Design Philosophy

The essence of a graph is to define how data flows between extensions. Subgraphs do not change this fundamental principle in their implementation, but serve as syntactic sugar that is ultimately flattened into the parent graph and started using the same mechanisms as regular graphs. This design simplifies the development of complex systems while not adding runtime complexity.

#### Design Principles

Subgraph design follows these core principles:

1. **Independence**: Any graph is a complete graph that can run independently or be embedded as a component into other graphs.

2. **Tool-Friendly**: Subgraphs provide additional information to help development tools better understand graph structures, improving the development experience while not adding runtime complexity.

3. **Flattening Mechanism**: Subgraphs are ultimately flattened into standard graph structures to ensure performance and compatibility.

4. **Simplicity**: Subgraph design aims to simplify the development process rather than add complexity, avoiding increased JSON structure complexity.

The purpose of subgraph design is to allow developers to use a graph as a black box without needing to understand its internal complexity. Therefore, at the point of referencing subgraphs, no special patching mechanisms are provided to adjust the internal state of subgraphs, to reduce complexity. If subgraph definitions need to be modified, the original definitions should be modified directly rather than patching at the reference point. This is done to avoid complexity spread and increased maintenance costs. For example, when introducing subgraph A into large graph B, if you need to adjust the extension property schema definitions inside subgraph A, you should directly modify the schema definitions of that extension rather than providing modification mechanisms at the reference point, because this might lead to adverse chain reactions, only modifying the extension's schema definitions without patching other places, thus causing errors elsewhere.

In other words, subgraphs primarily provide a mapping mechanism to expose internal elements of subgraphs to the outside, rather than providing mechanisms to patch internal elements of subgraphs.

### Subgraph Implementation Mechanism

Next, we will detail the implementation mechanism of subgraphs, including subgraph definition, reference methods, and the flattening process.

#### Subgraph Definition Example

The following is a `subgraph.json` example that can be used as an independent graph or as a subgraph referenced by other graphs:

```json
{
  "nodes": [
    {
      // Define extension node named ext_c
      "type": "extension",
      "name": "ext_c",
      "addon": "extension_c"
    },
    {
      // Define extension node named ext_d
      "type": "extension",
      "name": "ext_d",
      "addon": "extension_d"
    }
  ],
  "connections": [
    {
      // ext_c sends B command to ext_d
      "extension": "ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_d"
            }
          ]
        }
      ]
    }
  ],
  "exposed_messages": [
    // Represents message interfaces exposed by the graph to the outside, mainly for development tools to provide intelligent hints
    {
      // ext_d's B command exposed to the outside
      "extension": "ext_d",
      "type": "cmd_in",
      "name": "B"
    }
  ],
  "exposed_properties": [
    // Represents properties exposed by the graph to the outside
    {
      "extension": "ext_c",
      "name": "a",
      "alias": "a1"
    }
  ]
}
```

#### Subgraph Reference Example

Example of `graph.json` referencing subgraphs:

```json
{
  "nodes": [
    {
      // Define extension node named ext_a
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // Define extension node named ext_b
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // Reference subgraph, named subgraph_1
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "./ten_packages/extension/aaa/subgraph.json",
      "property": {
        // Properties written in this property field must exist in the subgraph's exposed_properties field,
        // otherwise it will be an error condition
        "app_id": "${env:AGORA_APP_ID}",
        "token": "<agora_token>",
        "channel": "ten_agent_test"
      }
    },
    {
      // Reference subgraph, named subgraph_2
      "type": "subgraph",
      "name": "subgraph_2",
      "import_uri": "./ten_packages/extension/bbb/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // First target is ext_b
              "extension": "ext_b"
            },
            {
              // Second target is ext_d in the subgraph, this is advanced usage
              "extension": "subgraph_1:ext_d"
            },
            {
              // Third target is the subgraph, but this requires the subgraph to explicitly expose the B cmd_in interface,
              // otherwise it will be treated as an error condition
              "subgraph": "subgraph_2"
            }
          ]
        }
      ]
    },
    {
      // ext_c in the subgraph sends cmd H to ext_a, this is advanced usage
      "extension": "subgraph_1:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Subgraph subgraph_2 sends cmd H to ext_a, but this requires the subgraph to expose the H
      // cmd_out interface, otherwise it will be treated as an error condition
      "subgraph": "subgraph_2",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

#### Key Concepts

The subgraph mechanism introduces three key concepts:

1. **Message Exposure** (exposed\_messages):
   * Subgraphs declare externally exposed interfaces through the `exposed_messages` field
   * Mainly used by development tools to implement intelligent hints and checks
   * Hides internal details of subgraphs, improving development experience

2. **Property Exposure** (exposed\_properties):
   * Subgraphs declare externally exposed properties through the `exposed_properties` field
   * Mainly used by development tools to implement intelligent hints and checks
   * Hides internal details of subgraphs, improving development experience

3. **Subgraph Reference and Naming**:
   * Reference other graph files through `"type": "subgraph"`
   * Each subgraph has a unique identifier name serving as a namespace
   * Prevents conflicts between elements with the same name in different subgraphs

4. **Cross-Graph Connections**:
   * Reference elements within subgraphs through namespace syntax (such as `subgraph_1:ext_d`), or reference subgraphs through the `subgraph` field
   * Makes elements within subgraphs interact with the main graph
   * Builds complex cross-graph message flows

In summary, the `exposed_messages` and `exposed_properties` fields describe how messages flow into the graph from the graph boundary, and development tools use these fields to provide intelligent hints, making it convenient for developers to specify how messages flow to the graph boundary.

#### Flattening Mechanism

Ultimately, graphs referencing subgraphs are flattened into regular graph structures, ensuring runtime uniformity and efficiency. However, this flattening result is not seen by users, but is flattened into memory for use in the TEN runtime or TEN manager. Note that in the flattened graph definition, there are the original extensions and the connections between these extensions.

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // ext_c from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "extension_c"
    },
    {
      // ext_d from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "extension_d",
      "property": {
        // The property field defined when referencing the subgraph will be flattened into the corresponding extension's property field
        "app_id": "${env:AGORA_APP_ID}"
      }
    },
    {
      // ext_e from subgraph is flattened with subgraph name as prefix
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "extension_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b"
            },
            {
              "extension": "subgraph_1_ext_d"
            },
            {
              "extension": "subgraph_2_ext_e"
            }
          ]
        }
      ]
    },
    {
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Internal connections within subgraphs are also flattened and included
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "subgraph_1_ext_d"
            }
          ]
        }
      ]
    }
  ]
  // exposed_messages and exposed_properties fields are discarded during flattening
}
```

The subgraph flattening mechanism follows these rules:

1. Before flattening, the colon (`:`) symbol indicates that an element is located within a subgraph (such as `subgraph_1:ext_d`).

2. After flattening, elements within subgraphs have the subgraph name added as a prefix to their names (such as `subgraph_1_ext_c`), ensuring global uniqueness.

3. The flattened graph definition no longer contains colon symbols, distinguishing between pre-flattening and post-flattening states.

4. Internal connections within subgraphs are preserved and included in the flattened graph, ensuring functional completeness.

### Advanced Features and Applications

As project complexity increases, advanced features of subgraphs can help better organize and manage systems.

#### Message Conversion and Subgraphs

Subgraphs fully support the message conversion (msg\_conversion) mechanism for handling message format conversion between different interfaces:

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "http://a.b.c.d/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1:ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "subgraph": "subgraph_2",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Development tools can use `exposed_messages` information to suggest compatibility and provide message conversion configuration interfaces. After configuration, conversion rules are automatically written into the graph definition, simplifying the development process.

After flattening, message conversion rules are correctly preserved, ensuring runtime behavior matches design intent:

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "addon_c"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "addon_d"
    },
    {
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "addon_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1_ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_2_ext_e",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Through this approach, developers can flexibly handle message format differences between different components without needing to understand underlying implementation details.

## Meaning of "subgraph\_1:ext\_c"

In the previous examples, `subgraph_1` in `subgraph_1:ext_c` does not represent the subgraph itself, but references an element in the `nodes` field. In subgraph applications, `subgraph_1` can be viewed as a namespace.

Through this mechanism, you can specify extensions at different locations as sources or targets in graph connections, enabling connections across multiple graphs.

The core concept is: graph connections are essentially connections between extensions. Through this mechanism, you can specify elements of different types in the `nodes` field as sources or targets of connections, enabling connections across multiple subgraphs or even multiple graphs.

## Connections Across Multiple Graphs

In addition to subgraphs, you can also specify connections to other graphs within a single graph through similar mechanisms.

### Connecting to Predefined Graphs in the Same TEN App

Example: How to connect to predefined graphs within the same TEN app in a graph:

```json
{
  "nodes": [
    {
      // Define extension named ext_a
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // Define extension named ext_b
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // Reference other graph, graph ID is default
      "type": "graph",
      "graph_id": "default"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // First target is ext_b
              "extension": "ext_b"
            },
            {
              // Second target is ext_d in another graph
              "extension": "default:ext_d"
            }
          ]
        }
      ]
    },
    {
      // ext_c in another graph sends cmd H to ext_a
      "extension": "default:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // Another graph sends cmd X to ext_a, but this requires the other graph to expose the X cmd_out interface,
      // otherwise it will be treated as an error condition. This error condition can be reported during static
      // checking if it can be statically detected, or at runtime.
      "graph": "default",
      "cmd": [
        {
          "name": "X",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

### Types of Graph Nodes

1. Subgraph

   ```json
   {
     "type": "subgraph",
     "name": "subgraph_1", // Meaning of namespace
     "import_uri": "http://a.b.c.d/subgraph.json"
   }
   ```

2. Local graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": true
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": false
   }
   ```

3. Remote graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": true,
     "app": "msgpack://127.0.0.1:8002/"
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singleton": false,
     "app": "msgpack://127.0.0.1:8002/"
   }
   ```


# Message Conversion (/docs/ten_framework/0.11.60/graph/msg_conversion)

## Message Conversion

TEN Framework provides powerful message conversion capabilities that allow you to transform and modify message content during message passing. By configuring conversion rules, you can achieve message field renaming, value replacement, and complex data structure transformations.

### Basic Configuration

Message conversion is configured through the `msg_conversion` configuration item:

```json
{
  "msg_conversion": {
    "type": "per_property",
    "keep_original": false,
    "rules": [
      {
        "path": "user.name",
        "conversion_mode": "fixed_value",
        "value": "default_user"
      },
      {
        "path": "data.items[0].id",
        "conversion_mode": "from_original",
        "original_path": "original_id"
      }
    ]
  }
}
```

### Configuration Parameters

* **type**: Conversion type, currently supports `per_property` (per-property conversion)
* **keep\_original**: Whether to retain original message content; when set to `false`, only converted fields are preserved
* **rules**: Array of conversion rules that define specific conversion logic

### Conversion Rules

Each conversion rule contains the following fields:

* **path**: Target field path, specifying the message field location to be set
* **conversion\_mode**: Conversion mode
  * `fixed_value`: Set a fixed value
  * `from_original`: Copy value from a specified path in the original message
* **value**: Fixed value used when conversion mode is `fixed_value`
* **original\_path**: When conversion mode is `from_original`, specifies the source field path in the original message

### Path Expressions

Path expressions in message conversion support multiple access methods:

* **Object Property Access**: `user.profile.name` - Access properties of nested objects
* **Array Index Access**: `items[0].name` - Access elements at specific array indices
* **Mixed Path Access**: `user.tags[1].value` - Combine object and array access

### Use Cases

Message conversion functionality is suitable for the following scenarios:

* **Field Renaming**: Convert field names in messages to formats required by target systems
* **Data Format Conversion**: Adjust message data structures to adapt to different processing modules
* **Default Value Setting**: Provide default values for missing fields
* **Data Cleansing**: Filter or transform data content that doesn't meet requirements


# Go (/docs/ten_framework/0.11.60/language_binding/go)

Let's build a Go application using the TEN framework.

## Creating the TEN App

First, we'll create a basic TEN Go app by integrating several pre-built TEN packages. Follow these steps:

```shell title="Terminal"
tman install app default_app_go
cd default_app_go

tman install protocol msgpack
tman install extension_group default_extension_group
```

### Installing a Default TEN Extension

Next, install a default TEN extension written in Go:

```shell title="Terminal"
tman install extension default_extension_go
```

### Declaring the Prebuilt Graph for Auto-Start

Now, we'll modify the `property.json` file of the TEN app to include a graph declaration. This will ensure the default extension starts automatically when the TEN app is launched.

```json title=".json"
"predefined_graphs": [
  {
    "name": "default",
    "auto_start": true,
    "nodes": [
      {
        "type": "extension_group",
        "name": "default_extension_group",
        "addon": "default_extension_group"
      },
      {
        "type": "extension",
        "name": "default_extension_go",
        "addon": "default_extension_go",
        "extension_group": "default_extension_group"
      }
    ]
  }
]
```

## Building the App

Unlike standard Go projects, the TEN Go app uses CGo, so you need to set up certain environment variables before building. A build script is already provided in the TEN runtime Go binding system package, so you can build the app with a single command:

```shell title="Terminal"
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```

The compiled binary, `main`, will be generated in the `/bin` folder.

## Starting the App

Since some environment variables need to be set, it is recommended to start the app using the provided script:

```shell title="Terminal"
./bin/start
```

## Debugging

If you are using Visual Studio Code (VSCode) as your development environment, you can use the following configurations to debug Go/C code.

### Debugging Go Code

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Go app (launch)",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "cwd": "${workspaceFolder}",
      "output": "${workspaceFolder}/bin/tmp",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

### Debugging C Code

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "C (launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/main",
      "cwd": "${workspaceFolder}",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

## CGO

### Generated Code

When interfacing Go with C, the cgo tool is crucial. When a C function is called from Go, cgo converts the Go source files into multiple output files, including both Go and C source files. These generated C source files are then compiled using compilers like gcc or clang.

Use the following command to generate these C/Go source files:

```bash title="Terminal"
mkdir out
go tool cgo -objdir out
```

Example:

```bash title="Terminal"
go tool cgo -objdir out cmd.go error.go
```

The generated files include:

```bash title="Terminal"
├── _cgo_export.c
├── _cgo_export.h
├── _cgo_flags
├── _cgo_gotypes.go
├── _cgo_main.c
├── _cgo_.o
├── cmd.cgo1.go
├── cmd.cgo2.c
├── error.cgo1.go
└── error.cgo2.c
```

* **\_cgo\_export.h** is key to the interoperability between Go and C within the cgo context. It contains necessary declarations for Go functions accessible from C.

  This file is automatically generated based on Go functions explicitly exported using the `//export` directive. It also includes definitions for C types corresponding to the Go types used in these functions, ensuring compatibility between the two languages.

  Example of type definitions:

```c title=".h"
typedef signed char GoInt8;
typedef unsigned char GoUint8;
typedef short GoInt16;
typedef unsigned short GoUint16;
typedef int GoInt32;
typedef unsigned int GoUint32;
typedef long long GoInt64;
typedef unsigned long long GoUint64;
typedef GoInt64 GoInt;
typedef GoUint64 GoUint;
typedef size_t GoUintptr;
typedef float GoFloat32;
typedef double GoFloat64;
```

* **\_cgo\_gotypes.go** contains the corresponding Go types defined in C and used in Go.

  For example, if you define a struct `ten_go_error_t` in a header file `common.h` and use `C.ten_go_error_t` in Go, there will be a corresponding Go type in `_cgo_gotypes.go`:

```go title=".go"
package ten

type _Ctype_ten_go_status_t = _Ctype_struct_ten_go_status_t

type _Ctype_struct_ten_go_status_t struct {
    err_no   _Ctype_int
    msg_size _Ctype_uint8_t
    err_msg  [256]_Ctype_char
    _        [3]byte
}
```

* **cmd.cgo1.go** is a cgo-generated counterpart to the original Go source file `cmd.go`. It replaces direct calls to C functions and types with calls to the generated Go functions and types provided by cgo.

  Example:

```go title=".go"
package ten

  func NewCmd(cmdName string) (Cmd, error) {
      // ...
      cStatus := func() _Ctype_struct_ten_go_status_t {
          var _cgo0 _cgo_unsafe.Pointer = unsafe.Pointer(unsafe.StringData(cmdName))
          var _cgo1 _Ctype_int = _Ctype_int(len(cmdName))
          _cgoBase2 := &bridge
          _cgo2 := _cgoBase2
          _cgoCheckPointer(_cgoBase2, 0 == 0)
          return _Cfunc_ten_go_cmd_create_custom_cmd(_cgo0, _cgo1, _cgo2)
      }()
      // ...
  }
```

* **cmd.cgo2.c** is a wrapper of the original C function called from Go.

  Example:

```c title=".c"
  CGO_NO_SANITIZE_THREAD void _cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd(void *v) {
      struct {
          const void* p0;
          int p1;
          char __pad12[4];
          ten_go_msg_t** p2;
          ten_go_error_t r;
      } __attribute__((__packed__, __gcc_struct__)) *_cgo_a = v;
      char *_cgo_stktop = _cgo_topofstack();
      __typeof__(_cgo_a->r) _cgo_r;
      _cgo_tsan_acquire();
      _cgo_r = ten_go_cmd_create_cmd(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2);
      _cgo_tsan_release();
      _cgo_a = (void*)((char*)_cgo_a + (_cgo_topofstack() - _cgo_stktop));
      _cgo_a->r = _cgo_r;
      _cgo_msan_write(&_cgo_a->r, sizeof(_cgo_a->r));
  }
```

So, the calling sequence of `C.ten_go_cmd_create_cmd()` from Go is:

```text
_Cfunc_ten_go_cmd_create_custom_cmd (auto-generated Go)
           |
           V
   _cgo_runtime_cgocall (Go)
           |
           V
      entrysyscall     // switch Go stack to C stack
           |
           V
_cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd (auto-generated C)
           |
           V
   ten_go_cmd_create_cmd (C)
           |
           V
      exitsyscall
```

### Incomplete Type

As mentioned earlier, the cgo tool generates corresponding Go types in `_cgo_gotypes.go` based on C header files imported via `import "C"`. If a struct is opaque in the C header, the cgo tool generates an incomplete type instead.

Example of an opaque C struct:

```c title=".h"
typedef struct ten_go_msg_t ten_go_msg_t;
```

The cgo tool will generate an incomplete type in Go:

```go title=".go"
type _Ctype_ten_go_msg_t = _Ctype_struct_ten_go_msg_t
type _Ctype_struct_ten_go_msg_t _cgopackage.Incomplete
```

#### What happens if you use the incomplete type in Go?

* **Incomplete types cannot be allocated.**

  Since `sys.NotInHeap` cannot be allocated on the Go heap, operations like `new` or `make` won't work. Attempting to create a new instance of an opaque struct in Go will result in a compiler error:

```go title=".go"
msg := new(C.ten_go_msg_t) // Error: can't be allocated in Go; it is incomplete (or unallocatable)
```

* **Pointers to incomplete types cannot be passed to C directly.**

  If you have a C function with a pointer to an opaque struct as a parameter, passing a Go pointer to this incomplete type directly to a C function will not work according to cgo rules. The Go compiler will require the pointer to be "pinned" to ensure it adheres to Go's garbage collector (GC) constraints.

### Rules for Using C.uintptr\_t Instead of a Pointer to an Opaque Struct

* **Benefits:**

  * `C.uintptr_t` and `uintptr` in Go are integers large enough to hold a C pointer, avoiding memory allocation or conversion when passing from Go to C.

* **Limitations:**
  * `uintptr` is an integer, not a pointer. It represents the bit pattern of an address with no pointer arithmetic.
  * `uintptr` cannot be dereferenced in Go. Converting it to `unsafe.Pointer` can cause issues with Go's GC.
  * Since `uintptr` is an integer, `nil` or `NULL` cannot be passed to C. Use `0` instead of `nil` to represent a null address.


# Manifest (/docs/ten_framework/0.11.60/metadata_system/manifest)

## What is a Manifest File

A Manifest file is a crucial configuration file in the TEN Framework used to store metadata for software packages. It defines essential information about Extensions or Apps, including basic information, dependencies, API interfaces, script configurations, and other key details.

## Purpose of Manifest Files

* **Package Identification**: Define the package type, name, and version information
* **Dependency Management**: Declare required dependencies and their version requirements
* **API Definition**: Describe data interfaces and command interfaces provided by the package
* **Package Configuration**: Specify files to be included in the release version
* **Script Management**: Define automation scripts for building, testing, and other tasks

## Extension Manifest Example

Here's a complete Manifest example for an AI chat extension:

```json
{
  "type": "extension",
  "name": "openai_chatgpt_python",
  "version": "0.1.0",
  "tags": ["ai", "llm", "python"],
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "^0.8.0"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "*.py",
      "README.md"
    ]
  },
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        }
      }
    },
    "data_in": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "data_out": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_in": [
      {
        "name": "flush"
      },
      {
        "name": "tool_register",
        "property": {
          "properties": {
            "tool": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "description": {
                  "type": "string"
                },
                "parameters": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                }
              },
              "required": ["name", "description", "parameters"]
            }
          }
        },
        "result": {
          "property": {
            "properties": {
              "response": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "tool_call",
        "property": {
          "properties": {
            "name": {
              "type": "string"
            },
            "args": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    ]
  },
  "scripts": {
    "test": "python -m pytest tests/",
    "build": "python setup.py build"
  }
}
```

## App Manifest Example

Here's a Manifest example for a voice assistant application:

```json
{
  "type": "app",
  "name": "voice_assistant_app",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "^0.8.0"
    }
  ],
  "api": {
    "property": {
      "properties": {
        "app_id": {
          "type": "string"
        },
        "log_level": {
          "type": "uint8"
        },
        "server_config": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "int32"
            },
            "ssl_enabled": {
              "type": "bool"
            }
          },
          "required": ["host", "port"]
        }
      }
    }
  }
}
```

## Key Field Descriptions

### Basic Information

* **type**: Package type, can be `extension` or `app`
* **name**: Unique identifier name for the package
* **version**: Version number following semantic versioning standards
* **tags**: Array of tags for categorization and search

### Dependency Configuration

* **dependencies**: List of dependencies, each containing type, name, and version requirements

### Package Configuration

* **package.include**: Specifies files and directories to include during packaging

### API Configuration

* **api.property**: Defines configuration properties for the extension
* **api.data\_in/data\_out**: Defines data input and output interfaces
* **api.cmd\_in/cmd\_out**: Defines command input and output interfaces

### Script Configuration

* **scripts**: Defines various automation scripts such as testing, building, etc.

## Best Practices

1. **Version Management**: Use semantic versioning to ensure version consistency
2. **Dependency Declaration**: Clearly declare all required dependencies and their version ranges
3. **API Design**: Clearly define input and output interfaces for easy integration with other components
4. **Complete Documentation**: Ensure that configurations in the Manifest are consistent with actual implementation


# Metadata System (/docs/ten_framework/0.11.60/metadata_system/metadata_system)

The TEN framework utilizes a consistent metadata system across various types of packages, including:

* App
* Extension Group
* Extension
* Protocol
* System

## Types of Metadata

In the TEN metadata system, there are two primary types of metadata:

### 1. **Manifest**

* **Location**: Stored in the root directory of the associated TEN package with the file name `manifest.json`.
* **Contents**:
  * **Package Name**: The name of the TEN package.
  * **Package Version**: The version of the TEN package, following semantic versioning.
  * **API Schema**: Defines schemas related to the package's properties and input/output messages.
    * **Property Schema**: These schemas are often defined in a `property.json` file in the root directory. The manifest can specify the schema for these properties.
    * **Message Schema**: Defines the schema for input/output messages handled by the package.

> ⚠️ **Note**: The API schema in `manifest.json` is *not* a JSON schema. Instead, it describes the metadata of TEN values, which are central to the TEN runtime, while JSON is merely a representation format.

### 2. **Property**

* **Location**: Typically stored in a `property.json` file in the root directory of the TEN package.
* **Purpose**: The `property.json` file stores initial property values, which are read-write during runtime. This means that properties can be modified while the TEN runtime is executing.

#### Example of `property.json`

```json
{
  "Darth Vader": "I am your father"
}
```

## Manifest File

The `manifest.json` file serves as the blueprint for the TEN package, defining its metadata, properties, and the input/output messages it handles.

### Example of `manifest.json`

```json
{
  "type": "app",
  "name": "default_app_cpp",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.0"
    }
  ],
  "api": {
    "property": {
      "exampleInt8": {
        "type": "int8"
      },
      "exampleString": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "detail": {
              "type": "string"
            },
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

### Purpose of TEN Schema in Manifest

The TEN schema in `manifest.json` provides the TEN runtime with metadata about the extension's external API, including the type information for properties and messages. This helps the runtime platform correctly handle the extension's properties and messages during operation.

### When TEN Schema is Used

1. **Property Validation**: When the TEN runtime gets/sets properties of a TEN package or message.
2. **Data Conversion**: When converting a JSON document to a TEN package or message property using the `from_json` API.
3. **Compatibility Check**: When the TEN manager checks if a message can be output from a source extension to a destination extension according to the TEN schema.

## Property Management

In the TEN framework, there are two types of properties:

1. **Message Properties**: These properties are associated with the messages exchanged between extensions within the framework. Message properties define the specific data or parameters that are carried within a message, such as command parameters, data payloads, or metadata related to audio/video frames.

2. **TEN Package Properties**: These properties are associated with the TEN packages themselves, such as extensions. TEN package properties define the configuration or settings specific to a package. For example, an extension might have properties that configure its behavior, such as runtime settings, initialization parameters, or other configuration data.

Both types of properties are managed within the TEN framework but serve different purposes, one focuses on the communication between components (message properties), while the other pertains to the configuration and operation of the components themselves (TEN package properties).

<figure><img src="..https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_system.png" alt=""><figcaption><p>Property system</p></figcaption></figure>

### Defining TEN Package Properties

The `property.json` file defines the properties of a TEN package. Here's an example:

```json
{
  "prop_1_name": 0,
  "prop_2_name": "prop_2_value",
  "prop_3_name": ["hello", "prop_3_sub_value"],
  "prop_4_name": {
    "prop_4_1_name": 1,
    "prop_4_2_name": "prop_4_sub_value"
  }
}
```

> ⚠️ **Note**: Each property name in the `property.json` file must be unique.

### TEN Schema for Properties

You can define a TEN schema for properties in the `manifest.json` file, enabling the TEN runtime to handle these properties more effectively. If a property does not have a corresponding TEN schema, the runtime platform will use the default JSON handling method (e.g., treating all JSON numbers as `float64`). If a TEN schema is provided, the runtime platform will use it to validate and process the property accordingly.

| Property | TEN Schema | Effect                                                                                          |
| -------- | ---------- | ----------------------------------------------------------------------------------------------- |
| Yes      | Yes        | The TEN runtime validates the property value based on the TEN schema (e.g., checking the type). |
| Yes      | No         | The TEN runtime uses default handling, treating all JSON numbers as `float64`.                  |

### Accessing Properties from an Extension

The TEN runtime provides APIs for extensions to access various properties.

### Accessing App Properties from an Extension

> ⚠️ **Note**: The behavior described in this section has not yet been implemented and is part of the future roadmap.

The TEN framework provides a powerful property access mechanism that allows extensions to read and modify app properties. Through the get/set property API, you can access all properties, including app properties. When the API parameter begins with the `app:` prefix, the system identifies this as a request to operate on app properties.

API design considerations for different language implementations:

* **Go**: Providing a synchronous version without callbacks is sufficient, making it concise and efficient.
* **Node.js**: Implemented as an async function without callbacks, aligning with Node.js's asynchronous programming model.
* **Python**: Ideally, versions both with and without callbacks should be provided. Currently, a synchronous version without callbacks can be implemented first, but accessing `app:` properties will use event wait, which may impact performance.
* **Async Python**: Implemented as an async function without callbacks, consistent with Python's asynchronous model.
* **C++**: Both versions with and without callbacks should be provided. Currently, a synchronous version without callbacks can be implemented first, but accessing `app:` properties will use event wait, which may impact performance.

This design enables CRUD operations on app properties through the unified get/set\_property API, maintaining interface consistency without affecting message property-related APIs.

Regarding the handling of `ten:` namespace fields, the following rules are adopted:

1. `get/set_property_to/from_json(nullptr)` operations do not process `ten:` fields
2. To access `ten:` fields, you must explicitly specify the complete path such as `get/set_property("ten:xxx")`
3. When using `get/set_property_to/from_json()`, `ten:` fields are only processed when a key in the form of `"ten:xxx"` is explicitly specified

This design maintains API simplicity while avoiding the complexity of creating additional commands for specific requirements.

### Specifying Property Values in the `start_graph` Command

Property values related to the TEN package can be specified in the `start_graph` command. The TEN runtime processes these properties according to the TEN schema (if defined) and stores them within the corresponding TEN package instance.

#### Example of Property Specification in `start_graph` Command

```json
{
  "nodes": [
    {
      "type": "extension_group",
      "name": "foo_extension_group",
      "addon": "foo_extension_group_addon"
    },
    {
      "type": "extension",
      "name": "bar_extension",
      "extension_group": "foo_extension_group",
      "property": {
        "prop_1_name": 0,
        "prop_2_name": "prop_2_value",
        "prop_3_name": ["hello", "prop_3_sub_value"],
        "prop_4_name": {
          "prop_4_1_name": 1,
          "prop_4_2_name": "prop_4_sub_value"
        }
      }
    }
  ]
}
```


# Property (/docs/ten_framework/0.11.60/metadata_system/property)

## Property File Overview

The Property file is a configuration file in the TEN Framework used to store attribute values for extensions. It defines runtime parameters and configuration options for extensions, serving as an essential component for their proper operation.

## Property File Format

Property files use JSON format and contain various configuration parameters required for extension operation, such as API keys, model configurations, runtime parameters, etc.

### Basic Property Example

Here's a simple Property file example:

```json
{
  "api_key": "your-api-key-here",
  "max_tokens": 1000,
  "temperature": 0.7,
  "model_config": {
    "name": "gpt-4",
    "version": "2024-01-01"
  },
  "allowed_languages": ["en", "zh", "es"]
}
```

### AI Model Configuration Example

Property file configuration example for AI model extensions:

```json
{
  "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxx",
  "model": "gpt-4",
  "max_tokens": 4096,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "prompt": "You are a helpful AI assistant.",
  "greeting": "Hello! How can I help you today?",
  "max_memory_length": 10,
  "vendor": "openai"
}
```

## Graph Configuration

Property files can also contain Graph Configuration, which defines connection relationships and data flow between extensions.

### Predefined Graph Configuration Example

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "nodes": [
          {
            "type": "extension",
            "name": "my_extension",
            "addon": "my_extension_addon",
            "property": {
              "setting1": "value1",
              "setting2": 42
            }
          }
        ],
        "connections": [
          {
            "extension": "source_extension",
            "cmd": [
              {
                "name": "process_cmd",
                "dest": [
                  {
                    "extension": "target_extension"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## Usage Guidelines

* Property files must be in valid JSON format
* Property values are loaded when the extension starts
* Extension behavior can be adjusted by modifying the Property file
* The graph configuration section defines connections and data flow between extensions


# Addon System (/docs/ten_framework/0.11.60/software_packages/addon_system)

The TEN framework is a software module system, where each software module is defined by the concept of an **addon**. Essentially, every software module in the TEN framework is an addon. For example, users can define the following types of TEN framework addons:

1. TEN **extension** addon
2. TEN **extension group** addon
3. TEN **protocol** addon

## Information Required to Load Addon

To load an addon into the TEN runtime, three pieces of information are needed:

1. **Addon name**

   It must be globally unique within the entire TEN app. Loading multiple addons with the same name into the TEN runtime is undefined behavior, and it may result in the second instance of the addon not being properly loaded into the TEN runtime.

2. **Addon folder location**

   In the TEN framework, when loading an addon, users must inform the TEN runtime where the addon's directory is located. This directory serves as the addon's `base_dir`. The TEN runtime will use this base directory to load other files related to the addon, such as the addon's `manifest.json` and `property.json` files.

   Basically, when designing the TEN framework's addon register API, it aims to automatically detect the directory where the addon is located, so the user does not need to explicitly specify the `base_dir`. However, in some special cases where the TEN framework cannot automatically detect the `base_dir`, the user will need to explicitly specify the folder where the addon is located for the TEN runtime.

3. **Addon instance itself**

## Timing for Registering Addon to TEN Runtime

There are two ways to register an addon into the TEN runtime:

1. Explicitly call the API to register the addon's `base_dir` to the TEN runtime.

   **Note** Currently, only the standalone testing framework provides such an API.

2. Place the addon in a designated location within the TEN app folder tree.

By placing the addon in the specified location, the TEN app will automatically load these addons when it starts. Currently, this designated location is under the `ten_packages/` directory in the TEN app. If the addons are placed in this format, they will be automatically loaded when the TEN app starts.

```text
.
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    └── protocol/
        └── <protocol_a>/
```


# App (/docs/ten_framework/0.11.60/software_packages/app)

App is the container in the TEN world, containing groups and extensions.
App can function as a process placed inside a Docker container (primarily used in the cloud).
App can also function as a thread embedded within an existing app (on client or server).
⇒ TEN can perform localized modifications (minor enhancements) on an existing project without requiring the entire project to be rewritten according to TEN.

![Run TEN as a Process or a Thread](https://ten-framework-assets.s3.amazonaws.com/doc-assets/run_ten_as_process_or_thread.png)

![Different mode of app running](https://ten-framework-assets.s3.amazonaws.com/doc-assets/different_mode_of_app_runnings.png)

## TEN App Folder Structure

In the TEN framework, developers can create TEN packages using various programming languages. Regardless of the language used, the overall folder structure of a TEN app remains consistent, ensuring uniformity across different implementations.

![TEN package folder structure](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_package_folder_structure.png)


# Extension (/docs/ten_framework/0.11.60/software_packages/extension)

## Life Cycle

The life cycle of an extension is divided into the following stages:

1. `on_configure`
2. `on_init`
3. `on_start`
4. `on_stop`
5. `on_deinit`

At all these stages, the extension can send messages, and if the sent message is a command type, the extension can also receive the result of the command it sent. In other words, there is no stage where the extension *cannot* send messages or receive the results of its own sent commands. So, in all these stages, the extension can send messages to other extensions, which helps implement dependencies between extensions in all stages.

Each life cycle stage corresponds to a callback function, and there is a corresponding `on_xxx_done()` function to mark the end of that life cycle stage.

| Life Cycle Callback | End of Life Cycle Function |
| ------------------- | -------------------------- |
| `on_configure`      | `on_configure_done`        |
| `on_init`           | `on_init_done`             |
| `on_start`          | `on_start_done`            |
| `on_stop`           | `on_stop_done`             |
| `on_deinit`         | `on_deinit_done`           |

### Stage Transition Graph

```mermaid
graph TD;
  on_configure((on_configure));
  on_init((on_init));
  on_start((on_start));
  on_stop((on_stop));
  on_deinit((on_deinit));

  on_configure-->on_configure_done
  on_configure_done-->on_init;
  on_configure-->on_stop;
  on_init-->on_init_done;
  on_init_done-->on_start;
  on_init-->on_stop;
  on_start-->on_stop;
  on_start-->on_start_done;
  on_stop-->on_stop_done;
  on_stop_done-->on_deinit;
  on_deinit-->on_deinit_done;
```

### on\_configure

Used to set the initial values of the extension's properties. This allows the `get_property` API from `ten_env` to be used in other lifecycle stages to retrieve properties set during the `on_configure()` stage. The `on_configure_done()` function is called to mark the end of the `on_configure` stage.

```c++
  void on_configure(ten::ten_env_t &ten_env) override {
    // Set the initial values of the extension's properties.
    ten_env.init_property_from_json(
      R"({
           "ten": {
             "uri": "msgpack://127.0.0.1:8001/",
             "log_level": 2
           }
         })", nullptr);
    ten_env.on_configure_done();
  }
```

### on\_init

Used to initialize the extension. Since the extension is not yet ready before `on_init_done()`, the TEN runtime will queue all messages sent to the extension until it's ready. Once ready, the queued messages are delivered to the extension. It's important to note that if the result is from a command sent by the extension itself, it is *not* subject to this restriction; the TEN runtime will directly pass the command's result back to the extension. This is because sending a command is a self-initiated action by the extension, so it must be prepared to receive and handle the result of its own commands.

### on\_start

This stage marks the runtime starting to send messages into the extension. The `on_start_done` function doesn't have a special purpose, but for consistency with other life cycle stages and to reduce learning complexity, the TEN framework includes it. Typically, `on_start_done` is invoked at the end of the `on_start` callback function. The TEN runtime will start delivering messages from other extensions to this extension after receiving its `on_start_done()`.

```c++
  void on_start(ten::ten_env_t &ten_env) override {
    // Some operations.
    ten_env.on_start_done();
  }
```

### on\_stop

There are many situations where the extension needs to stop, such as when the app or engine containing the extension is terminating. When the extension is about to stop, the TEN runtime uses this `on_stop` callback to notify the extension that it has entered the `on_stop` life cycle stage. The extension can then perform any necessary actions for termination.

### on\_deinit

After the extension calls `on_stop_done()`, it enters the `on_deinit` stage. During this stage, because the resources within the extension may no longer be fully available, the TEN runtime will not pass any messages from other extensions to this one.

## Relationship Between Extensions at Different Life Cycle Stages

Basically, there is no inherent relationship. Each extension operates independently, switching between its own life cycle stages. Extensions are independent of one another, and any dependencies between them must be explicitly implemented by the extensions themselves. The TEN runtime does not make any assumptions or guarantees. For example, if extension A needs to wait for extension B to complete its initialization before finishing its own, extension A can send a command to extension B during its `on_init`. Once extension B completes initialization and receives the command, it can reply with a result, and when extension A receives the result, it can call `on_init_done`.

```mermaid
sequenceDiagram
  participant A as Extension A
  participant B as Extension B

  Note over A,B: on_init
  activate A
  activate B
  deactivate B
  Note over B: on_init_done

  Note over B: on_start
  activate B
  deactivate B
  Note over B: on_start_done

  A->>B: cmd
  B-->>A: result

  deactivate A
  Note over A: on_init_done
```

Basically, in all other life cycle stages, if you want to implement the order of different extensions, you can achieve it using the same method.

## Interface with TEN Runtime

Extensions interact with the TEN runtime primarily through three interfaces:

1. **Lifecycle Callbacks**

   * These include callbacks like `on_init`, `on_deinit`, `on_start`, and `on_stop`.

2. **Callbacks for Receiving Messages**

   * These include callbacks such as `on_cmd`, `on_data`, `on_audio_frame`, and `on_video_frame`, which handle incoming messages to the extension.

3. **Functions for Sending Messages**

   * These include functions like `send_cmd`, `send_data`, `send_audio_frame`, and `send_video_frame`, which send messages out of the extension.

## Lifecycle Callbacks

The different stages of the extension's lifecycle and their connection to message handling are as follows:

* **on\_init \~ on\_init\_done**: Handles the extension's own initialization. At this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions.
* **on\_start \~ on\_start\_done**: At this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions. Since properties are initialized during `on_configure`, you can perform actions that depend on these properties being set up. However, as this is still part of the initialization phase, the extension will not receive messages initiated by others, avoiding the need for various checks. Active message sending is allowed.
* **After on\_start\_done \~ on\_stop\_done**: During this phase, the extension can normally send and receive all types of messages and their results.
* **on\_deinit \~ on\_deinit\_done**: Handles the extension's de-initialization. Similar to the `on_init` stage, at this stage, the extension can send messages and receive the results of commands it sends, but cannot receive messages actively sent by other extensions.

## Implementing Extensions in Different Languages

Within the TEN framework, extensions can be implemented in various languages such as C++, Go, and Python. Developers can use the same conceptual approach to implement extensions in different languages. Learning how to develop an extension in one language makes it relatively easy to do so in other languages as well.

## Asynchronous Message Processing in Extensions

<figure><img src="../../..https://ten-framework-assets.s3.amazonaws.com/doc-assets/asynchronous_message_processing.png" alt=""><figcaption><p>Asynchronous Message Processing</p></figcaption></figure>

Extensions process messages asynchronously. When the TEN runtime delivers a message to an extension through callbacks like `on_cmd`, `on_data`, `on_audio_frame`, or `on_video_frame`, the extension is not required to process the message immediately within the callback. Instead, the extension can delegate the message to other threads, processes, or even machines for processing. This allows for full utilization of multi-core and distributed computing resources.

After processing is complete, the results can be sent back to the TEN runtime through callbacks such as `send_cmd`, `send_data`, `send_audio_frame`, or `send_video_frame`. The entire process is asynchronous, meaning the extension doesn't need to send the processed results back before the `on_cmd`, `on_data`, `on_audio_frame`, or `on_video_frame` callbacks return. The results can be transmitted only when they are actually ready, using the appropriate send functions.


# Overview (/docs/ten_framework/0.11.60/software_packages/overview)

The directory structure of a basic TEN framework software package is shown below.

```text
.
├── bin/
├── src/
├── manifest.json
├── manifest-lock.json
├── property.json
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    ├── protocol/
    │   └── <protocol_a>/
    └── system/
        ├── <system_package_1>/
        └── <system_package_2>/
```

Depending on the programming language, there may be additional required files. For example, in a C++ project, since the build system is `ten_gn`, a `BUILD.gn` file would be present. In a Go project, there would be files such as `go.mod` and `go.sum`. In a Python project, you might find a `requirements.txt` file.

## TEN Framework Software Package Types

| Type            | Description                     |
| --------------- | ------------------------------- |
| App             | Contains a TEN app.             |
| Extension group | Contains a TEN extension group. |
| Extension       | Contains a TEN extension.       |
| Protocol        | Contains a TEN protocol.        |
| System          | Contains a TEN system package.  |

## TEN Framework Software Package Kinds

There are two types of software packages:

* Release package
* Development package

The relationship between these two package types is as follows:

* A release package can be generated from a development package through a build process.
* If we add source-related content to a release package, it becomes a development package.
* If we remove source-related content from a development package, it becomes a release package.

In short, the directory structures of development and release packages are very similar, with the difference being that a development package includes source-related content, whereas a release package does not.

### Development package

The primary purpose of a development package is to develop TEN software packages. For instance, if you want to create a new TEN extension, you can modify an existing development package to achieve this. This is because the development package contains all the source-related content, enabling the creation of new TEN software packages from it.

### Release package

The primary purpose of a release package is to create and run graphs. Since no build tasks are required, a release package can be used to perform these actions. However, a development package can also be used for these tasks. Essentially, the application scope of a development package is broader than that of a release package.


# Python Async Extension (/docs/ten_framework/0.11.60/software_packages/python_async_extension)

A **Python async extension** is a modular component that using Python's `asyncio` framework. If you want to wrap existing Python codes that use `asyncio` into a TEN extension, using the Python async extension is the simplest and most convenient option.

## Example: The Default Python Async Extension

Here’s how a Python async extension is structured:

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    async def on_configure(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_json = cmd.to_json()
        ten_env.log_debug(f"DefaultAsyncExtension on_cmd: {cmd_json}")

        # Mock async operation, e.g. network, file I/O.
        await asyncio.sleep(0.5)

        # Send a new command to other extensions and wait for the result. The
        # result will be returned to the original sender.
        new_cmd = Cmd.create("hello")
        cmd_result = await ten_env.send_cmd(new_cmd)
        ten_env.return_result(cmd_result, cmd)
```

Each method simulates a delay using `await asyncio.sleep()`.

## FAQ

### Aysnc loop for event handling

* Create a queue: use `asyncio.Queue`.
* Create an async task for event handling.

Here is the sample code:

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    queue = asyncio.Queue()
    loop:asyncio.AbstractEventLoop = None

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        self.loop = asyncio.get_event_loop()

        self.loop.create_task(self._consume())

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        self.queue.put(None)

    async def _consume(self) -> None:
        while True
            try:
                value = await self.queue.get()
                if value is None:
                    self.ten_env.log_info("async loop exit")
                    break

                # Code for processing values retrieved from the queue.

            except Exception as e:
                self.ten_env.log_error(f"Failed to handle {e}")
```

## Conclusion

TEN's Python async extension provide a powerful way to handle long-running tasks asynchronously. By integrating Python’s `asyncio` framework, the extensions ensure that operations such as network calls or file handling are efficient and non-blocking. This makes TEN a great choice for building scalable, modular applications with asynchronous capabilities.


# Standard Interface (/docs/ten_framework/0.11.60/standard_extension/standard_interface)

In the TEN framework, we encourage developers to create their own extensions and publish them to the TEN extension marketplace for other developers to use. To enable developers to easily select and replace extensions, we need to define standardized interface specifications so that developers can quickly find extensions that meet their requirements based on these standard interfaces.

In short, standard interfaces are syntactic sugar that allows TEN tools (such as TEN Manager) to more conveniently assist developers in selecting and replacing extensions.

TEN's interface definitions are similar to TypeScript's `.d.ts` files, allowing the TypeScript compiler to perform type checking, but these interface definitions are not used at runtime. Therefore, the functionality and code enhancements brought by standard interface design are mainly reflected in development tools like TEN Manager, rather than the runtime TEN Runtime. For example, TEN Manager needs to provide relevant RESTful APIs to tman designer Frontend, allowing users to select and replace extensions through the UI interface based on standard interface definitions.

## What are Standard Interfaces

In the TEN framework, each extension's interface consists of the following core components:

1. **Properties** - Configuration parameters of the extension
2. **Command**
   * **Input command** - Input commands
   * **Output command** - Output commands
3. **Data**
   * **Input data** - Input data
   * **Output data** - Output data
4. **Audio frame**
   * **Input audio frame** - Input audio frames
   * **Output audio frame** - Output audio frames
5. **Video frame**
   * **Input video frame** - Input video frames
   * **Output video frame** - Output video frames

### Significance and Purpose of Standard Interfaces

Standard interfaces normalize these components into unified interface standards. Developers can refer to these interface standards to implement their own extensions, while extensions can also declare support for specific standard interfaces. This allows development tools to provide better development support based on the standard interfaces declared by extensions, enabling users to select or replace required extensions based on standard interfaces.

In standard interfaces, Command/Data/Audio frame/Video frame are used to standardize interactions between extensions, achieving interchangeability and compatibility of extensions. Properties are used to standardize extension configuration parameters, achieving configuration compatibility with tman designer.

It should be noted that standardizing interactions between extensions is not the function of the interface itself, but the function of Command/Data/Audio frame/Video frame. Interfaces are just syntactic sugar for aggregation purposes, helping developers quickly specify extension interface definitions.

### Interface Checking and Error Reporting

In an extension's Command/Data/Audio frame/Video frame definitions, if there are two messages with the same name and identical schema definitions, this is not considered an error, but will be treated as duplicate definitions and ignored. This rule also applies to standard interfaces, allowing different standard interfaces to define common functionalities themselves, forming self-contained standard interface definitions. When an extension implements these standard interfaces with identical definitions, it won't cause errors. This feature is very useful in standard interfaces because standard interface definitions can be referenced. So when an extension implements multiple standard interfaces, if these standard interfaces (or other standard interfaces they reference) have the same messages and identical schemas, it won't cause errors.

Since the TEN framework adopts a runtime-development separation architecture (similar to the relationship between JavaScript and TypeScript's .d.ts files), standard interface checking and error reporting mainly occur during the development tool phase, not at runtime in TEN Runtime. Therefore, TEN Runtime will not report errors due to interface errors when starting the graph.

In the tman designer development tool, if an extension's interface definition is incomplete, prompts will be displayed indicating that certain functions are temporarily unavailable for that extension. For example, when using one extension to replace another extension's functionality, if the interface definition is incomplete, a prompt will indicate that this function is temporarily unavailable for this extension and needs to wait for the interface definition to be complete before use.

Additionally, since interface definitions can be independent files that are referenced, incomplete interface definitions are not considered errors during TEN Manager's publish/install operations.

### Interface Compatibility Rules

When replacing an extension that implements interface A with an extension that implements interface B, interface A must be compatible with interface B for the replacement to be possible. There are two modes of compatibility rules between interfaces:

**Strict Mode:**

* All messages in interface A must appear in interface B, and the schema of messages in interface A must be compatible with the schema of corresponding messages in interface B.

**Relaxed Mode:**

* Only messages in interface A that have connections must appear in interface B, and the schemas of these messages in interface A must be compatible with the schemas of messages with the same names in interface B.

Strict mode and relaxed mode are user-selectable, suitable for different scenarios. For example, if it's certain that a specific message X in interface A implemented by an extension will never be used, then being unable to replace with another desired extension due to message X's incompatibility or non-existence might be too strict.

## Implementation of Standard Interfaces

The APIs supported by TEN extensions are defined in their `manifest.json` files. The implementation of standard interfaces involves writing the defined API specifications into independent JSON files and then referencing these files in `manifest.json`.

```json
{
  "type": "extension",
  "name": "foo_extension",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.1"
    }
  ],
  "api": {
    "interface": [
      {
        "import_uri": "https://xxx/asr_interface.json"  // <= Reference to standard interface definition file
      },
      {
        "import_uri": "https://xxx/tts_interface.json"  // <= Reference to standard interface definition file
      }
    ],
    "property": {
      "bar": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "query_vector",
        "property": {
          "collection_name": {
            "type": "string"
          },
          "top_k": {
            "type": "int64"
          },
          "embedding": {
            "type": "array",
            "items": {
              "type": "float64"
            }
          }
        },
        "required": [
          "collection_name",
          "top_k",
          "embedding"
        ],
        "result": {
          "property": {
            "response": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "content": {
                    "type": "string"
                  },
                  "score": {
                    "type": "float64"
                  }
                }
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "publish",
        "property": {
          "message": {
            "type": "buf"
          }
        }
      },
      {
        "name": "set_presence_state"
      }
    ],
    "data_in": [
      {
        "name": "data"
      }
    ]
  }
}
```

### Example of Standard Interface Definition File

When TEN systems (including TEN Runtime, TEN Manager, etc.) parse `manifest.json`, they download the corresponding JSON files based on `import_uri` and merge their contents into `manifest.json`. The example content of the referenced standard interface file is shown below. As you can see, the content is the same as the `api` field in manifest.json, just separated into an independent file for easy reference. This independent standard interface file can also reference other standard interface files.

```json
{
  "interface": [
    {
      "import_uri": "https://xxx/stt_interface.json"// <= Reference to other standard interface definition files
    }
  ],
  "property": {
    "bar": {
      "type": "string"
    }
  },
  "cmd_in": [
    {
      "name": "query_vector",
      "property": {
        "collection_name": {
          "type": "string"
        },
        "top_k": {
          "type": "int64"
        },
        "embedding": {
          "type": "array",
          "items": {
            "type": "float64"
          }
        }
      },
      "required": [
        "collection_name",
        "top_k",
        "embedding"
      ],
      "result": {
        "property": {
          "response": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string"
                },
                "score": {
                  "type": "float64"
                }
              }
            }
          }
        }
      }
    }
  ],
  "cmd_out": [
    {
      "name": "publish",
      "property": {
        "message": {
          "type": "buf"
        }
      }
    },
    {
      "name": "set_presence_state"
    }
  ],
  "data_in": [
    {
      "name": "data"
    }
  ]
}
```

### Interface Merging Rules

The system merges the content of JSON files pointed to by `import_uri` into the `api` field of `manifest.json`, specifically including:

* `property`
* `cmd_in`, `cmd_out`
* `data_in`, `data_out`
* `audio_frame_in`, `audio_frame_out`
* `video_frame_in`, `video_frame_out`

**Important Rules:**

1. **Conflict Detection**: If fields of the same name and type are found during merging (such as `cmd_in` or `property` with the same name), it's not considered an error if the schemas are identical, otherwise the system will report an error.
2. **Recursive Merging**: The merging process is recursive, and the system will report an error if circular references are detected.
3. **Multiple Interface Support**: An extension can support multiple standard interfaces by defining multiple `interface` fields in `manifest.json`.
4. **Duplicate Reference Detection**: If different `interface` fields reference the same `import_uri`, the system will report an error.

## Storage Methods for Standard Interfaces

Standard interface files support the following storage methods:

<div className="fd-steps">
  <div className="fd-step">
    ### TEN System Package

    **Overview**: This method packages standard interface definition files into TEN system packages and then references them in `manifest.json`. You can also depend on this system package in the dependencies field of manifest.json, so that when installing the extension, the dependent standard interface system package will be installed together, providing the best development experience. Through this method, you can use TEN package's version mechanism to implement version management of standard interfaces, and use TEN package's dependency mechanism to implement automatic installation of standard interfaces.

    Using TEN system packages, in addition to standard interface JSON definition files, can also include related resources such as model files, common utility function libraries, supporting tools, etc. Therefore, when any of these resources have incompatible changes, they can be better reflected through TEN system package version number changes. For example, changes in the storage location or file names of JSON files within the system package are also considered breaking changes to the system package, so the major version number of the system package should be upgraded.

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "0.1.0"
        }
      ],
      "api": {
        "interface": [
          {
            "import_uri": "../../../ten_packages/system/foo_interface/interface.json"
          }
        ]
      }
    }
    ```

    **Advantages of Using TEN System Packages:**

    1. **Easy Distribution**: System packages can be uploaded to the TEN extension marketplace for users to download directly
    2. **Rich Resources**: The package not only contains interface definition files but can also include related resources such as:
       * Model files
       * Common utility function libraries
       * Supporting tools, etc.
    3. **Dependency Management**: System packages can serve as dependencies of extensions. When extensions are downloaded, related system packages will also be automatically downloaded
    4. **Version Management**: Not limited to standard interface protocols themselves, any resources within the system package are part of the system package's version management

    It should be noted that the version number of TEN system packages does not only represent the version number of standard interfaces, but represents the version number of the entire system package. Therefore, if resources within the system package have breaking changes, the major version number of the system package should be upgraded. For example, changes in the storage location or file names of JSON files within the system package, even if the content of JSON files hasn't changed, are considered breaking changes to the system package because the usage method needs to change for users, so the major version number of the system package should be upgraded.

    Therefore, you cannot judge the compatibility of standard interfaces solely based on the version number of system packages. The compatibility of standard interfaces still needs to be judged through interface compatibility rules.

    #### Support for Multi-Version Coexistence

    > **Note**: The functionality described in this section will basically not be implemented. It only describes possible implementation methods and considerations if it were to be done in the future.

    **Overview**: In some scenarios, a TEN application might need to use different versions of the same standard interface simultaneously. To support this requirement, TEN allows installing multiple versions of the same system package within the same application.

    **System Package Configuration:**

    First, declare support for multi-version coexistence in the system package's `manifest.json`:

    ```json
    {
      "type": "system",
      "name": "foo_interface",
      "version": "0.1.0",
      "install_policy": {
        "coinstallable": "true"
      }
    }
    ```

    **Extension Dependency Configuration:**

    In the `manifest.json` of extensions that depend on this system package, you also need to declare the use of multi-version coexistence:

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "=0.1.0", // Note: Only equal syntax can be used here, not >= syntax
          "install_policy": {
            "coinstallable": "true"
          }
        }
      ],
    }
    ```

    And during the entire dependency tree construction process, if a system package's coinstallable is true, then all places that depend on this system package need to declare coinstallable, otherwise the system will report an error.

    **File System Path:**

    After enabling multi-version coexistence, the path of system packages in the file system becomes:

    ```text
    ten_packages/system/<hash_value>_foo_extension_0_1_0/
    ```

    > **Note**: A mapping mechanism might be introduced in the future to allow users not to explicitly specify hash\_value during usage, but to automatically find corresponding system packages through some mapping mechanism. However, for this mechanism to work well, it would also require modifying the module loading mechanisms in languages supported by TEN, so that developers don't need to specify hash\_value in the module loading mechanisms of those languages, but can automatically find corresponding system packages through some mapping mechanism. This kind of deep language-internal modification is also one of the reasons why this single-package multi-version coexistence mechanism will basically not be implemented.
    >
    > **Note**: If there's really a need to use multiple versions of a standard interface within a TEN application, you can use the several methods described below. If a standard interface has accompanying common resources such as model files, common utility function libraries, supporting tools, etc., then many of these resources cannot achieve multi-version coexistence within a TEN application, which is also one of the reasons why this single-package multi-version coexistence mechanism will basically not be implemented.
  </div>

  <div className="fd-step">
    ### Network Address

    **Overview**: Deploy standard interface definition files to network servers and reference them through HTTP/HTTPS addresses:

    ```json
    {
      "interface": [
        {
          "import_uri": "http://xxx/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    > **Note**: When TEN Manager handles this type of reference, it can download files to its own cache and then use some mapping mechanism to allow TEN Manager to find the definitions.

    **Applicable Scenarios:**

    * **Lightweight Interfaces**: Interface definitions without accompanying common resources
    * **Version Flexibility**: Need to reference different versions of the same standard interface in one `manifest.json`
  </div>

  <div className="fd-step">
    ### Local Path

    **Overview**: Reference standard interface definition files in the local file system:

    **Absolute Path:**

    ```json
    {
      "interface": [
        {
          "import_uri": "file://c:/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **Relative Path:**

    ```json
    {
      "interface": [
        {
          "import_uri": "../asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **Applicable Scenarios:**

    Similar to the network address method:

    * **Lightweight Interfaces**: Suitable for interface definitions without accompanying common resources
    * **Version Flexibility**: Need to reference different versions of the same standard interface in one `manifest.json`
  </div>
</div>

## How to Filter Desired Extensions from the Cloud Store

The relationship between standard interfaces and the cloud store is reflected in packaging standard interfaces into TEN system packages. There are two ways to search for extensions that meet specific conditions:

1. **Dependency Search**: Search for extensions that depend on a certain system package. Since the metadata of extensions in the cloud store contains dependency information, the cloud store can filter out all extensions that depend on a certain system package accordingly.

2. **Tag Search**: Quickly search for extensions that meet certain conditions through TEN package tags, select and install them, then perform replacement operations. If they are TEN officially certified extensions, they can even be tagged with `ten:` prefix to indicate that this is an officially certified extension that indeed conforms to the declared standard interface definition.


# Check-Graph (/docs/ten_framework/0.11.60/ten_manager/check_graph)

`tman` provides the `check graph` command to validate predefined graphs or a `start_graph` command for correctness. To see the usage details, use the following command:

```shell
$ tman check graph -h
Check whether the graph content of the predefined graph or start_graph command is correct. For more detailed usage, run 'graph -h'

Usage: tman check graph [OPTIONS] --app <APP>

Options:
      --app <APP>
          The absolute path of the app defined in the graph. By default, the predefined graph will be read from the first one in the list.
      --predefined-graph-name <PREDEFINED_GRAPH_NAME>
          Specify a predefined graph name to check, otherwise, all predefined graphs will be checked.
      --graph <GRAPH>
          Specify the JSON string of a 'start_graph' command to be checked. If not specified, the predefined graph in the first app will be checked.
  -h, --help
          Print help
```

The `check graph` command is designed to handle graphs that may span multiple apps, so it does not require being run from the root directory of a specific app. Instead, it can be executed from any directory, with the `--app` parameter used to specify the folders of the apps involved in the graph.

## Example Usages

* **Check all predefined graphs in `property.json`**:

  ```shell
  tman check graph --app /home/TEN-Agent/agents
  ```

* **Check a specific predefined graph**:

  ```shell
  tman check graph --predefined-graph-name va.openai.azure --app /home/TEN-Agent/agents
  ```

* **Check a `start_graph` command**:

  ```shell
  tman check graph --graph '{
    "type": "start_graph",
    "seq_id": "55",
    "nodes": [
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_2__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      },
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_1__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      }
    ]
  }' --app /home/TEN-Agent/agents
  ```

## Prerequisites

* **Predefined Graph Definition Requirement**: If a predefined graph name is specified, the definition will be extracted from the `property.json` file of the app specified by the first `--app` parameter.

* **Package Installation Requirement**: Before running the `check graph` command, all extensions that the app depends on must be installed using the `tman install` command. This is necessary because the validation process requires information about each extension in the graph, such as APIs defined in their `manifest.json` files.

* **Unique App URI Requirement**: In a multi-app graph, each app's `property.json` must define a unique `ten::uri`. Additionally, the `uri` value cannot be set to `"localhost"`.

## Validation Rules

<div className="fd-steps">
  <div className="fd-step">
    ### Presence of Nodes

    The `nodes` array is required in any graph definition. If absent, an error will be thrown.

    * **Example (No nodes defined)**:

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": []
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        No extension node is defined in graph.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### Uniqueness of Nodes

    Each node in the `nodes` array represents a specific extension instance within a group of an app, created by a specified addon. Therefore, each extension instance should be uniquely represented by a single node. A node must be uniquely identified by the combination of `app`, `extension_group`, and `name`. Multiple entries for the same extension instance are not allowed.

    * **Example (Duplicate nodes)**:

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": [
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_2__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          },
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_1__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        Duplicated extension was found in nodes[1], addon: basic_hello_world_1__test_extension, name: test_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### Extensions used in connections should be defined in nodes

    All extension instances referenced in the `connections` field, whether as a source or destination, must be explicitly defined in the `nodes` field. Any instance not defined in the `nodes` array will cause validation errors.

    * **Example (Source extension not defined)**:

      Imagine that the content of `property.json` of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0] is not defined in nodes, extension_group: producer, extension: some_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

    * **Example (Destination extension not defined)**:

      Imagine that the content of `property.json` of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                },
                {
                  "type": "extension",
                  "name": "some_extension_1",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        }
                      ]
                    },
                    {
                      "name": "world",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        },
                        {
                          "extension": "consumer"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0].cmd[1] is not defined in nodes extension_group: some_group, extension: consumer.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The addons declared in the `nodes` must be installed in the app

    * **Example (The `ten::uri` in property.json is not equal to the `app` field in nodes)**:

      Imagine that the content of property.json of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ],
          "uri": "http://localhost:8001"
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

      The problem is all packages in the app will be stored in a map which key is the `uri` of the app, and each node in the graph is retrieved by the `app` field (which is `localhost` by default). The `app` in node (i.e., localhost) is mismatch with the `uri` of app (i.e., [http://localhost:8001](http://localhost:8001)).

    * **Example (the ten\_packages does not exist as the `tman install` has not executed)**:

      Imagine that the content of property.json of a TEN app is as follows.

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ]
        }
      }
      ```

      And the `ten_packages` directory does **\_NOT** exist.

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### In connections, messages sent from one extension should be defined in the same section

    * **Example**:

      Imagine that all the packages have been installed.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          },
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello_2",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        extension 'some_extension' is defined in connection[0] and connection[1], merge them into one section.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### In connections, the messages sent out from one extension should have a unique name in each type

    * **Example**:

      Imagine that all the packages have been installed.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              },
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "some_extension"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        - connection[0]:
         - Merge the following cmd into one section:
            'hello' is defined in flow[0] and flow[1].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The messages declared in the connections should be compatible

    The message declared in each message flow in the connections will be checked if the schema is compatible, according to the schema definition in the manifest.json of extensions. The rules of message compatible are as follows.

    * If the message schema is not found neither in the source extension nor the target extension, the message is compatible.

    * If the message schema is only found in one of the extensions, the message is incompatible.

    * The message is compatible only if the following conditions are met.

      * The property type is compatible.
        * If the property is an object, the fields both in the source and target schema must have a compatible type.
        * If the `required` keyword is defined in the target schema, there must be a `required` keyword in the source schema, which is the superset of the `required` in the target.

    * **Example**:

      Imagine that all the packages have been installed.

      The content of manifest.json in `addon_a` is as follows.

      ```json
      {
        "type": "extension",
        "name": "addon_a",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_out": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "string"
                }
              }
            }
          ]
        }
      }
      ```

      And, the content of manifest.json in `addon_b` is as follows.

      ```json
      {
        "type": "extension",
        "name": "addon_b",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_in": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "int8"
                }
              }
            }
          ]
        }
      }
      ```

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      Checking graph[0]... ❌. Details:
        - connections[0]:
          - cmd[0]:  Schema incompatible to [extension_group: some_group, extension: another_ext], { .foo: type is incompatible, source is [string], but target is [int8] }

      All is done.
      ❌ Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### The `app` in node must be unambiguous

    The `app` field in each node must met the following rules.

    * The `app` field must be equal to the `ten::uri` of the corresponding TEN app.

    * Either all nodes should have `app` declared, or none should.

    * The `app` field can not be `localhost`.

    * The `app` field can not be an empty string.

    * **Example (some of the nodes specified the `app` field)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        Either all nodes should have 'app' declared, or none should, but not a mix of both.
      ```

    * **Example (No `app` specified in all nodes, but some source extension specified `app` in the connections)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
         {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].the 'app' should not be declared, as not any node has declared it
      ```

    * **Example (No `app` specified in all nodes, but some target extension specified `app` in the connections)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext",
                    "app": "http://localhost:8000"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].cmd[0].dest[0]: the 'app' should not be declared, as not any node has declared it
      ```

    * **Example (The `app` field in nodes is not equal to the `ten::uri` of app)**:

      Same as [Rule 4](#id-4.-the-addons-declared-in-the-nodes-must-be-installed-in-the-app).

    * **Example (The `app` field is `localhost` in a single-app graph)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, and the graph seems to be a single-app graph, just remove the 'app' field
      ```

    * **Example (The `app` field is `localhost` in a multi-app graph)**:

      Checking graph with the following command.

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **Output**:

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, change the content of 'app' field to be consistent with 'ten::uri'
      ```
  </div>
</div>


# Overview (/docs/ten_framework/0.11.60/ten_manager/overview)

In the TEN ecosystem, almost everything is a component. For example, a TEN app is a type of component, a TEN extension is another type, and a TEN protocol is yet another. To assist TEN developers in managing all operations related to these components, there is a crucial tool called the TEN Manager.

<Callout type="info">
  `tman`

   is the abbreviation for TEN Manager.
</Callout>

The TEN Manager includes a TEN package manager, designed to simplify the management of TEN packages. This package manager handles tasks such as determining which dependency packages are necessary, fetching those dependencies, and installing them. In short, the TEN Manager streamlines the process of creating new TEN packages from scratch and makes it easier to install TEN packages from the TEN Cloud Store.


# Debugging (/docs/ten_framework/0.11.60/testing_and_debugging/debugging)

## For Developers of the TEN Framework

If you're using VS Code to develop the TEN framework, there is a `launch.json` file in the TEN framework source tree that provides default debug targets.

## For Users of the TEN Framework

If you're using VS Code to develop your own application based on the TEN framework, you can add configurations to your `.vscode/launch.json` file so that you can debug your application with breakpoints and variable inspection whichever programming language (C++/Golang/Python) you are using.

### Debugging in C++ Applications

If the application is written in C++, it means the extensions can be written
in C++ or Python.

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

* **Debugging C++ code with lldb**

  ```json
  {
    "name": "app (C++) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    }
  }
  ```

* **Debugging C++ code with gdb**

  ```json
  {
    "name": "app (C++) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### Debugging Python code with debugpy

You can use `debugpy` to debug Python code. However, since the program is not started directly through the Python interpreter, the Python code is executed by the embedded Python interpreter. Therefore, you can only attach a debugger to a running process.

First, you need to install `debugpy` in the Python environment:

```shell
pip install debugpy
```

Then, you need to start the application with environment variable `TEN_ENABLE_PYTHON_DEBUG` and `TEN_PYTHON_DEBUG_PORT` set. For example:

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

If the environment variable `TEN_ENABLE_PYTHON_DEBUG` is set to `true`, then the application will block until the debugger is attached. If the environment variable `TEN_PYTHON_DEBUG_PORT` is set, then the debugger server will listen on the specified port for incoming connections.

Then, you can attach the debugger to the running process. Here is an example of the configuration:

**Debugging Python code with debugpy**

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### Debugging C++ and Python code at the same time

If you want to start debugging C++ and Python code at the same time with one click in VSCode, you can do the following:

1. Define a task for delaying several seconds before attaching the debugger:

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. Add the following configurations to the `launch.json` file:

   ```json
   "configurations": [
        {
            "name": "app (C++) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch", // "launch" or "attach"
            "program": "${workspaceFolder}/bin/worker", // The executable path
            "cwd": "${workspaceFolder}/",
            "env": {
                "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
                "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (C++) (debugpy, attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "justMyCode": false,
            "preLaunchTask": "delay 3 seconds"
        }
   ],
   "compounds": [
        {
            "name": "app (C++) (lldb, launch) and app (C++) (debugpy, attach)",
            "configurations": [
                "app (C++) (lldb, launch with debugpy)",
                "app (C++) (debugpy, attach with delay)"
            ]
        }
   ]
   ```

Then, you can start debugging C++ and Python code with the compound configuration `app (C++) (lldb, launch) and app (C++) (debugpy, attach)`.

### Debugging in Go applications

If the application is written in Go, it means the extensions can be written in Go or C++ or Python.

#### Debugging Go code with delve

You can use the following configuration to debug in your Go extensions:

```json
{
  "name": "app (golang) (go, launch)",
  "type": "go",
  "request": "launch",
  "mode": "exec",
  "cwd": "${workspaceFolder}/",
  "program": "${workspaceFolder}/bin/worker", // The executable path
  "env": {
    "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
    "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
    "TEN_APP_BASE_DIR": "${workspaceFolder}"
  }
}
```

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

* **debugging C++ code with lldb**

  ```json
  {
    "name": "app (Go) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    },
    "initCommands": ["process handle SIGURG --stop false --pass true"]
  }
  ```

* **debugging C++ code with gdb**

  ```json
  {
    "name": "app (Go) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // The executable path
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### Debugging Python code with debugpy

Refer to the section above for debugging Python code with debugpy. Start the application with environment variable `TEN_ENABLE_PYTHON_DEBUG` and `TEN_PYTHON_DEBUG_PORT` set.

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

Then, you can attach the debugger to the running process. Here is an example of the configuration:

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### Debugging C++, Go and Python code at the same time

If you want to start debugging C++, Go and Python code at the same time with one click in VSCode, you can do the following:

1. Define a task for delaying several seconds before attaching the debugger:

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. Add the following configurations to the `launch.json` file:

   ```json
   "configurations": [
        {
            "name": "app (golang) (debugpy, remote attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "delay 3 seconds",
            "justMyCode": false
        },
        {
            "name": "app (golang) (go, launch with debugpy)",
            "type": "go",
            "request": "launch",
            "mode": "exec",
            "cwd": "${workspaceFolder}/",
            "program": "${workspaceFolder}/bin/worker",
            "env": {
                "TEN_APP_BASE_DIR": "${workspaceFolder}",
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (golang) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/bin/worker",
            "cwd": "${workspaceFolder}/",
            "env": {
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            },
            "initCommands": [
                "process handle SIGURG --stop false --pass true"
            ]
        },
   ],
   "compounds": [
        {
            "name": "Mixed Go/Python",
            "configurations": [
                "app (golang) (go, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ],
        },
        {
            "name": "Mixed Go/Python/C++ (lldb)",
            "configurations": [
                "app (golang) (lldb, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ]
        }
   ]
   ```

If you want to debug C++ and Python code at the same time, you can use the compound configuration `Mixed Go/Python/C++ (lldb)`. With this configuration, you can perform breakpoint debugging for C++, Go and Python code simultaneously. However, the variable inspection is only supported for C++ and Python but not for Go code.

If you want to debug Go and Python code at the same time, you can use the compound configuration `Mixed Go/Python`. With this configuration, you can perform breakpoint debugging for Go and Python code simultaneously and inspect variables for both Go and Python code.

### Debugging in Python applications

If the application is written in Python, it means the extensions can be written in Python or C++.

#### Debugging C++ code with lldb or gdb

You can use the following configurations to debug C++ code:

```json
{
  "name": "app (Python) (cpp, launch)",
  "type": "cppdbg", //
  "request": "launch", // "launch" or "attach"
  "program": "/usr/bin/python3", // The Python interpreter path
  "args": ["main.py"],
  "cwd": "${workspaceFolder}/",
  "environment": [
    {
      "name": "PYTHONPATH",
      "value": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface"
    },
    {
      "name": "TEN_APP_BASE_DIR",
      "value": "${workspaceFolder}/"
    }
  ],
  "MIMode": "gdb" // "gdb" or "lldb"
}
```

#### Debugging Python code with debugpy

Refer to the section above for debugging Python code with debugpy. You can use
the following configurations to debug Python code:

```json
{
  "name": "app (Python) (debugpy, launch)",
  "type": "debugpy",
  "python": "/usr/bin/python3",
  "request": "launch",
  "program": "${workspaceFolder}/main.py",
  "console": "integratedTerminal",
  "cwd": "${workspaceFolder}/",
  "env": {
    "PYTHONPATH": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface",
    "TEN_APP_BASE_DIR": "${workspaceFolder}/",
    "TEN_ENABLE_PYTHON_DEBUG": "true"
  }
}
```

#### Debugging C++ and Python code at the same time

If you want to start debugging C++ and Python code at the same time, you can launch the application with the configuration `app (Python) (debugpy, launch)` and attach the debugger to the running process with the following configuration:

```json
{
  "name": "app (Python) (cpp, attach)",
  "type": "cppdbg",
  "request": "attach", // "launch" or "attach"
  "program": "${workspaceFolder}/bin/worker", // The executable path
  "cwd": "${workspaceFolder}/",
  "MIMode": "gdb",
  "environment": [
    {
      "name": "LD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    },
    {
      "name": "DYLD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    }
  ]
}
```


# For Developers of TEN framework (/docs/ten_framework/0.11.60/testing_and_debugging/for_developers_of_ten_framework)

The TEN framework includes three types of test suites:

* Unit Tests
* Smoke Tests
* Integration Tests

The TEN framework uses `gtest` as the testing framework for unit tests and smoke tests, and `pytest` as the testing framework for integration tests.

## Unit Tests

The source directory for unit tests is `tests/ten_runtime/unit`. If you need to add new unit test cases, please place them in the `tests/ten_runtime/unit directory`.

For C++ projects, after successful compilation, you can navigate to the output directory (e.g., `out/linux/x64/tests/standalone/`, depending on the target OS and CPU) and execute the following command:

```shell
./ten_runtime_unit_test
```

## Smoke Tests

The source directory for smoke tests is `tests/ten_runtime/smoke`. If you need to add new smoke test cases, please place them in the `tests/ten_runtime/smoke` directory.

Navigate to the output directory (e.g., `out/linux/x64/tests/standalone`, depending on the target OS and CPU) and execute the following command:

```shell
./ten_runtime_smoke_test
```

## Loop Testing

You can perform multiple rounds of testing using the following commands.

To run only unit tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;
```

To run only smoke tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

To run both unit tests and smoke tests:

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

## Integration Tests

Integration tests are black-box tests for the TEN framework. The source directory for integration tests is `tests/ten_runtime/integration`. These tests are used to validate real-world execution scenarios.

The file structure for integration tests is as follows:

```text
tests/ten_runtime/integration/
  ├── test_1/
  │    ├── test_case.py
  │    └── ...
  ├── test_2/
  │    ├── test_case.py
  │    └── ...
  └── ...
```

To execute the integration tests, use the following command:

```shell
cd path/to/out/
pytest tests/ten_runtime/integration/
```

## Testing Tricks

### Linux

On Linux, you can use the following technique to slow down the execution of TEN, which can help in triggering timing-related bugs.

```shell
sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...
```

* `taskset 0x1`: Use only 1 CPU core to run the program.
* `taskset 0x3`: Use only 2 CPU cores to run the program.


# For Users of the TEN Framework (/docs/ten_framework/0.11.60/testing_and_debugging/for_users_of_ten_framework)

## Standalone Testing of Extension

The TEN framework provides an independent extension testing mechanism. This mechanism allows developers to test extensions without relying on other TEN concepts, such as other TEN extensions, TEN graphs, or TEN apps. When developers need to test the behavior of an extension without running the entire TEN app, this independent extension testing mechanism is especially useful.

The standalone testing framework for TEN extensions follows two key principles:

1. Compatibility with any native testing framework of the used language.

   For example, if a TEN extension is developed using C++, the Google gtest/gmock testing framework can be used alongside the TEN extension standalone testing framework to achieve independent testing of the C++ TEN extension.

2. Users don't need to modify any code in the extension under test.

   The exact same extension code used in runtime can be fully tested using this standalone testing framework.

3. The standalone testing flow for extensions developed in different languages follows the same design principles and usage methods.

   This ensures that once you learn the standalone testing concepts and methods for one language, you can quickly grasp the standalone testing concepts and methods for TEN extensions in other languages.

For users, the TEN extension standalone testing framework introduces two main concepts:

1. **extension\_tester**
2. **ten\_env\_tester**

The role of **extension\_tester** is similar to a testing driver, responsible for setting up and executing the entire testing process. **ten\_env\_tester**, on the other hand, behaves like a typical TEN extension's `ten_env` instance, enabling users to invoke functionalities within the standalone testing framework from the callback of `extension_tester`, such as sending messages to the extension under test and receiving messages from the extension.

From the API design of **extension\_tester** and **ten\_env\_tester**, you can see that they are very similar to **TEN extension** and **ten\_env**, and this is by design. The main purpose is to allow users familiar with extension development to quickly get accustomed to the standalone testing framework and then develop standalone test cases for the extension itself.

However, for testing purposes, there are inevitably APIs and features dedicated specifically to testing. To prevent these test-specific functionalities and APIs, which are not needed during actual runtime, from polluting the runtime API set, the standalone testing framework is designed not to directly use or extend the API sets of **extension** and **ten\_env**. Instead, it introduces types and API sets that are exclusive to standalone testing. This separation ensures that the types and API sets used in actual runtime and those used during testing do not interfere with each other, avoiding any negative side effects.

## Standalone Testing Framework Internal

Internally, the TEN extension standalone testing framework implicitly starts a test app, loads the extension addon to be tested (let’s call it extension A), and initializes a graph containing both the extension A to be tested and another extension (let’s call it extension B) dedicated to testing other extensions. All input and output messages of extension A are redirected to extension B, allowing users to customize inputs and outputs during the testing process and complete the actual testing work.

```mermaid
sequenceDiagram
  participant tester as Extension Tester
  participant testing as Extension Testing
  participant tested as Extension Tested

  tester->>testing: send messages
  testing->>tested: send messages

  tested-->>testing: send results
  testing-->>tester: send results
```

Basically, you can think of this testing extension hidden within the testing framework as a proxy between the extension being tested and the tester. It acts as a proxy extension within the TEN graph, facilitating the exchange of messages between the tested extension and the tester using the language of the TEN environment.

## Basic Testing Process

The basic testing process and logic are as follows:

1. Create an extension tester to manage the entire standalone testing process.
2. Inform the standalone testing framework of the folder containing the extension to be tested.
3. Set a testing mode, such as a mode for testing a single extension.
4. Start the testing.

## C++

Here is an example of TEN extension standalone testing using Google gtest:

```c++
class extension_tester_basic : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    auto new_cmd = ten::cmd_t::create("hello_world");
    ten_env.send_cmd(std::move(new_cmd),
                     [](ten::ten_env_tester_t &ten_env,
                        std::unique_ptr<ten::cmd_result_t> result) {
                       if (result->get_status_code() == TEN_STATUS_CODE_OK) {
                         ten_env.stop_test();
                       }
                     });
  }
};

TEST(Test, Basic) {
  // 1. Create an extension tester to manage the entire standalone testing process.
  auto *tester = new extension_tester_basic();

  // 2. Set a testing mode, such as a mode for testing a single extension.
  tester->set_test_mode_single("default_extension_cpp");

  // 3. Start the testing.
  tester->run();

  delete tester;
}
```

## Golang

TODO: To be added.

## Python

```python
class ExtensionTesterBasic(ExtensionTester):
    def check_hello(self, ten_env: TenEnvTester, result: CmdResult):
        statusCode = result.get_status_code()
        print("receive hello_world, status:" + str(statusCode))

        if statusCode == StatusCode.OK:
            ten_env.stop_test()

    def on_start(self, ten_env: TenEnvTester) -> None:
        new_cmd = Cmd.create("hello_world")

        print("send hello_world")
        ten_env.send_cmd(
            new_cmd,
            lambda ten_env, result: self.check_hello(ten_env, result),
        )

        print("tester on_start_done")
        ten_env.on_start_done()


def test_basic():
    # 1. Create an extension tester to manage the entire standalone testing process.
    tester = ExtensionTesterBasic()

    # 2. Set a testing mode, such as a mode for testing a single extension.
    tester.set_test_mode_single("default_extension_python")

    # 3. Start the testing.
    tester.run()
```


# How to Debug with Logs (/docs/ten_framework/0.11.60/testing_and_debugging/how_to_debug_with_logs)

We will introduce how to view logs, as they are now ported to stdout, making them very easy to view.

For example, if a TEN Agent is running on localhost:3000, you might see output similar to the following in the logs:

```bash
...
[GIN] 2024/09/29 - 09:29:42 | 200 |     634.458µs |    192.168.65.1 | POST     "/ping"
[agora_5bnxv1] 09-29 09:29:42.895  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ How about you?], text_recv_ts: 1727602181374572, read_bytes: 37200, sent_frames: 117, first_frame_latency: 306ms, finish_latency: 428ms, synthesis_first_byte_latency: 300ms, synthesis_finish_latency: 332ms
[agora_5bnxv1] 09-29 09:29:43.350  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Anything exciting happening?], text_recv_ts: 1727602181383864, read_bytes: 63600, sent_frames: 199, first_frame_latency: 300ms, finish_latency: 452ms, synthesis_first_byte_latency: 295ms, synthesis_finish_latency: 332ms
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_5bnxv1 requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
2024/09/29 09:29:45 INFO handlerPing end worker="&{ChannelName:agora_5bnxv1 HttpServerPort:10024 LogFile:/tmp/astra/app-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.json Pid:7455 QuitTimeoutSeconds:60 CreateTs:1727602125 UpdateTs:1727602185}" requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |    1.038208ms |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
2024/09/29 09:29:45 ERROR handlerPing channel not existed channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |     442.375µs |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:48 INFO handlerPing start channelName=agora_5bnxv1 requestId=6fb07d85-
...
```

Once you interact with the TEN Agent, you should see entries like the following:

```bash title="Bash"
[agora_5bnxv1] 09-29 09:35:25.049  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ and the World Cup in Russia.], text_recv_ts: 1727602522313774, read_bytes: 64000, sent_frames: 200, first_frame_latency: 418ms, finish_latency: 568ms, synthesis_first_byte_latency: 413ms, synthesis_finish_latency: 458ms
[agora_5bnxv1] 09-29 09:35:25.570  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ In tech,], text_recv_ts: 1727602522347918, read_bytes: 29200, sent_frames: 92, first_frame_latency: 427ms, finish_latency: 515ms, synthesis_first_byte_latency: 419ms, synthesis_finish_latency: 460ms
[agora_5bnxv1] 09-29 09:35:26.502  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ we saw some big advancements like the rise of AI and electric cars becoming more mainstream.], text_recv_ts: 1727602522461866, read_bytes: 178800, sent_frames: 559, first_frame_latency: 510ms, finish_latency: 928ms, synthesis_first_byte_latency: 503ms, synthesis_finish_latency: 800ms
[agora_5bnxv1] 09-29 09:35:27.029  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Personally,], text_recv_ts: 1727602522468652, read_bytes: 34400, sent_frames: 108, first_frame_latency: 442ms, finish_latency: 524ms, synthesis_first_byte_latency: 436ms, synthesis_finish_latency: 463ms
```

When you see logs like these, it means the system is working properly and is logging every sentence you speak.


# Doc Window (/docs/ten_framework/0.11.60/tman_designer/doc_window)

TMAN Designer has a built-in documentation window that allows you to view TEN Framework concepts and terminology. For example, you can select `About App` from the App dropdown menu to open the documentation window and display the App's description.

![TMAN Designer Doc Window](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_doc_window.png)


# Overview (/docs/ten_framework/0.11.60/tman_designer/overview)

TEN Framework offers high flexibility and provides rich functionalities, including TEN Cloud Store, package management, multi-language support, and multi-platform support. However, these features may require a longer learning period and higher technical threshold to master TEN Framework for AI Agent development and customization. To facilitate developers' use, TEN Framework provides a command-line tool called TEN Manager, which enables developers to easily modify and customize AI Agents built with TEN Framework.

Since command-line tools are not as user-friendly as graphical user interfaces, TEN Manager includes a web UI-based visual development interface called TMAN Designer, in addition to the command-line interface. It provides rich functionality to help developers develop and debug TEN Apps and TEN Extensions. This means that developers only need to obtain the TEN Manager tool to access a complete TEN Framework development environment with both command-line and graphical user interfaces.

You can get TMAN Designer from the following link:

```text
https://github.com/TEN-framework/ten-framework/releases
```

TMAN Designer supports the following operating systems and architectures. Please obtain the appropriate version for your platform:

* Windows
  * x86\_64
* Linux
  * x86\_64
* MacOS
  * x86\_64
  * arm64

After downloading and extracting, use the following command to start TMAN Designer:

```shell
tman designer
```

You can execute this command in the root directory of any TEN App, or in any location. If executed in the root directory of a TEN App, TMAN Designer will load that TEN App by default. You can also manually load any TEN App in TMAN Designer. Therefore, starting TMAN Designer in the root directory of a specific TEN App is not mandatory, but it simplifies the initial step of manually loading a TEN App.

After startup, you will see information like the following, indicating that TMAN Designer starts on port 49483 by default:

```text
🏆  Starting server at http://0.0.0.0:49483
```

You can interact with TMAN Designer using the following URL:

```text
http://127.0.0.1:49483/
```

> Note: If your TMAN Designer is running on a remote machine, please use the IP address of the remote machine to access TMAN Designer.

## Overview

After opening the TMAN Designer page in your browser, you will see the following interface:

![TMAN Designer Overview](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_overview.png)

In the upper right corner of TMAN Designer, you can see several function buttons:

* The rightmost button displays the current version of TMAN Designer and automatically checks for new version releases. If an update is available, an upward arrow will be displayed. Clicking it will open the download page for the new version.
* The first button on the left is the [TEN Agent](https://agent.theten.ai/) button. Clicking it will open the TEN Agent page. TEN Agent is a complete AI Agent implementation built using TEN Framework. One of the main goals of TMAN Designer is to enable developers to easily develop, debug, and customize TEN Agent through TMAN Designer.
* The second button on the left is the [TEN Framework](https://github.com/TEN-framework/ten-framework) GitHub page. Clicking it will open the GitHub repository for TEN Framework. You can visit this page to learn more about TEN Framework, view source code, or submit issues.
* On the far left, there are two buttons for switching between light and dark modes and selecting languages.

In the upper left corner of TMAN Designer, there are several dropdown menus that developers can use to access various functions of TMAN Designer.

## Working with TEN Apps

### Loading an Existing TEN App

You can load an existing TEN App using the Load App button in the App menu. After clicking the button, you will see the following interface:

![Loading an Existing TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_load_app.png)

Through this file browser dialog, developers can specify the root directory of an existing TEN App to load it.

### Managing Loaded TEN Apps

You can manage loaded TEN Apps using the Manage Loaded App(s) button in the App menu. After clicking the button, you will see the following Apps Manager interface:

![Managing Loaded TEN Apps](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_manage_loaded_apps.png)

Developers can use this dialog to:

* Unload loaded TEN Apps
* Reload specified TEN Apps
* Reload all loaded TEN Apps
* Install all dependencies for specified Apps
* Run functions for specified Apps

### Installing All Dependencies for an App

Developers can install various TEN packages in a TEN App. All TEN packages that a TEN App depends on are recorded in the `manifest.json` file. Here is an example of a `manifest.json` file:

```json
{
  "type": "app",
  "name": "aaa",
  "version": "0.8.18",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.8.18"
    },
    {
      "type": "extension",
      "name": "ffmpeg_muxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_demuxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_client",
      "version": "^0.1.0"
    },
    {
      "type": "protocol",
      "name": "msgpack",
      "version": "^0.8.18"
    }
  ],
  "scripts": {
    "start": "python3 tools/run_script.py start",
    "build": "python3 tools/run_script.py build"
  }
}
```

From the above configuration, we can see that this TEN App depends on the following TEN packages:

* `ten_runtime`: The system package of TEN Framework
* `ffmpeg_demuxer`: A TEN Extension package for decoding audio and video
* `ffmpeg_muxer`: A TEN Extension package for encoding audio and video
* `ffmpeg_client`: A TEN Extension package for processing audio and video using the other two ffmpeg extension packages
* `msgpack`: A TEN Protocol package for message serialization and deserialization using the MessagePack protocol

After completing the installation of all dependencies for the TEN App, the TEN package file structure in the system looks like this:

```text
├── ten_packages
│   ├── extension
│   │   ├── ffmpeg_client
│   │   ├── ffmpeg_demuxer
│   │   └── ffmpeg_muxer
│   ├── protocol
│   │   └── msgpack
│   └── system
│       ├── ffmpeg
│       └── ten_runtime
```

It's worth noting that even though it's not explicitly declared in the `manifest.json`, there is an additional `ffmpeg` system package in the system. This is because `ffmpeg_demuxer` and `ffmpeg_muxer` depend on the `ffmpeg` system package, and TEN Manager automatically identifies and installs these transitive dependencies.

Through the Apps Manager interface in TMAN Designer, developers can perform the Install All operation on a specified App to easily complete the installation of all dependencies. After clicking the button, you will see the following interface showing the installation progress:

![Installing All Dependencies for an App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_install_all.png)

### Installing TEN Extensions from the Cloud Store

You can open the TEN Cloud Store and install TEN Extensions to the current TEN App using the Open Extension Store button in the Graph menu. After clicking the button, you will see the following interface:

![TEN Cloud Store](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_store.png)

In the TEN Cloud Store, developers can search for the required TEN Extensions and click the Install button to install them to the current TEN App. After installation, you can view the installed TEN Extensions in the current TEN App.

### Running a TEN App

In the Apps Manager, you can perform preset runtime operations on the specified App. After clicking the Run button in the Apps Manager, you will see the following interface:

![Running a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_run.png)

In the dialog, developers can select the preset operation to execute from the dropdown menu. These typically include the following operations:

* Run App: Start the App and display the App's output information in the TMAN Designer message window.
* Build App: Compile the App. Since TEN Framework supports multiple programming languages, some languages need to be compiled while others do not. For example, TEN Apps developed in C++ need to be compiled, while TEN Apps developed in Python do not.

Below is the build interface for a TEN App developed in C++:

![Building a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_build.png)

After the build is complete, you can continue with the startup operation for the TEN App. The system will display a window showing the TEN App's runtime logs in real-time:

![Starting a TEN App](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_start.png)

## Working with TEN Graphs

### Loading an Existing TEN Graph

You can load an existing TEN Graph using the Open Existing Graph button in the Graph menu. After successful loading, you will see an interface similar to the following:

![TEN Graph](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_graph_overview.png)

Using this Graph as an example, it contains 3 TEN Extensions, namely:

* `ffmpeg_demuxer`
* `ffmpeg_muxer`
* `ffmpeg_client`

Each TEN Extension has its corresponding inputs and outputs. For example, `ffmpeg_demuxer` has the following inputs and outputs:

* 2 cmd inputs
* 1 cmd output
* 1 audio\_frame output
* 1 video\_frame output

Clicking on the number buttons for inputs and outputs opens a dialog like the following, which is used to manage input and output connections:

![TEN Message Input/Output Dialog](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_msg_info_dialog.png)

### Auto-Layout Graph

After manually adjusting the positions of TEN Extensions in the TEN Graph, developers can click the Auto Layout button in the Graph menu to automatically arrange the TEN Extensions in the TEN Graph.

### Opening the Context Menu for TEN Extensions

In the Graph, right-clicking on each TEN Extension opens the TEN Extension's context menu:

![TEN Extension Context Menu](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_context_menu.png)

In the context menu, you can open the TEN Extension's `manifest.json` file, `property.json` file, or directly start a terminal in the root directory of the TEN Extension, making it convenient for developers to modify and customize the TEN Extension:

![Opening Manifest](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_context_menu_effect.png)


# Customize via Code (/docs/ten_agent_examples/customize_agent/modify-main)

Configuring extensions via TMAN Designer should well help you get started with basic modifications. But what happens when you need to implement more complex changes that aren't supported by the visual interface?
Before making any code changes, it's essential to understand the existing structure and functionality of the `main` extension.

***

In the TEN Framework, **connections** are used to define how video and audio data flow between extensions. This approach reduces unnecessary complexity and ensures high performance—a core strength of TEN.

For **events and text-based data**, which are typically less resource-intensive, handling them directly in code offers greater flexibility and efficiency for implementing complex logic.

To make this workflow easier, the latest version of the default TEN Agent app introduces a built-in **“main” extension** in every graph. This extension acts as a central hub for orchestrating components and managing their connections.

With this design:

1. **Video/audio data** always flows through connections between extensions. This lets you take advantage of the framework’s performance optimizations without touching low-level media processing logic.
2. **The “main” extension** can communicate with any other extension and collect the event/text data it needs.

   * For **inbound events/data**, you must register connections from the source extensions to the “main” extension in the `property.json` file.
   * For **outbound events/data**, you can call `setDests` at runtime to specify destination extensions, without needing predefined connections in `property.json`.
3. **All application-specific logic** can reside inside the “main” extension.
4. **Other extensions** should remain stateless and independent, making them easy to reuse.

The "main" extension can be written in **any language supported by the TEN Framework**. For example, you could implement the main extension in Node.js while keeping other extensions in Python or C++. This design neatly avoids Node.js performance bottlenecks with media processing, particularly in production environments.

***

## Comparison of Main Extension Variants

Choose the right architecture for your use case:

| Aspect                | Python Cascade                                        | Python Realtime V2V                                  | Node.js Cascade                                   |
| --------------------- | ----------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------- |
| **Language**          | Python                                                | Python                                               | TypeScript/JavaScript                             |
| **Use Case**          | Traditional voice agent with ASR → LLM → TTS pipeline | Direct multimodal (MLLM) with real-time vision/audio | High-performance voice agent with media streaming |
| **Event Model**       | Decorator-based `@agent_event_handler()`              | Match/case with `on_data()` method                   | Direct event handlers in class methods            |
| **Message Flow**      | **S2C**: Receives typed events from runtime           | **S2C**: Receives all events via single `on_data`    | **Direct**: Routes messages as they arrive        |
| **LLM Type**          | Standard LLM (text-only)                              | Multimodal LLM (vision + audio + text)               | Standard LLM (text-only)                          |
| **Streaming Support** | Text streaming with sentence splitting                | MLLM streaming with modality support                 | Streaming with sentence parsing                   |
| **Interruption**      | Barge-in detection via `_interrupt()`                 | Event-based via `ServerInterruptEvent`               | Barge-in with sentence fragment reset             |
| **Tool Support**      | Via `@tool_handler()` decorator                       | Via match/case `FunctionCallEvent`                   | Via direct agent tool registration                |
| **Configuration**     | Via `config.yaml` in Python                           | Via `config.py` with `greeting`                      | Via manifest and property.json                    |
| **Performance**       | Good for standard workloads                           | Optimized for real-time multimodal                   | Best for media-heavy workloads                    |
| **Complexity**        | Moderate (decorators + event bus)                     | Moderate (match/case dispatch)                       | Moderate (direct method routing)                  |
| **Best For**          | Learning TEN fundamentals                             | Modern AI with vision + speech                       | Production voice agents                           |

### Quick Decision Guide

**Choose Python Cascade** if you want to learn how the main extension works or build a standard voice agent in Python.

**Choose Python Realtime V2V** if you need multimodal capabilities (camera input, vision processing) alongside voice, or want to experiment with advanced LLMs that support multiple modalities.

**Choose Node.js Cascade** if you're building a production voice agent and want the performance benefits of Node.js while avoiding media processing bottlenecks through the connection-based audio flow.

***

## Main Extension Documentation

### Python Cascade Pattern

Traditional ASR → LLM → TTS pipeline with decorator-based event handling. Ideal for learning TEN fundamentals or building standard voice agents.

* [Python Cascade Main](./main_python_cascade) — Complete guide with decorator patterns, event routing, and LLM integration
* Best for: Learning, standard voice agents, Python-only environments

### Python Realtime V2V Pattern

Multimodal MLLM integration with real-time vision and audio. Modern approach for advanced AI capabilities.

* [Python Realtime V2V Main](./main_python_realtime) — Event routing with match/case, C2S primitives, tool handling, and function calls
* Best for: Vision + speech, multimodal AI, modern LLMs with vision

### Node.js Cascade Pattern

High-performance voice agent using TypeScript/JavaScript with optimized media streaming.

* [Node.js Cascade Main](./main_nodejs_cascade) — ASR handling, LLM response streaming, tool registration, and natural interruption
* Best for: Production voice agents, performance-critical applications, Node.js teams

***

## Common Tasks

### Understanding Event Flow

* **Python Cascade**: Decorator-based event handlers (`@agent_event_handler()`) - See [ASR event handling section](./main_python_cascade#event-routing)
* **Python Realtime V2V**: Match/case dispatch - See [Event routing section](./main_python_realtime#event-routing)
* **Node.js Cascade**: Direct method handlers - See [ASR handling section](./main_nodejs_cascade)

### Handling Tool Calls

* **Python Cascade**: Tool decorator pattern
* **Python Realtime V2V**: Function call event handling - See [C2S primitives](./main_python_realtime#c2s-primitives-sending-to-mllm)
* **Node.js Cascade**: Agent tool registration

### Implementing Interruption

Learn how to handle user interruption (barge-in) in your pattern:

* **Python Cascade**: `_interrupt()` method in extension
* **Python Realtime V2V**: `ServerInterruptEvent` handling - See [Patterns section](./main_python_realtime#common-implementation-patterns)
* **Node.js Cascade**: Natural barge-in with sentence fragment reset - See [LLM handling section](./main_nodejs_cascade)


# NodeJS - Cascade Main (/docs/ten_agent_examples/customize_agent/modify-main/main_nodejs_cascade)

import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
import { Callout } from 'fumadocs-ui/components/callout';

# Main Agent Extension for Node.js

The file [`index.ts`](./index.ts) defines the **MainControlExtension** — the entry point for Node.js agents in the TEN Framework.

This is the class that wires together **ASR results**, **LLM responses**, **tool registration**, and **interruption handling**.
If you're building your own solution on top of TEN (with Node.js), start here.

<Tabs items={['Overview', 'ASR Handling', 'LLM Handling', 'Patterns', 'Summary']}>
  <Tab>
    ## Quick File Layout

    ```
    .
    ├── index.ts          → Main extension: message routing + interruption
    ├── helper.ts         → Utilities for sending Cmd/Data, sentence parsing
    └── agent/
        ├── agent.ts      → Event bus + orchestration
        ├── events.ts     → Event classes (ASR, LLM, Tools, User)
        ├── llm_exec.ts   → LLM execution queue and handlers
        └── struct.ts     → Zod schemas for TTS/ASR/LLM messages
    ```

    ## Architecture Overview

    <img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/nine.svg" alt="Architecture" />

    ## Core Architecture

    ### MainControlExtension Class

    Extends `Extension` from `ten-runtime-nodejs`. Normalizes runtime messages into **typed events** (`ASRResultEvent`, `LLMResponseEvent`, etc.). Routes events into the `Agent` class. Manages **interruption** when users talk over the assistant.

    ### Key Properties

    ```ts
    class MainControlExtension extends Extension {
      tenEnv!: TenEnv;
      agent!: Agent;
      config!: MainControlConfig;

      joinedUserCount: number = 0;
      session_id: string = "0";
      turn_id: number = 0;
      sentenceFragment: string = "";
    }
    ```

    **`session_id` + `turn_id`** track conversation state. **`sentenceFragment`** accumulates partial output for sentence splitting.
  </Tab>

  <Tab>
    ## ASR Event Handling

    When ASR detects speech, it's turned into an `ASRResultEvent`.

    ```ts
    if (event.final || event.text.length > 2) {
      await this._interrupt(); // flush LLM/TTS if user is overlapping
    }
    if (event.final) {
      this.turn_id += 1;
      await this.agent.queueLLMInput(event.text);
    }
    await this._sendTranscript("user", event.text, event.final, Number(this.session_id));
    ```

    ### Behavior

    **Partial speech** is logged as transcript and may trigger interrupt if overlapping. **Final speech** is queued for LLM input and increments turn counter. **Overlap detection** via `_interrupt()` cancels current LLM/TTS tasks.

    <Callout type="tip">
      Adjust interrupt sensitivity in `_onASRResult` based on your use case (stricter vs. looser barge-in).
    </Callout>
  </Tab>

  <Tab>
    ## LLM Response Handling

    LLM responses are streamed sentence by sentence, then forwarded to TTS and transcript collectors.

    ```ts
    if (!event.is_final && event.type === "message") {
      const [sentences, fragment] = parseSentences(this.sentenceFragment, event.delta);
      this.sentenceFragment = fragment;
      for (const s of sentences) {
        await this._sendToTTS(s, false);
      }
    }

    await this._sendTranscript(
      "assistant", event.text, event.is_final, 100,
      event.type === "reasoning" ? "reasoning" : "text"
    );
    ```

    ### Behavior

    **Streaming** sends TTS short sentences for natural, fluent speech. **Sentence splitting** via `parseSentences()` breaks output into natural pauses. **Partial transcripts** are logged as they arrive. **Final transcripts** are marked complete with reasoning/text type.

    ### Tool Registration

    The LLM can register tools dynamically.

    ```ts
    async _onToolRegister(event: ToolRegisterEvent) {
      await this.agent.registerLLMTool(event.tool, event.source);
    }
    ```

    Define your tool schema in `events.ts` and handle execution in your `Agent` logic.

    ### Interruption Handling

    Natural barge-in is handled by `_interrupt()`.

    ```ts
    async _interrupt() {
      this.sentenceFragment = "";
      await this.agent.flushLLM();
      await sendData(this.tenEnv, "tts_flush", "tts", { flush_id: uuidv4() });
      await sendCmd(this.tenEnv, "flush", "agora_rtc");
    }
    ```

    Flushes **LLM requests** (cancels pending generations). Cancels **TTS playback** (stops audio playback). Signals **RTC to stop** streaming. This ensures the user can always cut in naturally.
  </Tab>

  <Tab>
    ## Implementation Patterns

    ### Custom ASR Flow

    Edit `_onASRResult` to filter or preprocess speech:

    ```ts
    async _onASRResult(event: ASRResultEvent) {
      // Custom preprocessing
      let text = event.text.toLowerCase().trim();

      // Filter unwanted phrases
      if (text.includes("ignore")) return;

      // Then continue with normal flow
      await this._handleASRNormally(text, event.final);
    }
    ```

    ### Custom LLM Output

    Modify `_onLLMResponse` to change text before TTS or UI:

    ```ts
    async _onLLMResponse(event: LLMResponseEvent) {
      // Transform response
      event.delta = event.delta.replace("technical_term", "simplified_term");

      // Continue normally
      await this._handleLLMNormally(event);
    }
    ```

    ### Add a Tool

    1. Extend `ToolRegisterEvent` in `events.ts`
    2. Register in `Agent` via `registerLLMTool()`
    3. Handle execution in tool handler

    ### Tweak Responsiveness

    Adjust `_interrupt()` for stricter or looser barge-in:

    ```ts
    // Stricter (require more speech)
    if (event.text.length > 5) {
      await this._interrupt();
    }

    // Looser (interrupt on any sound)
    if (event.text.length > 0) {
      await this._interrupt();
    }
    ```

    <Callout type="warning">
      Node.js extension changes require a build step. Run `task build` to rebuild all Node.js extensions.
    </Callout>
  </Tab>

  <Tab>
    ## Supporting Components

    **`agent.ts`** manages event queues for ASR & LLM, plus handler registration. **`llm_exec.ts`** provides async queue that sends requests to LLMs and handles streaming responses, aborts, and tool calls. **`events.ts`** defines event classes like `ASRResultEvent`, `LLMResponseEvent`, `ToolRegisterEvent`. **`helper.ts`** provides `sendCmd`, `sendData`, and `parseSentences` for splitting output into natural phrases.

    ## Architecture Summary

    `index.ts` (`MainControlExtension`) is the **entry point** and **router**. It normalizes runtime messages into **typed events** and controls **interruption**. Supporting files (`agent/`, `helper.ts`) provide queues, event models, and execution helpers. The Cascade pattern flows: **ASR → LLM → TTS** with natural interruption handling.

    By following this structure, you can **quickly extend or replace** parts of the flow to build your own Node.js-based conversational agent on TEN.
  </Tab>
</Tabs>


# Python - Cascade Main (/docs/ten_agent_examples/customize_agent/modify-main/main_python_cascade)

# Building on the Main Extension

The file [`extension.py`](./extension.py) is the **control center** of the agent.
If you want to build your own solution on top of the TEN Framework, this is the file you should understand first.

It shows how runtime messages (ASR, LLM, tools, user events) are captured, normalized, and redirected into the agent workflow.
By following its patterns, you can easily **extend** or **replace** parts of the pipeline with your own logic.

***

## Quick File Layout

```
main_python/
├── extension.py      → Main message router (start here!)
└── agent/
    ├── agent.py      → Event bus and orchestration
    ├── events.py     → Typed event definitions (ASR, LLM, Tools, User)
    ├── llm_exec.py   → Manages LLM requests and responses
    └── decorators.py → Event binding helpers
```

You mostly need to know **`extension.py`**. The other files provide typed events and execution helpers.

***

## Architecture Overview

<img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/nine.svg" alt="Architecture" />

***

## How `extension.py` Works

<div className="fd-steps">
  <div className="fd-step">
    ### Event Routing

    The extension listens to runtime messages and turns them into **typed events**.
    For example, an ASR result from ASR extension becomes an `ASRResultEvent` object that the agent can use.

    ```python
    @agent_event_handler(ASRResultEvent)
    async def _on_asr_result(self, event: ASRResultEvent):
        if not event.text:
            return
        if event.final or len(event.text) > 2:
            await self._interrupt()
        if event.final:
            self.turn_id += 1
            await self.agent.queue_llm_input(event.text)
        await self._send_transcript("user", event.text, event.final, int(self.session_id))
    ```

    Here’s what happens:

    * **Partial speech** → streamed to transcript/logging
    * **Final speech** → sent into the LLM queue
    * **Overlap detected** → `_interrupt()` flushes ongoing TTS/LLM so the user can speak naturally

    ***
  </div>

  <div className="fd-step">
    ### Handling LLM Responses

    When the LLM responds, the extension splits sentences and forwards them to TTS while also recording transcripts.

    ```python
    @agent_event_handler(LLMResponseEvent)
    async def _on_llm_response(self, event: LLMResponseEvent):
        if not event.is_final and event.type == "message":
            sentences, self.sentence_fragment = parse_sentences(
                self.sentence_fragment, event.delta
            )
            for s in sentences:
                await self._send_to_tts(s, False)

        await self._send_transcript(
            "assistant", event.text, event.is_final, 100,
            data_type=("reasoning" if event.type == "reasoning" else "text"),
        )
    ```

    Key points:

    * **Streaming**: You can send partial outputs sentence by sentence to TTS for natural speech.
    * **Final output**: Marked and sent to the transcript collector.
    * **Reasoning traces**: Can be separated if you want to show them differently in your UI.

    ***
  </div>

  <div className="fd-step">
    ### Tool Registration

    The extension lets the LLM register tools dynamically.

    ```python
    @agent_event_handler(ToolRegisterEvent)
    async def _on_tool_register(self, event: ToolRegisterEvent):
        await self.agent.register_llm_tool(event.tool, event.source)
    ```

    To add your own tool, define its metadata in `events.py`, then handle it in your solution.
    This makes it easy to plug in **APIs, databases, or custom functions** that the LLM can call.

    ***
  </div>

  <div className="fd-step">
    ### Interruption

    Natural conversation needs interruption (barge-in).
    If the user speaks while the assistant is still generating, `_interrupt()` flushes LLM and TTS:

    ```python
    async def _interrupt(self):
        self.sentence_fragment = ""
        await self.agent.flush_llm()
        await _send_data(
            self.ten_env, "tts_flush", "tts", {"flush_id": str(uuid.uuid4())}
        )
        await _send_cmd(self.ten_env, "flush", "agora_rtc")
    ```

    This ensures the assistant doesn’t “talk over” the user.

    ***
  </div>
</div>

## How to Extend for Your Own Solution

* **Custom ASR behavior**: Edit `_on_asr_result` to filter text, add punctuation, or preprocess before sending to LLM.
* **Custom LLM logic**: Change `_on_llm_response` to transform text before TTS, or enrich transcripts with metadata.
* **Add tools**: Use `ToolRegisterEvent` to expose your own APIs or functions.
* **Custom interruption policy**: Tweak `_interrupt()` to make the agent more/less tolerant of overlapping speech.

## Summary

* **`extension.py` is the heart of the agent**: it routes runtime messages into typed events and applies core conversation logic.
* By modifying or extending these handlers, you can **quickly build your own conversational AI solution** on top of TEN.
* Everything else (`agent.py`, `llm_exec.py`, `events.py`) supports this routing, but the patterns in `extension.py` are what you’ll reuse the most.


# Python - Realtime V2V Main (/docs/ten_agent_examples/customize_agent/modify-main/main_python_realtime)

import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
import { Callout } from 'fumadocs-ui/components/callout';

# Realtime V2V / MLLM Agent (`extension.py`)

The file [`extension.py`](./extension.py) is the **entry point** of the realtime voice-to-voice / multimodal-LLM app.
It consumes **Server→Client (S2C)** events (transcripts, tool calls, interrupts, user join/leave), normalizes them into typed events, and routes them to the agent.
It also provides **Client→Server (C2S)** primitives to set context, send messages, return tool results, and trigger responses.

<Tabs items={['Overview', 'Event Routing', 'C2S Primitives', 'Patterns', 'Summary']}>
  <Tab>
    ## Quick File Layout

    ```
    .
    ├── extension.py      → Main extension: routing + context + trigger + interrupt
    ├── config.py         → Runtime config (e.g., greeting)
    ├── helper.py         → Cmd/Data send helpers
    └── agent/
        ├── agent.py      → Event queue, function call handling
        └── events.py     → Typed events (UserJoined, Transcripts, FunctionCall, Interrupt, etc.)
    ```

    <Callout type="warning">
      **Audio data** is not handled here. RTC audio is sent **directly to MLLM via TEN graph connections**.
      `extension.py` only reacts to the **S2C transcript events** generated by the MLLM server.
    </Callout>

    ### Architecture Overview

    <img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/ten.svg" alt="Architecture" />
  </Tab>

  <Tab>
    ## Event Routing

    Unlike decorator-based handlers, this extension uses a **single `on_data` method with a `match/case`** to dispatch events.

    ```python
    async def on_data(self, ten_env: AsyncTenEnv, data: Data):
        event = parse_event(data)  # converted to a typed AgentEvent

        match event:
            case UserJoinedEvent():
                self._rtc_user_count += 1
                await self._greeting_if_ready()

            case UserLeftEvent():
                self._rtc_user_count -= 1

            case ToolRegisterEvent():
                await self.agent.register_tool(event.tool, event.source)

            case FunctionCallEvent():
                await self.agent.call_tool(event.call_id, event.function_name, event.arguments)

            case InputTranscriptEvent():
                self.current_metadata = {"session_id": event.metadata.get("session_id", "100")}
                self.session_ready = True
                await self._greeting_if_ready()

            case OutputTranscriptEvent():
                await self._send_transcript("assistant", event.text, event.is_final, event.stream_id)

            case ServerInterruptEvent():
                await self._interrupt()

            case _:
                self.ten_env.log_warn(f"[MainControlExtension] Unhandled event: {event}")
    ```

    **S2C events handled:** `UserJoinedEvent` / `UserLeftEvent` track users. `InputTranscriptEvent` captures user speech text from RTC audio. `OutputTranscriptEvent` sends assistant response. `ToolRegisterEvent` registers new tools. `FunctionCallEvent` handles LLM tool requests. `ServerInterruptEvent` stops ongoing output.
  </Tab>

  <Tab>
    ## C2S Primitives (Sending to MLLM)

    `extension.py` provides simple methods for sending instructions to the MLLM server:

    <div className="fd-steps">
      <div className="fd-step">
        ### Set context

        ```python
        async def _set_context_messages(self, messages: list[MLLMClientMessageItem]):
            await _send_data(
                self.ten_env,
                DATA_MLLM_IN_SET_MESSAGE_CONTEXT,
                "v2v",
                MLLMClientSetMessageContext(messages=messages).model_dump(),
            )
        ```
      </div>

      <div className="fd-step">
        ### Send a message

        ```python
        async def _send_message_item(self, message: MLLMClientMessageItem):
            await _send_data(
                self.ten_env,
                DATA_MLLM_IN_SEND_MESSAGE_ITEM,
                "v2v",
                MLLMClientSendMessageItem(item=message).model_dump(),
            )
        ```
      </div>

      <div className="fd-step">
        ### Trigger a response

        ```python
        async def _send_create_response(self):
            await _send_data(
                self.ten_env,
                DATA_MLLM_IN_CREATE_RESPONSE,
                "v2v",
                MLLMClientCreateResponse().model_dump(),
            )
        ```
      </div>

      <div className="fd-step">
        ### Send function output

        (Sent from `agent.py` after handling a `FunctionCallEvent`)

        ```python
        await _send_data(
            self.ten_env,
            DATA_MLLM_IN_FUNCTION_CALL_OUTPUT,
            "v2v",
            MLLMClientFunctionCallOutput(
                output=result,
                call_id=call_id,
            ).model_dump(),
        )
        ```
      </div>
    </div>
  </Tab>

  <Tab>
    ## Common Implementation Patterns

    ### Greeting Recipe

    The greeting is handled in `_greeting_if_ready()` and triggered when the first user joins and session is ready:

    ```python
    async def _greeting_if_ready(self):
        if self._rtc_user_count == 1 and self.config.greeting and self.session_ready:
            await self._send_message_item(
                MLLMClientMessageItem(
                    role="user",
                    content=f"say {self.config.greeting} to me",
                )
            )
            await self._send_create_response()
    ```

    This ensures the assistant greets the user automatically.

    ### Function Call Handling

    **S2C**: `FunctionCallEvent` means the model requests a tool. **Agent**: executes the tool logic. **C2S**: return result with `DATA_MLLM_IN_FUNCTION_CALL_OUTPUT` and the same `call_id`. The model may continue its response afterwards.

    ### Interruption

    ```python
    async def _interrupt(self):
        await _send_cmd(self.ten_env, "flush", "agora_rtc")
    ```

    Triggered on `ServerInterruptEvent`. Stops RTC playback/streaming.

    ### Supporting Files

    **`agent.py`** queues events, executes tools, sends function outputs. **`events.py`** defines typed events: `UserJoinedEvent`, `UserLeftEvent`, `InputTranscriptEvent`, `OutputTranscriptEvent`, `ToolRegisterEvent`, `FunctionCallEvent`, `ServerInterruptEvent`. **`helper.py`** provides wrappers for `_send_cmd`, `_send_data`. **`config.py`** holds config like `greeting`. **Graph (`property.json`)** wires RTC audio → MLLM.
  </Tab>

  <Tab>
    ## Events/API Summary

    | Direction | Event / Channel                | Purpose                         |
    | --------- | ------------------------------ | ------------------------------- |
    | **S2C**   | `UserJoinedEvent`              | Track users; trigger greeting   |
    | **S2C**   | `UserLeftEvent`                | Track users leaving             |
    | **S2C**   | `InputTranscriptEvent`         | User speech text from RTC audio |
    | **S2C**   | `OutputTranscriptEvent`        | Assistant response text         |
    | **S2C**   | `ToolRegisterEvent`            | Register new tool               |
    | **S2C**   | `FunctionCallEvent`            | Model requests tool             |
    | **S2C**   | `ServerInterruptEvent`         | Stop ongoing output             |
    | **C2S**   | `_set_context_messages([...])` | Provide system/dev/user context |
    | **C2S**   | `_send_message_item(...)`      | Send user/developer message     |
    | **C2S**   | `_send_create_response()`      | Trigger assistant response      |
    | **C2S**   | Function output (from agent)   | Return tool results             |

    <Callout type="tip">
      For detailed event documentation and parameters, see the [API Reference](/docs/ten_agent_examples/api-reference/events).
    </Callout>
  </Tab>
</Tabs>


# Configure Extension Nodes (/docs/ten_agent_examples/customize_agent/tman-designer/configure_modules)

Extensions (or modules) are the core building blocks of a voice agent. They define the capabilities and functionalities of the agent. With TMAN Designer, you can easily add, remove, or replace extensions of same kind to tailor the agent to your specific needs.

You can follow [this guide]("/docs/ten-agent/customize_agent/tman-designer/") to learn to launch TMAN Designer if you haven't done so already.

## Replace Extension Node

Replacing can be very useful when you want to replace a node with another node of same protocol, as it will keep the original connections while changing the underlying implementation (e.g. to a different vendor model).

To replace a module or extension in TMAN Designer, follow these steps:

1. Right click on the node you would like to replace, in the context menu, select "Replace Node With..."
2. In the popup window, select the extension implementation you would like to use with "Addon Name" text input field.
3. Click "Replace Node" to confirm the replacement.

<Callout title="Note">
  The replacing extension will be using the properties defined in `property.json` by default.
</Callout>

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/three.png)

## Add Extension Node

You can add an extension node to a graph with following steps:

1. Right click on an empty area in the canvas, in the context menu, select "Add Node"
2. In the popup window, select the extension implementation you would like to use with "Addon Name" text input field, and give the node a name with "Node Name" text input field. The node name needs to be unique in a graph.
3. Click "Add Node" to confirm the addition.

<Callout title="Note">
  The added extension will be using the properties defined in `property.json` by default.
</Callout>

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/six.png)

## Remove Extension Node

You can remove an extension you no longer need from graph by following these steps:

1. Right click on the node you want to remove, in the context menu, select "Delete"
2. Confirm the removal in the popup window.

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/four.png)

## Supported Extension Categories

Some key categories of extensions are widely adopted in voice ai agent. In TEN-Agent these extensions are built using certain shared protocols so that it's easy to switch.

Following is a list of supported extension categories:

### Audio Speech Recognition (ASR)

* aliyun\_asr
* aliyun\_asr\_bigmodel\_python
* aws\_asr\_python
* azure\_asr\_python
* bytedance\_asr
* deepgram\_asr\_python
* gladia\_asr\_python
* google\_asr\_python
* openai\_asr\_python
* soniox\_asr\_python
* speechmatics\_asr\_python
* tencent\_asr\_python
* xfyun\_asr\_bigmodel\_python
* xfyun\_asr\_dialect\_python
* xfyun\_asr\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Large Language Model (LLM)

* openai\_llm2\_python
* dify\_llm2\_python
* coze\_llm2\_python

<Callout title="Note">
  Most LLM models are compatible with openai api schema. You can switch vendors by changing base\_uri / model to vendor you want.
</Callout>

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Text to Speech (TTS)

* azure\_tts\_python
* bytedance\_tts\_duplex
* cartesia\_tts2
* elevenlabs\_tts2\_python
* fish\_audio\_tts\_python
* google\_tts\_python
* humeai\_tts\_python
* minimax\_tts\_websocket\_python
* openai\_tts2\_python
* polly\_tts
* tencent\_tts\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Multimodal Realtime V2V Model (MLLM)

* azure\_mllm\_python
* glm\_mllm\_python
* gemini\_mllm\_python
* openai\_mllm\_python
* stepfun\_mllm\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Tools (TOOLS)

* weatherapi\_tool\_python
* vision\_analyze\_tool\_python
* mcp\_client\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>


# Configure Extension Properties (/docs/ten_agent_examples/customize_agent/tman-designer/configure_properties)

Every extension has a set of configurable properties. The configurable properties can be checked via the extension's `manifest.json` file.
The properties can be used to customize the behavior of the extension during runtime.

You can follow [this guide](/docs/ten-agent/customize_agent/tman-designer/) to learn to launch TMAN Designer if you haven't done so already.

## Configure Node Properties

1. Right click on a node you want to configure, in the context menu, select "Update Properties"
2. In the popup window, you may see a set of properties displayed as a form for you to edit.
3. Make the necessary changes and click "Save Change" to apply the updates.

![Properties Dialog](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/properties_dialog.png)

## Dynamic Fields

"Dynamic fields" are essentially a way to define an object without strictly specifying all of its possible keys upfront. Instead of enforcing a fixed schema, the object can accept and carry through any extra properties provided at runtime.

The main purpose is **flexibility and extensibility**:

* You can design an object with only the core or required fields defined.
* Any additional fields can be passed in dynamically, without changing the original object definition.
* When the object is handed off to an external SDK or API, those dynamic fields are transparently forwarded along with the known fields.

This means you don’t have to constantly update your object definition whenever new attributes are needed by the external system—you can just attach them on the fly.

To set a dynamic field, you can follow these steps:

1. Right click on a node you want to configure, in the context menu, select "Update Properties"
2. In the popup window, click on top tab to switch to "Dynamic Fields"
3. Choose the object key route where you want to add the dynamic field, set the name of the field, choose the data type and default value.
4. Confirm "Add Field" to finish adding the dynamic field.
5. Back to "Node Properties", you should see the newly added dynamic field in the "View Form Values" preview.

![Dynamic Fields](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/dynamic_fields.png)


# Use TMAN Designer (/docs/ten_agent_examples/customize_agent/tman-designer)

TEN Manager includes a web UI-based visual development interface called TMAN Designer, which eases the process of creating and customizing voice agents. You can refer to [TEN Framework docs](/docs/ten_framework/tman_designer/overview) for more details about TMAN Designer.

This section provides various examples of how to use TMAN Designer to create and customize voice agents for different use cases.

<Callout title="Deprecated use of playground">
  In earlier versions of TEN Agent, we supported to use playground to customize voice agents. This has been deprecated. For latest version of TEN Agent, please use TMAN Designer to customize your voice agents.
</Callout>

## Launching TMAN Designer

1. Make sure `task run` is running and TEN Manager server is started. If you haven't started it yet, you can follow the instructions in [Get Started](/docs/ten_agent/getting_started) to set it up.
2. Open the designer UI at [http://localhost:49483](http://localhost:49483) to configure your agent.
3. You should see the TMAN Designer interface now, the graphs of your current loaded TEN App should be displayed in the canvas.

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/five.png)


# Story Teller (/docs/ten_agent_examples/playground/use-cases/run_story_telling)

This guide will help you to run the Story Teller usecase with Large Language Model in the TEN-Agent Playground.

## STT + TTS + LLM

### Prerequisites

* Make sure you have the TEN-Agent playground running. If not, follow the [Run Playground](../../demo) guide to start the playground.
* You will need following information prepared:
  * STT info, any supported STT can be used. [Deepgram](https://deepgram.com/) is relatively easy to register and get started with.
  * TTS info, any supported TTS can be used. [Fish.Audio](https://fish.audio/) is relatively easy to register and get started with.
  * LLM info, For this use case only [OpenAI](https://openai.com) or OpenAI API compatible models are supported.
  * RTC info, currently only Agora RTC is supported. You can register your account at [Agora](https://www.agora.io/). We assume you have your App ID and App Certificate ready when you configure your `.env` file.

### Steps

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select the graph type `story_teller`.
3. Click on `Module Picker` to open the module selection.
4. If you preferred STT/TTS module is not by default selected, you can select the module from the dropdown list. Note you will need to configure the module with the correct information like API key, etc.
5. The `LLM` module is preconfigured to have `OpenAI ChatGPT` selected, don't change it.
6. Click on `Save Change` to apply the module to the graph.
7. Click on the Button to the right of the graph selection to open the property configuration. You will see a list of properties that can be configured for the selected Large Language Model.
8. Configure the properties with the information you prepared.
9. Click on `Save Change` to apply the properties to the Large Language Model.
10. If you see the success toast, the properties are successfully applied to the Large Language Model.
11. You are all set! Now you can start speaking to the Voice Assistant by clicking on the `Connect` button. Note you will need to wait a few seconds for agent to initialzie itself.

### Using Azure STT

Azure STT is integrated within RTC extension module. That's why if you want to use Azure STT, you will need to select `story_teller_integrated_stt` graph type.

### Bind Tools

The story\_teller use case is preconfigured to use `openai_image_generate_tool`, so usually you don't need to change anything.

## Realtime V2V

### Prerequisites

* Make sure you have the TEN-Agent playground running. If not, follow the [Run Playground](../../demo) guide to start the playground.
* You will need following information prepared:
  * Realtime API Key
* RTC info, currently only Agora RTC is supported. You can register your account at [Agora](https://www.agora.io/). We assume you have your App ID and App Certificate ready when you configure your `.env` file.

### Steps

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select the graph type `story_teller_realtime`.
3. Click on `Module Picker` to open the module selection.
4. The `V2V` module is preconfigured to have `OpenAI Realtime` selected. You can select other V2V modules from the dropdown list if needed. Note you will need to copy the `prompt` property from the `OpenAI Realtime` module to the new module, as module properties will be reset to default when switching.
5. Click on `Save Change` to apply the module to the graph if you have changed the V2V module, or if you have not changed the V2V module, you can skip this step.
6. Click on the Button to the right of the graph selection to open the property configuration. You will see a list of properties that can be configured for the selected V2V module.
7. Configure the `Realtime API Key` property with the information you prepared. If you have changed the V2V module in previous steps, do remember to copy the `prompt` property from the `OpenAI Realtime` module to the new module.
8. Click on `Save Change` to apply the property to the V2V module.
9. If you see the success toast, the property is successfully applied to the V2V module.
10. You are all set! Now you can start speaking to the Voice Assistant by clicking on the `Connect` button. Note you will need to wait a few seconds for agent to initialzie itself.

### Bind Tools

The story\_teller\_realtime use case is preconfigured to use `openai_image_generate_tool`, so usually you don't need to change anything.


# Voice Assistant (/docs/ten_agent_examples/playground/use-cases/voice-assistant)

This guide will help you to run the Voice Assistant with Large Language Model in the TEN-Agent Playground.

## STT + TTS + LLM

### Prerequisites

* Make sure you have the TEN-Agent playground running. If not, follow the [Run Playground](../../demo) guide to start the playground.
* You will need following information prepared:
  * STT info, any supported STT can be used. [Deepgram](https://deepgram.com/) is relatively easy to register and get started with.
  * TTS info, any supported TTS can be used. [Fish.Audio](https://fish.audio/) is relatively easy to register and get started with.
  * LLM info, any supported LLM can be used. It's recommended to use [OpenAI](https://openai.com).
  * RTC info, currently only Agora RTC is supported. You can register your account at [Agora](https://www.agora.io/). We assume you have your App ID and App Certificate ready when you configure your `.env` file.

### Steps

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select the graph type `voice_assistant`.
3. Click on `Module Picker` to open the module selection.
4. If you preferred STT/TTS module is not by default selected, you can select the module from the dropdown list. Note you will need to configure the module with the correct information like API key, etc.
5. From `LLM` module, click on the dropdown and select your preferred Large Language Model.
6. Click on `Save Change` to apply the module to the graph.
7. Click on the Button to the right of the graph selection to open the property configuration. You will see a list of properties that can be configured for the selected Large Language Model.
8. Configure the properties with the information you prepared.
9. Click on `Save Change` to apply the properties to the Large Language Model.
10. If you see the success toast, the properties are successfully applied to the Large Language Model.
11. You are all set! Now you can start speaking to the Voice Assistant by clicking on the `Connect` button. Note you will need to wait a few seconds for agent to initialzie itself.

### Using Azure STT

Azure STT is integrated within RTC extension module. That's why if you want to use Azure STT, you will need to select `voice_assistant_integrated_stt` graph type.

### Bind Weather Tool to your LLM

You can bind weather tool to your LLM module in the TEN-Agent Playground.
It's recommended to use OpenAI LLM below.

1. When you have your agent running. Open Module Picker.
2. Click on the button to the right of the LLM module to open the tool selection.
3. Select `Weather Tool` from the popover list.
4. Click on `Save Change` to apply the tool to the LLM module.
5. If you see the success toast, the tool is successfully applied to the LLM module.
6. You are all set! Now you can ask the agent about the weather by speaking to it.

## Realtime V2V

### Prerequisites

* Make sure you have the TEN-Agent playground running. If not, follow the [Run Playground](../../demo) guide to start the playground.
* You will need following information prepared:
  * Realtime API Key
* RTC info, currently only Agora RTC is supported. You can register your account at [Agora](https://www.agora.io/). We assume you have your App ID and App Certificate ready when you configure your `.env` file.

### Steps

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select the graph type `voice_assistant_realtime`.
3. Click on `Module Picker` to open the module selection.
4. Select your preferred V2V module from the dropdown list.
5. Click on `Save Change` to apply the module to the graph.
6. Click on the Button to the right of the graph selection to open the property configuration. You will see a list of properties that can be configured for the selected V2V module.
7. Configure the `Realtime API Key` property with the information you prepared.
8. Click on `Save Change` to apply the property to the V2V module.
9. If you see the success toast, the property is successfully applied to the V2V module.
10. You are all set! Now you can start speaking to the Voice Assistant by clicking on the `Connect` button. Note you will need to wait a few seconds for agent to initialzie itself.

### Bind Weather Tool to your V2V

You can bind weather tool to your V2V module in the TEN-Agent Playground.

1. When you have your agent running. Open Module Picker.
2. Click on the button to the right of the V2V module to open the tool selection.
3. Select `Weather Tool` from the popover list.
4. Click on `Save Change` to apply the tool to the V2V module.
5. If you see the success toast, the tool is successfully applied to the V2V module.
6. You are all set! Now you can ask the agent about the weather by speaking to it.


# Coze Chat Bot (/docs/ten_agent_examples/playground/use-cases/voice-assistant/run_coze)

In this tutorial, we will show you how to make Coze Bot speak in TEN-Agent playground.

## Prerequisites

* Make sure you have the TEN-Agent playground running. If not, follow the [Run Playground](../../demo) guide to start the playground.
* You will need following information prepared:
  * Coze info:
    * Coze Bot ID
    * Coze Token
    * Coze Base URL (only needed if you need to access non-global environment)
  * STT info, any supported STT can be used. [Deepgram](https://deepgram.com/) is relatively easy to register and get started with.
  * TTS info, any supported TTS can be used. [Fish.Audio](https://fish.audio/) is relatively easy to register and get started with.
  * RTC info, currently only Agora RTC is supported. You can register your account at [Agora](https://www.agora.io/). We assume you have your App ID and App Certificate ready when you configure your `.env` file.

## Steps

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select the graph type `voice_assistant`.
3. Click on `Module Picker` to open the module selection.
4. If you preferred STT/TTS module is not by default selected, you can select the module from the dropdown list. Note you will need to configure the module with the correct information like API key, etc.
5. From `LLM` module, click on the dropdown and select `Coze Chat Bot`.
6. Click on `Save Change` to apply the module to the graph.
7. Click on the Button to the right of the graph selection to open the property configuration. You will see a list of properties that can be configured for the `Coze Chat Bot` module.
8. Configure the `Coze Bot ID`, `Coze Token`, and `Coze Base URL` properties with the information you prepared.
9. Click on `Save Change` to apply the property to the `Coze Chat Bot` module.
10. If you see the success toast, the property is successfully applied to the `Coze Chat Bot` module.
11. You are all set! Now you can start speaking to Coze Bot by clicking on the `Connect` button. Note you will need to wait a few seconds for agent to initialzie itself.

## Using Azure STT

Azure STT is integrated within RTC extension module. That's why if you want to use Azure STT, you will need to select `voice_assistant_integrated_stt` graph type.


# Dify Chat Bot (/docs/ten_agent_examples/playground/use-cases/voice-assistant/run_dify)

In this tutorial, we will show you how to make Dify Bot speak in TEN-Agent playground.

## Prerequisites

* Make sure you have the TEN-Agent playground running. If not, follow the [Run Playground](./getting_started) guide to start the playground.
* You will need following information prepared:
  * Dify info:
    * Dify API Key
  * STT info, any supported STT can be used. [Deepgram](https://deepgram.com/) is relatively easy to register and get started with.
  * TTS info, any supported TTS can be used. [Fish.Audio](https://fish.audio/) is relatively easy to register and get started with.
  * RTC info, currently only Agora RTC is supported. You can register your account at [Agora](https://www.agora.io/). We assume you have your App ID and App Certificate ready when you configure your `.env` file.

> You can use any Agent / Chat Assistant defined in Dify platform. Each Agent / Chat Assistant has its own API Key.

## Steps

1. Open the playground at [localhost:3000](http://localhost:3000) to configure your agent.
2. Select the graph type `voice_assistant`.
3. Click on `Module Picker` to open the module selection.
4. If you preferred STT/TTS module is not by default selected, you can select the module from the dropdown list. Note you will need to configure the module with the correct information like API key, etc.
5. From `LLM` module, click on the dropdown and select `Dify Chat Bot`.
6. Click on `Save Change` to apply the module to the graph.
7. Click on the Button to the right of the graph selection to open the property configuration. You will see a list of properties that can be configured for the `Dify Chat Bot` module.
8. Configure the `Dify API Key` property with the information you prepared.
9. Click on `Save Change` to apply the property to the `Dify Chat Bot` module.
10. If you see the success toast, the property is successfully applied to the `Dify Chat Bot` module.
11. You are all set! Now you can start speaking to Dify Bot by clicking on the `Connect` button. Note you will need to wait a few seconds for agent to initialzie itself.

## Using Azure STT

Azure STT is integrated within RTC extension module. That's why if you want to use Azure STT, you will need to select `voice_assistant_integrated_stt` graph type.

## Troubleshooting

If you encounter any issues, please check the following:

* Make sure you have the correct API Key for Dify Chat Bot.
* Make sure in Dify your Chat Assistant has a valid model key configured.
* Make sure you have the correct STT and TTS modules selected and configured.
* Make sure you have the correct graph type selected.


# 应用场景 (/cn/docs/application_scenarios)

TEN framework 旨在使 AI 功能集成和应用开发变得更加高效和灵活。开发者可以快速创建和部署 AI 功能扩展，并将它们组合成定制化的应用程序或服务。典型的 AI 场景包括：

* **智能语音处理**：可以将语音识别（STT）、语音合成（TTS）和情感分析等扩展组合，构建对话系统。

* **视觉识别系统**：可以利用图像识别和视频分析扩展，构建具有人脸识别、物体检测和运动追踪功能的安防或零售解决方案。

* **推荐系统和数据分析**：可以组合机器学习和数据处理扩展，创建个性化推荐系统和分析工具。

* **交互式教育和培训平台**：可以将交互式教学、智能辅导和评估等扩展组装成完整的教育科技解决方案。

* **健康监测和诊断系统**：可以利用医学图像分析和 AI 驱动的诊断扩展，开发疾病监测和早期诊断系统。

通过 TEN framework，开发者可以创建、共享和扩展 AI 扩展，实现快速开发和创新。灵活的架构确保了可扩展性和可维护性，为现代 AI 解决方案提供了坚实的基础。


# 目标 (/cn/docs/goal)

TEN framework 的愿景是建立一个模块化、灵活且可扩展的平台，以提升开发速度、可维护性和扩展性。该框架旨在支持模块化开发的最佳工程实践，通过平台提供的全面机制，使开发者能够创建健壮、高效且适应性强的应用程序。


# 欢迎使用 TEN (/cn/docs)

欢迎使用 TEN！TEN（Transformative Extensions Network）是一个用于创建对话式 AI 的语音代理框架。

TEN framework 具有以下优势：

1. **原生支持高性能实时多模态交互**

   如果您的 AI 应用涉及复杂的视听场景，TEN 是您的最佳选择。它提供高性能和低延迟，并对各种扩展之间的交互进行了广泛优化，以确保 AI 应用的高效开发。

2. **支持多种编程语言和平台**

   开发者可以使用多种编程语言（如 C++、Go 和 Python，未来还将支持 JavaScript/TypeScript）创建模块化和可重用的扩展。此外，TEN framework 可以在 Windows、Mac、Linux 和移动设备等各种平台上无缝运行。

3. **边缘云集成**

   通过 TEN framework，部署在边缘和云环境中的扩展可以轻松组合，创建多样化的应用场景。对于注重隐私的边缘部署，小型模型可以利用本地计算能力来降低成本和延迟，同时可以集成云端大型模型以实现成本和性能的最佳平衡。

4. **突破模型局限的灵活性**

   TEN framework 允许创建超越单一大型模型局限性的复杂 AI 应用。通过简单的拖放式响应式编程方法，可以轻松构建满足各种需求的代理。TEN 还便于将 AI 与视听工具、数据库、监控系统、RAG 等进行集成。

5. **实时代理状态管理**

   TEN 具备管理实时代理状态的能力，能够实现动态响应并实时调整代理行为。

***

想要看看 TEN 的实际应用？请访问 **[agent.theten.ai](https://agent.theten.ai)** 体验由 TEN framework 驱动的真实 AI 代理。

要开始您的 AI 代理开发之旅，请探索我们专门的 [**TEN Agent**](https://github.com/ten-framework/TEN-Agent) 代码仓库。它包含了帮助您快速高效入门的丰富资源。

如果您能给 TEN Agent 仓库点个星标，我们将不胜感激。您的支持将帮助我们不断成长和进步！🌟


# 核心概念 (/cn/docs/key_concepts)

## 模块化

1. **明确的模块边界**：框架支持清晰定义控制平面和数据平面的边界，这些边界可以静态声明。这便于使用离线工具进行检查和后处理。

2. **动态服务组合**：进程内的模块可以在运行时动态组合，无需静态编译，实现灵活的服务配置。

3. **简化的模块集成**：在多模块组合场景中，只需要声明即可，无需修改任何模块代码或编写胶水代码。

4. **去中心化的模块耦合**：架构允许不同团队独立开发模块，模块间无耦合，支持动态替换。

5. **灵活的模块接口通信**：支持两种类型的接口通信——一种要求双方参数一致，另一种允许参数差异并由平台提供桥接机制。这种灵活性对于独立的模块开发和部署至关重要。

## 编排

1. **有向循环图（DCG）拓扑**：支持复杂的图结构进行模块组合，在控制平面和数据平面都能实现 1 对 n、n 对 1 和 n 对 m 的交互关系。

2. **跨进程和线程编排**：支持跨不同进程和线程的模块实例协调，形成完整的服务场景。

   ![跨进程和线程编排](https://ten-framework-assets.s3.amazonaws.com/doc-assets/cross_process_and_thread_orchestration.png)

3. **动态编排配置**：编排可以在运行时动态配置，也可以在服务执行前预配置。

## 灵活的配置设置

1. **层级配置支持**：不同模块实例可以有不同的配置可能性，能够适应不同的层级上下文。

2. **多样化配置能力**：配置可以预设、在运行时动态更改或作为更新分发。

## 多样的发布能力

1. **支持二进制和源码发布**：模块可以以源码或二进制形式分发，同一模块可以同时支持这两种模式。

2. **云市场集成**：无论是源码还是二进制形式的模块，都可以发布到云市场，便于下载和集成到项目中。

## 多实例能力

1. **平台多实例**：平台避免使用全局变量，支持在单个业务进程中运行多个实例。

2. **模块多实例**：模块开发时不依赖全局变量，允许同一模块的多个实例无冲突运行。

3. **编排流水线多实例**：支持在单个进程中并发或顺序执行多个编排流水线。

## 多样的分发能力

1. **分发配置动态切换**：模块可以跨进程和线程分发，以实现高可用、高并发和便捷的热更新。

2. **分发配置解耦**：分发配置与模块代码和编译/构建过程解耦，简化部署。

## 灵活的输入/输出支持

1. **支持控制平面和数据平面**：框架支持模块间的命令调用和原始数据（音频/视频/数据）传输。

2. **支持跨进程和线程通信**：模块可以同时处理跨进程和线程的外部输入/输出操作。

3. **RPC 特性**：外部输入/输出操作具有 RPC 特性，实现跨进程模块的无缝集成。

4. **触发多个下游组件**：单个动作可以触发多个下游模块，并将聚合响应返回给原始模块。

5. **声明式外部输入/输出**：接口可以静态声明，支持离线工具进行静态检查。

6. **简化的外部输入/输出代码**：函数调用接口（理想情况下自动生成）简化了模块开发者的编码工作。

7. **支持非序列化数据**：架构支持传输非序列化数据，工具可自动识别和标记此类用法。

8. **透明的跨进程通信**：平台抽象了底层传输逻辑，简化跨进程交互。

9. **支持同步和异步控制平面操作**：控制平面支持同步和异步交互，允许双向命令和响应流。

10. **支持受限执行上下文**：支持同一线程内模块间的同步交互。

## 模块开发自由度

1. **使用第三方库的自由**：模块可以自由使用第三方库来实现所需功能。

2. **使用操作系统特性的自由**：开发者可以在模块内自由创建和终止原生线程，提供最大的开发灵活性。

## 编程语言支持

1. **支持多种编程语言**：框架支持使用多种语言（如 C/C++、Go、Java、Python）编写模块，实现多样化的开发环境。

   ![编程语言支持](https://ten-framework-assets.s3.amazonaws.com/doc-assets/various_language_support.png)

2. **跨语言模块集成**：使用不同语言编写的模块可以在同一进程中运行。

3. **跨语言统一交互**：模块在不同语言中具有一致的接口和行为。

4. **跨语言统一数据类型**：平台提供统一的数据类型，确保不同编程语言间的数据交换一致性。

## 丰富的外部交互能力

1. **UI 提供者**：架构支持后端服务为前端提供 UI 组件。

2. **RESTful 接口**：后端服务可以暴露 RESTful 接口用于客户端交互。

3. **无侵入式集成现有服务**：框架可以集成到现有服务中，无需完全重写。

4. **单线程运行**：框架可以作为独立进程运行，或作为现有服务中的一个线程运行，避免与现有业务逻辑冲突。

## 测试友好

1. **多样化测试机制**：框架支持多种测试策略，包括单元测试、集成测试、模块测试和黑盒测试。

2. **独立模块测试**：模块可以独立测试，便于进行全面有效的测试。

## 离线处理能力

1. **包管理器**：框架包含 CLI 和包管理器，用于管理模块及其组合。

2. **编排工具**：提供 GUI 工具用于离线开发和编排，提升开发者体验。

3. **可视化调试工具**：架构支持可视化调试工具，帮助可视化模块交互。

## 健壮性

1. **编排流水线高并发**：平台支持单进程内至少 100 个编排流水线的并发实例。

2. **RESTful 接口高并发**：平台可在单进程中支持超过 20,000 个 RESTful 客户端。

3. **非 RESTful 接口高并发**：平台可在单进程中支持超过 1,000 个非 RESTful 客户端。


# TEN Turn Detection (/cn/docs/ten_turn_detection)

# TEN Turn Detection

Intelligent model for detecting conversation turns in human-AI dialogue systems, identifying three key utterance states for natural conversation flow.

## Core Functionality

Classifies user text into three states:

* **Finished**: Complete thought expressed; user expects a response
* **Unfinished**: Clearly unfinished utterance where user pauses but intends to continue
* **Wait**: User requests AI to pause or stop speaking

## Technology

* **Model**: Transformer-based (Qwen2.5-7B)
* **Languages**: English and Chinese
* **Inference**: Production-ready deployment scripts included

## Performance Metrics

Strong accuracy across all detection categories:

| Metric               | Accuracy |
| -------------------- | -------- |
| Finished (English)   | 90.64%   |
| Unfinished (English) | 98.44%   |
| Wait State           | 91%      |

Performance exceeds comparable open-source alternatives.

## Use Cases

* **Full-Duplex Conversation** — Enable interruption detection and natural turn-taking
* **Conversation Flow Management** — Determine when to wait for user continuation vs. respond
* **Pause/Stop Handling** — Recognize when users request the AI to pause or stop
* **Multi-Language Support** — Deploy in English and Chinese speaking applications

## Getting Started

```python
# Basic usage
from ten_turn_detection import TurnDetection

detector = TurnDetection()
state = detector.detect("user input text")
# Returns: "finished", "unfinished", or "wait"
```

## Integration with TEN Agent

TEN Turn Detection works seamlessly with TEN VAD for complete conversation management:

1. **TEN VAD** detects speech activity (when user is speaking)
2. **TEN Turn Detection** analyzes utterance to determine conversation state
3. **Agent** responds appropriately based on detected turn state

This enables natural, responsive voice conversations with proper turn-taking.

## Datasets

Bilingual test datasets (TEN-Turn-TestSet) are available for:

* Validation and testing
* Fine-tuning on domain-specific language
* Benchmarking model performance

## Resources

* **GitHub Repository**: [TEN-framework/ten-turn-detection](https://github.com/TEN-framework/ten-turn-detection)
* **Model Weights**: Available via Hugging Face
* **Datasets**: Bilingual test datasets included

## License

Apache 2.0 (see repository for details)


# TEN VAD (/cn/docs/ten_vad)

# TEN VAD - Voice Activity Detector

TEN VAD is a lightweight, high-performance Voice Activity Detector (VAD) designed for real-time speech detection with superior accuracy and minimal computational overhead.

## Overview

TEN VAD delivers exceptional precision compared to industry alternatives like WebRTC VAD and Silero VAD, while consuming significantly fewer computational resources and memory.

**Key Performance Metrics:**

* Real-Time Factor: 0.015 on AMD Ryzen (exceptionally fast)
* Library Size: Only 306KB (minimal footprint)
* Superior precision compared to WebRTC and Silero VAD
* Frame-level speech activity detection for fine-grained control

## Features

### Multi-Platform Support

* **Desktop**: Linux, Windows, macOS
* **Mobile**: Android, iOS
* **Web**: Browser integration via WebAssembly

### Multi-Language Bindings

Build VAD detection in your preferred language:

* Python (with numpy)
* C/C++
* Java (with JNI support for Android)
* Go
* JavaScript/WebAssembly

### Technology Stack

* **ONNX Model Format** for cross-platform deployment and optimization
* **WebAssembly Support** for seamless browser integration
* **JNI Bindings** for native Android support

## Use Cases

* **Real-time Voice Assistants** — Detect speech onset and offset for responsive interaction
* **Voice Conferencing** — Identify active speakers in multi-party conversations
* **Speech Recognition Preprocessing** — Optimize ASR pipeline by filtering silence
* **Mobile Applications** — Lightweight detection on resource-constrained devices
* **Browser-Based Applications** — Client-side speech detection without server round-trips

## Performance Highlights

TEN VAD outperforms comparable alternatives across diverse hardware platforms:

* Minimal computational complexity
* Reduced memory usage
* Fast inference time (Real-Time Factor \< 1)
* Accurate frame-level detection

This makes it ideal for:

* Edge deployment scenarios
* Real-time conversational AI
* Mobile and web applications
* Multi-stream processing

## Getting Started

### Installation

```bash
# Python
pip install ten-vad

# JavaScript/Node.js
npm install ten-vad
```

### Basic Usage

```python
from ten_vad import VAD

# Initialize VAD
vad = VAD()

# Process audio frames
for frame in audio_stream:
    is_speech = vad.detect(frame)
    if is_speech:
        print("Speech detected!")
```

## Integration with TEN Agent

TEN VAD is a core component of TEN Agent's speech processing pipeline:

* Enables natural interruption handling in voice interactions
* Improves responsiveness by detecting speech boundaries
* Reduces unnecessary processing of silence

Use TEN VAD with the TEN framework for:

* **Voice Activity Detection** — Identify when users are speaking
* **Turn Detection** — Combined with TEN Turn Detection for intelligent conversation flow
* **Barge-in Support** — Enable users to interrupt AI responses naturally

## Resources

* **GitHub Repository**: [TEN-framework/ten-vad](https://github.com/TEN-framework/ten-vad)
* **Documentation**: Full API reference and examples
* **Community**: 1.6k stars, 131 forks, active Discord and WeChat communities

## Latest Updates (2025)

Recent releases include:

* Python inference with ONNX models on Linux and macOS
* Full Golang support across major operating systems
* Java support on multiple platforms including Android
* Enhanced model accuracy and performance optimizations

## License

TEN VAD is open-source and community-driven. Check the repository for licensing details.


# 架构 (/cn/docs/ten_agent_examples/architecture_flow)

整个系统包含多个协同工作的组件，以提供无缝的体验。主要组件包括：

* **TEN Agent App**: 主要应用程序，负责协调扩展并管理它们之间的数据流。它作为后台进程运行。根据图的配置，agent 启动所需的扩展并处理它们之间的数据流。
* **前端 UI**: 一个基于 Web 的界面，用于与 agent 交互。它允许用户配置 agent、启动/停止 agent 以及与 agent 对话。
* **Web 服务器**: 一个简单的 Golang Web 服务器，用于处理 HTTP 请求。它负责处理传入的请求和启动/停止 agent 进程。它还传递诸如 `graph_name` 之类的参数来确定要使用的图。

系统的流程如下：

<img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/eight.svg" alt="Architecture" />


# 常见问题 (/cn/docs/ten_agent_examples/faqs)

## 为什么我看到 `/app/agnets/bin/start: not found`？

Windows 用户偶尔会遇到行尾问题，这可能会导致诸如 `/app/agnets/bin/start: not found` 之类的错误。

为了获得最佳体验，我们建议将 Git 配置为在 Windows 上自动将行尾转换为 LF。

要解决此问题：

1. 配置 Git 以在 Windows 上自动将行尾转换为 LF：

   ```bash title="终端"
   git config --global core.autocrlf false
   ```

2. 重新克隆存储库

或者，您可以直接从 GitHub 下载并解压缩 ZIP 文件。

## 如何验证我的互联网连接？

请确保您的 **HTTPS** 和 **SSH** 都已连接到互联网。

测试 **HTTPS** 连接：

```bash title="终端"
ping www.google.com

# 您应该看到以下输出：
PING google.com (198.18.1.94): 56 data bytes
64 bytes from 198.18.1.94: icmp_seq=0 ttl=64 time=0.099 ms
64 bytes from 198.18.1.94: icmp_seq=1 ttl=64 time=0.121 ms
```

测试 **SSH** 连接：

```bash title="终端"
curl www.google.com

# 您应该看到以下输出：
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<h1>301 Moved Permanently</h1>
</body>
</html>
```

## 如何刷新 env 文件？

要在开发过程中查看更新的更改，请按照以下步骤操作：

1. 停止服务器
2. 保存对 `.env` 文件的更改
3. 运行 `source .env` 以刷新环境变量

## Windows 上的行尾问题

Windows 用户偶尔会遇到行尾问题，这可能会导致诸如“agent/bin/start is not a valid directory”之类的错误。

要解决此问题：

1. 配置 Git 以在 Windows 上自动将行尾转换为 LF：

   ```bash title="终端"
   git config --global core.autocrlf false
   ```

2. 重新克隆存储库

或者，您可以直接从 GitHub 下载并解压缩 ZIP 文件。


# 快速开始 (/cn/docs/ten_agent_examples/getting_started)

在本章节中，让我们一起构建 TEN Agent 的 Playground。

## 前提条件

import { Tab, Tabs } from 'fumadocs-ui/components/tabs'

<Tabs items={['API Keys', '安装', '最低系统要求']} defaultIndex={0}>
  <Tab>
    Agora [ App ID
    ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project)
    和 [ App Certificate
    ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project)
    (每月有免费额度)
  </Tab>

  <Tab>
    <p>
      [Docker](https://www.docker.com/) / [Docker
      Compose](https://docs.docker.com/compose/)
    </p>

    <p>
      [Node.js(LTS) v18](https://nodejs.org/en)
    </p>
  </Tab>

  <Tab>
    🎉 CPU >= 2 核

    😄 内存 >= 4 GB
  </Tab>
</Tabs>

**Apple Silicon 上的 Docker 设置**

<Callout>
  对于 Apple Silicon Mac，请在 Docker 设置中取消选中 "Use Rosetta for x86/amd64
  emulation"。注意：这可能会导致在 ARM 上的构建时间变慢，但在部署到 x64
  服务器时性能将恢复正常。
</Callout>

## 下一步

**1. 克隆 TEN Agent 仓库**

```sh title="终端"
git clone https://github.com/TEN-framework/TEN-Agent.git
```

**2. 准备配置文件**

在你的代码编辑器中打开 TEN Agent。在项目根目录中使用 `cd` 命令从 `.env.example` 创建 `.env` 文件。

```sh title="终端"
cp ./.env.example ./.env
```

**3. 在 .env 文件中设置 Agora App ID 和 App Certificate**

打开 `.env` 文件并填写 Agora App ID 和 App Certificate。这些将用于连接 Agora RTC 扩展。

```bash title=".env"
# Agora App ID
# Agora App Certificate(only required if enabled in the Agora Console)
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=

# Extension: elevenlabs_tts
# ElevenLabs TTS key
ELEVENLABS_TTS_KEY=

# Extension: deepgram_asr
# Deepgram ASR key
DEEPGRAM_API_KEY=
```

**4. 启动 agent 构建工具容器**

在同一目录下，运行 `docker` 命令来组合容器：

```bash title="终端"
docker compose up -d
```

**5. 进入容器**

使用以下命令进入容器：

```bash title="终端"
docker exec -it ten_agent_dev bash
```

**6. 构建 agent**

使用以下命令构建 agent：

```bash title="终端"
task use
```

**7. 启动 Web 服务器**

使用以下命令启动 Web 服务器：

```bash title="终端"
task run
```

**8. 编辑 Playground 设置**

在 [localhost:3000](http://localhost:3000) 打开 Playground 来配置你的 agent。

1. 选择一个图类型 (例如，语音 Agent，实时对话 Agent)
2. 选择一个对应的模块
3. 选择一个扩展并配置其 API 密钥设置 (若你还未在前面步骤的.env中配置环境变量)


# TEN Agent 示例概述 (/cn/docs/ten_agent_examples/overview)

TEN Agent 示例是您学习使用 TEN 框架构建对话式 AI 代理的综合指南。这些示例演示了创建实时、多模态语音代理的最佳实践，并与 Gemini 2.0 Live 和 OpenAI Realtime API 等现代 LLM 集成。

## 您将学到什么

TEN Agent 示例部分提供实际、实践性的文档，帮助您：

* **快速开始** — 在 10 分钟内设置并运行您的第一个语音代理
* **理解架构** — 了解 TEN Agent 如何编排 ASR、LLM、TTS 和 RTC 组件
* **自定义代理** — 通过模块化的"main"扩展来扩展和修改代理行为
* **构建扩展** — 为 LLM、语音服务和外部 API 创建自定义扩展
* **部署代理** — 在生产环境中打包和部署您的代理

## 文档结构

### 快速开始

如果您是 TEN Agent 的新手，请从这里开始。您将设置开发环境并运行您的第一个语音代理。

### 架构与设计

了解 TEN Agent 的工作原理，包括实时通信流、事件处理和组件交互。

### 自定义指南

学习如何通过灵活的 main 扩展模式来修改和扩展代理以满足您的特定需求。

### 扩展开发

为 LLM、STT/TTS 提供商和自定义集成构建自己的扩展。

### API 参考

事件、架构和配置选项的完整参考文档。

### 教程与示例

演示特定用例和模式的真实示例。

## 核心概念

**实时语音代理**：TEN Agent 支持全双工对话，具有自然的中断处理，提供无缝的语音交互。

**模块化架构**：通过自包含的扩展来扩展功能。核心代理逻辑位于"main"扩展中，您可以根据自己的用例进行定制。

**多语言支持**：使用 Python、Node.js、Go 或 C++ 构建扩展。在单个代理中混合和匹配语言。

**生产就绪**：TEN Agent 提供了大规模部署对话式 AI 所需的基础设施和模式。

## 相关资源

* [TEN Agent 仓库](https://github.com/TEN-framework/TEN-Agent) — 源代码和示例
* [TEN Framework 文档](/docs/ten_framework) — 框架深入探讨
* [TEN Framework 仓库](https://github.com/TEN-framework/ten-framework) — 核心框架


# RTC 优先的设计原理 (/cn/docs/ten_agent_examples/rtc_first_design)

本文档解释了为什么我们的系统采用 **RTC 优先** 策略，而不是主要基于 WebSockets 搭建。
对于开发者来说，这意味着你将获得一个 **低延迟、媒体优先的管道**，它已经内置好了，无需从零实现设备采集、编解码器或传输逻辑。

***

## 为什么选择 RTC 优先？

* **低延迟：** RTC 通过使用 UDP + SRTP 避免了 TCP 的队头阻塞。
* **媒体原生：** 可直接访问麦克风、摄像头和屏幕捕获。
* **编解码器集成：** 支持 Opus、VP8/VP9/AV1、H.264，并具备硬件加速。
* **同步能力：** 音频、视频和数据通道在同一会话中对齐。
* **跨平台：** 适用于浏览器、移动端和原生应用。

相比之下，**WebSockets** 更适合结构化数据交换，但缺乏：

* 原生媒体采集
* 编码/解码支持
* 带宽自适应
* 内置同步

***

## RTC 与 WebSocket 对比

| 功能     | RTC                        | WebSocket    |
| ------ | -------------------------- | ------------ |
| 传输协议   | UDP（SRTP、DTLS、ICE、拥塞控制）    | TCP（单一有序流）   |
| 延迟     | \~50–150ms（为实时优化）          | 更高（队头阻塞）     |
| 媒体支持   | 原生麦克风/摄像头/屏幕采集             | 无（仅原始二进制/文本） |
| 编码/解码  | 内置编解码器（Opus、VP8、H.264、AV1） | 外部/手动实现      |
| 同步能力   | 多流音视频同步 + 数据通道             | 需要手动实现       |
| 自适应能力  | 带宽估计、自适应码率、前向纠错（FEC）       | 不支持          |
| 最佳适用场景 | 通话、直播、AI Agents、实时应用       | 聊天、信令、异步事件   |

***

我们的系统默认集成了由 [Agora RTC](https://www.agora.io/en/) 提供支持的 RTC 模块，因此开发者无需操心设备采集、编解码或传输底层实现。

从实际使用角度来看：

* **配置最小化：** RTC 开箱即用。
* **WebSockets 仍然有用：** 非常适合信令、配置或偶发的异步更新。
* **RTC 是推荐通道：** 在实时、多模态交互中，延迟、同步和质量才是最重要的。

通过采用 **RTC 优先**，我们能够确保：

* 跨设备和平台的实时性能。
* 一致的开发体验，内置媒体管道。
* 能够灵活扩展异步特性，同时保持 RTC 的保证。

简而言之，开发者可以专注于 **构建功能**，而不是重新造一套传输栈。


# 构建系统 (/cn/docs/ten_framework/build_system)

## 构建系统

### C++

TEN framework 使用 `ten_gn` 构建系统来编译 C++ 应用程序和 C++ 扩展。 `ten_gn` 是一个基于 Google GN 的构建系统。

### Golang

TEN framework 使用标准的 go 命令来构建 TEN framework Golang 项目。

## 构建

### C++ 应用程序

要构建 TEN framework C++ 应用程序，请在 TEN framework C++ 应用程序的根目录中运行以下命令。

`<os>` 可以是以下之一：

* linux
* mac
* win

`<arch>` 可以是以下之一：

* x86
* x64
* arm
* arm64

`<build_type>` 可以是以下之一：

* debug
* release

```shell
ten_gn gen <os> <arch> <build_type>
ten_gn build <os> <arch> <build_type>
```

### C++ 插件

此处的术语 `addon` 包括以下内容：

* 扩展
* 扩展组
* 协议

构建 TEN framework C++ 插件的方法与构建 TEN framework C++ 应用程序的方法相同。

### Golang 应用程序

要构建 TEN Golang 项目，请在 TEN Golang 项目的根目录中运行以下命令：

```shell
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```


# 云商店 (/cn/docs/ten_framework/cloud_store)

TEN 云商店类似于 Android 上的 Google Play 和 iOS 上的 App Store，提供各种 TEN 软件包。 可以将这些软件包下载到 TEN 应用程序中，以帮助和增强开发。

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

## 数据库中存储的 TEN 软件包的信息

1. **软件包类型**

   表示软件包类型的字符串，例如 `extension`。

2. **软件包名称**

   表示软件包名称的字符串，例如 `builtin_extension`。

3. **软件包版本**

   遵循语义版本控制 (semver) 的字符串，用于指定软件包版本，例如 `3.0.0`。

4. **依赖项**

   对象数组，每个对象包含以下字段：

   * **软件包类型**

     字符串，与顶级的 `package type` 匹配。

   * **软件包名称**

     字符串，与顶级的 `package name` 匹配。

   * **版本**

     遵循 semver 的字符串，用于指定软件包的版本要求，例如 `~3.0.0`。


# 概述 (/cn/docs/ten_framework/concept_overview)

![TEN framework](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_framework.png)

## TEN runtime （TEN Runtime）

TEN runtime 是专为 TEN framework 设计的应用程序运行平台。它提供了一种灵活的架构，可以无缝集成用多种编程语言开发的扩展，并处理各种类型的数据流。通过对扩展的生命周期、数据流向和线程进行统一管理，TEN runtime 使开发者能够高效构建多样化的应用和服务。

## 应用（App）

TEN framework 中的应用可以作为独立的进程运行，也可以作为现有进程中的线程运行。 这种灵活性允许根据应用程序的需求选择不同的部署选项。

## 扩展组（Extension Group）

扩展组是一种为执行指定特定线程的机制。 给定组中的所有扩展都在同一线程上执行，从而确保一致和同步的处理。

## 扩展（Extension）

扩展是 TEN framework 的基本构建块。 开发人员可以使用各种编程语言创建扩展，并将它们组合起来以构建不同的应用程序和场景。 该框架的设计支持跨语言协作，使得不同语言编写的扩展能够在同一应用程序或服务中无缝地协同工作。

例如，由于 C++ 在处理音频和视频数据方面具有性能优势，因此开发人员可能会使用 C++ 创建用于实时通信 (RTC) 的扩展，同时使用 Python 开发 AI 扩展，以利用其强大的库进行数据分析和机器学习。 然后，可以将这些扩展集成到单个应用程序中，从而利用每种语言的优势。

## 图（Graph）

在 TEN framework 中，“图” (Graph) 描绘了扩展之间的数据流转过程。它编排了数据在不同扩展之间的流动方式，明确了数据流的参与者以及它们之间的交互关系。例如，可以将语音转文本（STT）扩展的输出定向到大语言模型（LLM）扩展进行进一步处理。

TEN framework 支持扩展之间的四种主要类型的数据流：

* 命令
* 数据
* 视频帧
* 音频帧

通过在图中定义这些数据流，开发者可以创建扩展之间的通信，以及单向的数据传输通道，这在处理音频和视频数据时尤其有用。

## 应用、组和扩展之间的关系

* **应用**：

  一个应用可以执行多个图，这些图可以是静态预定义的，也可以是动态组装的。

* **图**：

  图由多个协同工作的扩展构成，旨在创建一个有意义的场景。每个图的实例在应用程序内作为一个独立的会话运行。

* **扩展组**：

  扩展组的概念类似于线程。 以相同语言编写且位于同一扩展组中的扩展在运行时在同一线程上运行。 开发人员不需要直接管理线程； 他们只需"声明"每个扩展所属的组即可。

* **扩展**：

  框架中的每个扩展都分配有一个唯一的 ID，其结构为：`app-uri/graph-name/group-name/extension-name`

![概念的层次关系](https://ten-framework-assets.s3.amazonaws.com/doc-assets/hierarchical_relationship_of_concepts.png)

## TEN 云商店

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

TEN 云商店的功能类似于 Google Play 商店或 Apple 的 App Store，为扩展提供了一个市场。 开发人员可以共享他们的扩展或下载其他人创建的扩展。 这些扩展可以集成到 TEN 应用程序中，从而促进开发并扩展功能。

## TEN 管理器

TEN 管理器是一种简化扩展管理的工具。 它处理诸如上传、共享和安装扩展之类的任务，自动管理它们及其环境之间的依赖关系。 这使得扩展的安装和发布方便高效，从而简化了在 TEN framework 内的开发过程。


# 依赖项 (/cn/docs/ten_framework/dependencies)

TEN framework 利用了多个第三方库。有些专门用于测试，而另一些则集成到 TEN runtime 中。以下是对这些库的描述，以及在 TEN framework 中使用它们所需的任何必要修改。

## Google gn

实例 ID

| 操作系统  | 架构    | 实例 ID                                         |
| ----- | ----- | --------------------------------------------- |
| Linux | x64   | BzX0zkfwFeUn9MaDVqm6FugmTIy-hFpgNUx43O1fN00C  |
| Linux | arm64 | rT\_12w1Iv6ug8CJ4j0VQekA0qTDq6CwoAqGWasIKFcEC |
| Win   | x64   | 1QlqF0FPVt82ba5f48HxHpv5xPqOmyaThoR3TicuJ8QC  |

直接从 [Google GN 网页](https://chrome-infra-packages.appspot.com/p/gn/gn/) 下载。

Commit ID

| 操作系统 | 架构        | Commit ID                                |
| ---- | --------- | ---------------------------------------- |
| Mac  | universal | 18602f6cf1168cf78302024043edc02e8bad2ffb |

从[Gn仓库](https://gn.googlesource.com/gn)clone编译。

## Google ninja

版本：1.12.1

直接从 [Ninja 发布页面](https://github.com/ninja-build/ninja/releases) 下载。

## yyjson

版本：0.10.0

[MIT 许可证](https://github.com/ibireme/yyjson/blob/master/LICENSE)

这在 TEN framework 核心中用于解析和生成 JSON 数据。有关详细信息，请参阅 `third_party/yyjson`。

## libuv

版本：1.50.0

[MIT 许可证](https://github.com/libuv/libuv#licensing)

这是 TEN 运行时中使用的一个事件循环库。有关详细信息，请参阅 `third_party/libuv`。

## msgpack-c

版本：6.1.0

[Boost 软件许可证，版本 1.0](https://github.com/msgpack/msgpack-c#license)

用于 C 的 MessagePack 库。有关详细信息，请参阅 `third_party/msgpack`。

## libwebsockets

版本：4.3.2

[MIT 许可证](https://github.com/warmcat/libwebsockets/blob/main/LICENSE)

规范 libwebsockets.org 网络库。有关详细信息，请参阅 `third_party/libwebsockets`。

```diff
--- /home/wei/MyData/Temp/libwebsockets-4.3.2/CMakeLists.txt
+++ libwebsockets/CMakeLists.txt
@@ -851,6 +851,7 @@
  add_definitions(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE)
  # Fail the build if any warnings
  add_compile_options(/W3 /WX)
+ add_compile_options(/Zc:preprocessor /wd4819)
  # Unbreak MSVC broken preprocessor __VA_ARGS__ behaviour
  if (MSVC_VERSION GREATER 1925)
   add_compile_options(/Zc:preprocessor /wd5105)
```

应用以下补丁。

```diff
diff --git a/third_party/libwebsockets/cmake/lws_config.h.in b/third_party/libwebsockets/cmake/lws_config.h.in
index f3f4a9d79..e8d36c3ae 100644
--- a/third_party/libwebsockets/cmake/lws_config.h.in
+++ b/third_party/libwebsockets/cmake/lws_config.h.in
@@ -238,3 +238,9 @@
 #cmakedefine LWS_WITH_PLUGINS_API
 #cmakedefine LWS_HAVE_RTA_PREF

+// NOTE (Liu): The libwebsockets library will use external variables from mbedtls
+// if 'LWS_WITH_MBEDTLS' is enabled. On Windows, variable declarations in the header
+// file must start with '__declspec(dllimport)', otherwise, the
+// symbols cannot be accessed.
+// Refer to third_party/mbedtls/include/mbedtls/export.h.
+#define USING_SHARED_MBEDTLS
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
index e8a0cb2d4..84a164e90 100755
--- a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
+++ b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
@@ -183,7 +183,12 @@ int ssl_pm_new(SSL *ssl)
         mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, version);
     } else {
         mbedtls_ssl_conf_max_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
-    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, 1);
+
+    // NOTE: mbedtls added 'ssl_conf_version_check()' since v3.1.0, and
+    // mbedtls only enables TLS 1.2 by default. So the 'min_tls_version' and
+    // 'max_tls_version' must be '0x303', or 'mbedtls_ssl_setup' will
+    // fail.
+    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
     }

     mbedtls_ssl_conf_rng(&ssl_pm->conf, mbedtls_ctr_drbg_random, &ssl_pm->ctr_drbg);
```

```diff
diff --git a/third_party/libwebsockets/CMakeLists.txt b/third_party/libwebsockets/CMakeLists.txt
index 92638143a..746f9b6a6 100644
--- a/third_party/libwebsockets/CMakeLists.txt
+++ b/third_party/libwebsockets/CMakeLists.txt
@@ -547,9 +547,12 @@ SET(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/lib")

 SET(LWS_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}")

-# Put absolute path of dynamic libraries into the object code. Some
-# architectures, notably Mac OS X, need this.
-SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
+# Commented out to avoid using absolute paths in the install_name on macOS.
+# When CMAKE_INSTALL_NAME_DIR is set to an absolute path, binaries that link
+# against libwebsockets will hardcode this absolute path, making the library
+# difficult to relocate and potentially causing "library not found" errors at
+# runtime if the library is installed in a different location.
+# SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
```

### 修复在 Windows 上链接 mbedtls

如果 CMake 版本高于 3.24，请应用以下补丁，因为 `find_package` 自 3.24 起支持 `GLOBAL` 关键字。

```diff
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
index e34151724..79b15089d 100644
--- a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
+++ b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
-        find_library(MBEDTLS_LIBRARY mbedtls)
-        find_library(MBEDX509_LIBRARY mbedx509)
-        find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+        # find_library(MBEDTLS_LIBRARY mbedtls)
+        # find_library(MBEDX509_LIBRARY mbedx509)
+        # find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+        # set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+
+        # Set the custom search path.
+        set(MBEDTLS_DIR "${MBEDTLS_BUILD_PATH}/cmake")
+
+        // NOTE (Liu): We should use 'find_package' rather than 'find_library'
+        // to search for the mbedtls libraries. 'find_library' only finds a
+        // library, shared or static, without any settings from the external
+        // library.

-        set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+        // The 'lib/tls/CMakeLists.txt' depends on the mbedtls libraries, so we set the scope to 'GLOBAL' here.
+        find_package(MbedTLS GLOBAL)
+        set(LWS_MBEDTLS_LIBRARIES MbedTLS::mbedcrypto MbedTLS::mbedx509 MbedTLS::mbedtls)
```

```diff
diff --git a/third_party/libwebsockets/BUILD.gn b/third_party/libwebsockets/BUILD.gn
index 4c89c2e2e..e6d641b9c 100644
--- a/third_party/libwebsockets/BUILD.gn
+++ b/third_party/libwebsockets/BUILD.gn
@@ -84,6 +84,10 @@ cmake_project("websockets") {
   }

   options += [
+  // The libwebsockets will use the 'MBEDTLS_BUILD_PATH' variable to set the
+  // search path of 'find_package'.
+  "MBEDTLS_BUILD_PATH=" + rebase_path("$root_gen_dir/cmake/mbedtls"),
+
```

并删除 `third_party/libwebsockets/cmake/lws_config.h.in` 中的 `#define USING_SHARED_MBEDTLS`。

## curl

版本：8.1.2

[类 MIT 许可证](https://github.com/curl/curl/blob/master/COPYING)

一种使用 URL 语法传输数据的命令行工具和库，支持 DICT、FILE、FTP、FTPS、GOPHER、GOPHERS、HTTP、HTTPS、IMAP、IMAPS、LDAP、LDAPS、MQTT、POP3、POP3S、RTMP、RTMPS、RTSP、SCP、SFTP、SMB、SMBS、SMTP、SMTPS、TELNET 和 TFTP。libcurl 提供了无数强大的功能。

有关详细信息，请参阅 `third_party/curl`。

### curl 的补丁

修补 `lib/CMakeLists.txt`，将共享库名称从 `_imp.lib` 更改为 `.lib`。

```cmake
if(WIN32)
  if(BUILD_SHARED_LIBS)
    if(MSVC)
      // Add "_imp" as a suffix before the extension to avoid conflicting with
      // the statically linked "libcurl.lib"
      //set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX "_imp.lib")
      set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX ".lib")
    endif()
  endif()
endif()
```

在 `lib/ws.h` 中导出 `Curl_ws_done`，因为需要调用此函数以防止内存泄漏。

```c
CURL_EXTERN void Curl_ws_done(struct Curl_easy *data);
// ^^^^^^^^
```

## clingo

版本：5.8.0

[MIT 许可证](https://github.com/potassco/clingo/blob/master/LICENSE.md)

一种用于逻辑程序的接地器和求解器。

有关详细信息，请参阅 `third_party/clingo`。

ten\_manager 实际上使用 clingo-sys crate 来调用 clingo，而 clingo-sys 包含了自己的 clingo 源代码。然而，这个包含的 clingo 源代码是 5.6.2 版本，在 cmake 4 环境中存在编译错误。因此，我们需要使用 cargo 的 patch 功能来使用最新的本地版本 clingo 源代码（已修复了 cmake 4 编译错误）。此外，由于 clingo-sys 已经包含了 clingo 源代码，我们实际上并不需要 third\_party/clingo 文件夹。但是，由于 clingo-sys 在 Windows 上仍然将 clingo 编译为共享库，我们目前仍然需要使用 third\_party/clingo 文件夹在 Windows 上编译 clingo 的共享库，然后使用 gn 脚本将 Windows 共享库复制到 ten\_manager 的最终输出文件夹。未来，我们应该直接使用从 clingo-sys 中包含的 clingo 源代码编译的 Windows 共享库进行复制，这样就可以移除 third\_party/clingo 文件夹。

## FFmpeg

版本：6.0

[GPL 或 LGPL 许可证](https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md)

FFmpeg 代码库主要采用 LGPL 许可，可选组件采用 GPL 许可。有关详细信息，请参阅 LICENSE 文件。

用于 ffmpeg 扩展，主要用于测试目的。有关详细信息，请参阅 `third_party/ffmpeg`。

## libbacktrace

版本：1.0

[BSD 许可证](https://github.com/ianlancetaylor/libbacktrace/blob/master/LICENSE)

一个可以链接到 C/C++ 程序中以生成符号回溯的 C 库。

> ⚠️ **注意：**
> 我们已经对 `libbacktrace` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/backtrace`。

## uthash

版本：2.3.0

[BSD 许可证](https://github.com/troydhanson/uthash/blob/master/LICENSE)

用于哈希表等的 C 宏。

> ⚠️ **注意：**
> 我们已经对 `uthash` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅文件头中提到 `uthash` 的 `core/include/ten_utils/container` 下的文件。

## uuid4

[MIT 许可证](https://github.com/gpakosz/uuid4/blob/master/LICENSE.MIT)
[WTFPLv2 许可证](https://github.com/gpakosz/uuid4/blob/master/LICENSE.WTFPLv2)

C 中的 UUID v4 生成。

> ⚠️ **注意：**
> 我们已经对 `uuid4` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/lib/sys/general/uuid.c`。

## zf\_log

[MIT 许可证](https://github.com/wonder-mice/zf_log/blob/master/LICENSE)

用于 C/ObjC/C++ 的核心日志库。

> ⚠️ **注意：**
> 我们已经对 `zf_log` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/log`。


# 消息系统 (/cn/docs/ten_framework/message_system)

## 消息类型

在 TEN framework 中，消息被分为四种类型：

1. 命令 (Command)
2. 数据 (Data)
3. 视频帧 (Video Frame)
4. 音频帧 (Audio Frame)

命令和非命令消息类型之间的主要区别在于，命令会产生结果，而非命令则不会。

与这些类型相对应，扩展有四个消息回调函数：

1. `OnCmd`
2. `OnData`
3. `OnVideoFrame`
4. `OnAudioFrame`

以下是 TEN framework 消息的分类图：

```text
┌── 有结果
│   └── 命令 (Command)
│       ├── TEN framework 内置命令
│       │    => 消息名称以 `ten:` 开头
│       └── 非 TEN framework 内置命令
│            => 消息名称不以 `ten:` 开头
└── 无结果
    ├── 数据 (Data)
    │   ├── TEN framework 内置数据
    │   │    => 消息名称以 `ten:` 开头
    │   └── 非 TEN framework 内置数据
    │        => 消息名称不以 `ten:` 开头
    ├── 视频帧 (Video Frame)
    │   ├── TEN framework 内置视频帧
    │   │    => 消息名称以 `ten:` 开头
    │   └── 非 TEN framework 内置视频帧
    │        => 消息名称不以 `ten:` 开头
    └── 音频帧 (Audio Frame)
        ├── TEN framework 内置音频帧
        │    => 消息名称以 `ten:` 开头
        └── 非 TEN framework 内置音频帧
             => 消息名称不以 `ten:` 开头
```

## 消息名称

在 TEN framework 中，消息名称用于区分相同类型但用途不同的消息。扩展使用不同的消息名称来决定它们需要执行的操作。

消息名称的命名规则如下：

1. 第一个字符必须是字母 (`a-z`、`A-Z`) 或下划线 (`_`)。
2. 后续字符可以是字母 (`a-z`、`A-Z`)、数字 (`0-9`) 或下划线 (`_`)。

## 消息流

![消息流](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_flow.png)

TEN framework 支持三种消息流模式：

1. **单消息，单结果：**

   适用于一个动作产生单个结果的情况。

2. **单消息，多结果：**

   适用于一个动作产生多个结果的情况。

3. **多消息：**

   适用于不需要结果的动作。

## 扩展的消息接口

扩展的 `manifest.json` 文件中的 `api` 字段定义了该扩展的外部消息接口。`api` 字段可以包含以下类型的 API 信息：

* `cmd_in`
* `cmd_out`
* `data_in`
* `data_out`
* `audio_frame_in`
* `audio_frame_out`
* `video_frame_in`
* `video_frame_out`

以下是一个 API 定义的示例：

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

TEN framework 中不同扩展之间的消息配对机制类似于传统编程语言中的函数调用。以下是一个简单的类比：

1. 消息名称类似于函数名称，扩展消息之间的配对机制类似于函数调用机制。
2. 消息属性类似于函数参数。
3. 当 TEN framework 确定扩展 A 的输出消息 `foo` 与扩展 B 的输入消息 `foo` 配对时，这类似于扩展 A 中的函数指针指向扩展 B 中的函数 `foo`。
4. 配对仅基于消息名称，而不是消息属性。这意味着 TEN framework 不支持类似于函数重载的机制。

因此，消息名称对于每个扩展的输入/输出消息至关重要，充当该扩展的控制平面接口。

尽管 TEN framework 在单消息级别提供了与语言无关的调用机制，但它还允许通过 TEN 模式对扩展之间的调用关系进行静态和动态检查。这有助于检测潜在问题，例如源扩展和目标扩展之间对属性类型的不同解释。用户可以定义自己的消息名称，并通过图确定扩展之间的调用关系。TEN framework 还提供了静态检查工具，以确保这些关系不会导致任何问题。

## TEN framework 中的消息所有权概念

在 TEN framework 中，一旦消息成功发送，发送消息的扩展将无法再使用它。此概念对于维护框架内的消息安全和线程安全至关重要。

![消息处理遵循所有权概念](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_ownership.png)

消息可以沿两个方向发送：**发送 (send)** 和 **返回 (return)**。只有命令消息具有返回方向；数据、音频帧和视频帧消息则没有。消息发送后，可能会经过各种处理步骤，允许扩展继续使用该消息可能会导致线程安全问题。例如，处理消息的另一个扩展可能在不同的线程中运行。即使初始扩展继续使用该消息没有任何明显问题，此行为也是未定义的，开发人员不应依赖它。

TEN runtime 实施了各种安全措施，以确保扩展在消息成功发送后无法使用该消息。

### 消息所有权转移至扩展

当 TEN framework 通过 `OnCmd`、`OnData`、`OnVideoFrame` 或 `OnAudioFrame` 回调函数将消息传递给扩展时，框架会将消息的所有权转移给该扩展。这种所有权的转移意味着 TEN framework 保证只有接收扩展才能访问该消息，从而确保消息安全和线程安全。该扩展可以自由使用该消息，并且在不再需要该消息时必须将所有权返回给 TEN framework。返回所有权后，该扩展不得再与该消息进行任何交互。

### 消息所有权转移至 TEN runtime

当扩展使用 `SendCmd`、`ReturnResult`、`SendData`、`SendVideoFrame` 或 `SendAudioFrame` 等 API 将消息发送回 TEN runtime 时，该扩展会将消息的所有权返回给 TEN runtime。

### 与消息相关的数据的所有权

所有权的概念不仅限于消息本身，还延伸到与该消息关联的所有资源。与消息一样，每个资源的所有权一次只能属于一个扩展，并且它反映了消息的所有权。这种设计确保了 TEN framework 中的消息和线程安全。

例如，如果扩展 A 拥有一个消息并使用该消息中的缓冲区，则它必须在不再需要它们的时候将消息和缓冲区的所有权都返回给 TEN runtime。

### 默认复制语义

为了防止多个扩展同时处理同一资源而导致数据竞争等问题，TEN framework 在跨 TEN runtime 和扩展边界传输数据时默认使用**复制语义**。复制确保了所有权的分离，从而维护安全性。

### 借用语义

TEN framework 还支持**借用语义**，允许扩展使用资源而无需取得所有权。这种方法在保持消息和线程安全的同时，提高了资源的利用率。借用 API 必须与 GiveBack API 结合使用，以确保资源的安全性和完整性。

当扩展将消息的所有权返回给 TEN runtime 时，如果所有关联的资源也已返回，则运行平台才会接受该消息。例如，如果扩展 A 借用了数据消息的缓冲区，但在返回消息本身之前没有返回缓冲区的所 有权，则 TEN runtime 将拒绝该消息（例如，`SendData` 将会失败）。

### 借用 API

扩展可以使用借用 API 直接访问消息中的资源，从而实现临时使用而无需转移所有权。

### GiveBack API

GiveBack API 用于将资源的所有权返回给 TEN runtime，确保资源得到安全管理并在不再需要时被释放。

## 命令

TEN 客户端和 TEN 扩展可以向其他 TEN 扩展发送命令。

如果命令用于不需要结果的场景，则没有问题。TEN runtime 将自动丢弃任何不必要的结果。

### 结果

结果是对命令的响应。例如，当 TEN 扩展 A 向 TEN 扩展 B 发送命令 X 时，在处理完该命令后，TEN 扩展 B 会将结果发送回 TEN 扩展 A。结果可以包含任何有意义的细节，使 TEN 扩展 A 能够获得有关执行结果的更多信息。

一个典型的结果的 JSON 表示形式可能如下所示：

```json
{
  "ten": {
    "status_code": 400
  }
}
```


# 准备工作 (/cn/docs/ten_framework/preparation)

TEN framework 支持 Windows、Linux 和 Mac 作为开发环境，但建议使用 Linux 或 Mac。

## 开发环境

### Linux

以 Ubuntu 为例，您必须首先安装以下软件包：

* gcc：TEN framework 支持的 C/C++ 编译器之一。
* clang：TEN framework 支持的另一种 C/C++ 编译器。
* clang-format：TEN framework 在运行时使用的代码格式化工具。
* clang-tidy：静态代码分析器。
* cmake：构建系统。版本必须大于 3.13。
* Python3：TEN framework Python 绑定所需。
* pytest：用于 TEN framework 的集成测试。

### Mac

要设置您的 Mac 开发环境，您需要以下内容：

* XCode
* CocoaPods
* 个人账户

按照以下说明准备环境：

```shell
brew install llvm googletest doxygen ninja clang-format
brew install include-what-you-use
```

由于通过 `brew` 安装的 `cmake` 版本可能已过时，那么您需要从官方网站安装 `cmake` 并设置符号链接。

```shell
ln -s /Applications/CMake.app/Contents/bin/cmake /usr/local/bin/cmake
```

如果 `cmake` 找不到 `clang-tidy`，您可以使用以下命令解决此问题：

```shell
ln -sf /usr/local/opt/llvm/bin/clang-tidy /usr/local/bin/clang-tidy
```

要构建 TEN framework 的各种语言绑定，您可以使用以下命令安装所需的语言：

```shell
brew install python golang
```

### Windows

如果您需要在 Microsoft Windows 上开发 TEN framework，您可以手动安装以下软件包：

* Visual Studio 社区版（最高至 2022）

  请务必选择与 clang 相关的工具并将 clang 二进制文件路径添加到 `PATH` 环境变量。

* cmake

* python

> ⚠️ **注意：**
> 在 Windows 上安装 `cmake` / `python` 时，安装路径不应包含空格。

## ten\_gn

TEN framework 使用 `ten_gn` 作为其构建系统。`ten_gn` 是一个基于 Google GN 的构建系统。`ten_gn` 的源代码位于 TEN framework 仓库的 `core/ten_gn` 目录中。请将 `ten_gn` 目录路径添加到您开发环境的系统 `PATH` 环境变量中。

## 使用 Docker 容器

我们提供了预先编写的 Docker 文件，允许您创建一个容器，其中包含从源代码构建 TEN framework 所需的所有必要软件包。

### Ubuntu 22.04

导航到 `tools/docker_for_building/ubuntu/22.04` 并运行以下命令来创建并进入构建环境。

```shell
docker-compose up -d
docker-compose run ten-building-ubuntu-2204
```


# 性能分析 (/cn/docs/ten_framework/profiling)

## TEN framework GO 项目

使用 `gperftools` 来分析您的程序的 CPU 使用率和堆内存使用率。以下步骤介绍了如何准备环境、使用 `gperftools` 来分析您的程序，以及分析性能数据。

### 安装 `gperftools`

要在 Ubuntu 上安装 `gperftools`，请运行以下命令：

```shell
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

### 使用 `gperftools` 分析 CPU 使用率

要分析您的程序的 CPU 使用率，请运行以下命令：

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/.../cpu

exec /.../program
```

程序退出时，将生成一个 CPU 性能数据文件。数据将存储在 `/.../cpu.0001.prof` 文件中。

如果您想定期生成 CPU 性能数据，可以设置 `CPUPROFILE_FREQUENCY` 环境变量。例如，要每 30 秒生成一次 CPU 性能数据，请运行以下命令：

```shell
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/.../cpu CPUPROFILE_FREQUENCY=30 exec /.../program
```

#### 分析 CPU 性能数据

要分析 CPU 性能数据，请运行以下命令：

```shell
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### 使用 `gperftools` 分析堆内存使用率

要分析您的程序的堆内存使用率，请运行以下命令：

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

建议使用 `HEAP_PROFILE_TIME_INTERVAL` 环境变量来定期生成堆内存性能数据。所有堆内存性能数据将存储在 `HEAPPROFILE` 目录中。

#### 分析堆内存性能数据

每个生成的堆内存性能数据文件都将具有 `.heap` 扩展名。要分析堆内存性能数据，您可以将 `.heap` 文件转换为人类可读的文本格式。例如，要将 `/.../heap` 中的 `.heap` 文件转换为文本格式，请运行以下命令：

```shell
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

性能数据将如下所示：

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

您还可以将 `.heap` 文件转换为其他格式，例如 PDF 或 SVG。例如，要将 `/.../heap` 中的 `.heap` 文件转换为 PDF 格式，请运行以下命令：

```shell
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

PDF 格式的性能数据将与文本输出类似，但以图形方式可视化。

此外，您可以比较两组堆内存性能数据。例如，要比较 `/.../heap.0001.heap` 和 `/.../heap.0002.heap` 中的堆内存性能数据，请运行以下命令：

```shell
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

PDF 将显示两组数据之间堆内存使用率的差异。

如果您定期生成堆内存性能数据，我们提供了一个脚本来分析目录中的所有堆内存性能数据，并生成一个 Excel 文件，显示堆内存使用率趋势。例如，要分析 `/.../heap` 中的所有堆内存性能数据，请运行以下命令：

```shell
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

该脚本将在 `/.../raw` 目录中生成带有符号的原始性能数据，在 `/.../text` 目录中生成人类可读的文本性能数据，并在 `/.../heap.xlsx` 中生成 Excel 文件。

您可以绘制折线图以显示堆内存使用率趋势。

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### 使用 `pprof` 进行分析

我们建议使用 `pprof` 来分析用 Go 编写的程序。分析 Go 程序的最简单方法是将 `pprof_app_go` 用作您的 TEN 应用程序。激活后，应用程序会读取环境变量，以确定是否激活 `pprof` 服务器和堆分析器。

```shell
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### 分析 `pprof` 性能数据

要将堆性能数据转换为文本格式，请使用以下命令：

```shell
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

要分析所有堆性能数据并生成 Excel 文件，请运行：

```shell
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

您可以绘制折线图以显示堆内存使用率趋势：

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

这将为您提供 Go 堆性能数据趋势的可视化表示。


# 版本系统 (/cn/docs/ten_framework/version_system)

TEN framework 遵循 semver 规范。

## TEN 包的版本信息

TEN 包的版本信息记录在其自身的 `manifest.json` 文件的 `version` 字段中。对于每个 TEN 包，此 `version` 字段都是强制性的。

```json
{
  "version": "1.0.0"
}
```

## 依赖项的版本要求

> ⚠️ **注意：**
> 如果未指定依赖项的版本要求，则默认使用最新版本。

```text
└── ten_packages/
    ├── extension/
    ├── extension_group/
    ├── protocol/
    └── system/
```

这些包通过其 `manifest.json` 文件中 `type` 字段的值来区分。

```json
{
  "type": "system" // "extension", "extension_group", "protocol"
}
```

在 TEN 包中，依赖的 TEN 包的版本要求指定如下。

```json
{
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.1"
    },
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "1.0.3"
    },
    {
      "type": "extension",
      "name": "default_http_extension_cpp",
      "version": "1.0.0"
    },
    {
      "type": "extension_group",
      "name": "default_extension_group",
      "version": "1.0.0"
    }
  ]
}
```

## 版本要求规范

版本要求字符串可以包含多个版本要求字符串，以逗号分隔。每个版本要求字符串支持以下运算符：

* `>`、`>=`、`=`

  手动指定版本范围或确切的依赖项版本。

* `~`

  用于指定具有一定更新灵活性的最低版本。如果指定了 major、minor 和 patch 版本，或者仅指定了 major 和 minor 版本，则仅允许 patch 级别的更改。如果仅指定了 major 版本，则允许 minor 和 patch 级别的更改。

* `^`

  可以省略 `^` 运算符，这意味着如果版本要求字符串不包含任何运算符，则默认为 `^`。例如，字符串 `0.1.12` 表示 `^0.1.12`。虽然 `0.1.12` 看起来像一个特定版本，但它实际上指定了一个允许与 SemVer 兼容的更新的版本要求。

  只要不修改 major、minor 或 patch 组合中最左边的非零数字，就允许更新。在这种情况下，如果您升级 TEN 包，它将更新到 `0.1.13` 版本（如果是最新的 `0.1.z` 版本），但不会更新到 `0.2.0`。如果指定的版本字符串为 `1.0`，它将更新到 `1.1`（如果是最新的 `1.y` 版本），但不会更新到 `2.0`。

  > ⚠️ **注意：**
  > 版本 `0.0.x` 被认为与任何其他版本都不兼容。


# 构建系统 (/cn/docs/ten_framework/0.11.59/build_system)

## 构建系统

### C++

TEN framework 使用 `ten_gn` 构建系统来编译 C++ 应用程序和 C++ 扩展。 `ten_gn` 是一个基于 Google GN 的构建系统。

### Golang

TEN framework 使用标准的 go 命令来构建 TEN framework Golang 项目。

## 构建

### C++ 应用程序

要构建 TEN framework C++ 应用程序，请在 TEN framework C++ 应用程序的根目录中运行以下命令。

`<os>` 可以是以下之一：

* linux
* mac
* win

`<arch>` 可以是以下之一：

* x86
* x64
* arm
* arm64

`<build_type>` 可以是以下之一：

* debug
* release

```shell
ten_gn gen <os> <arch> <build_type>
ten_gn build <os> <arch> <build_type>
```

### C++ 插件

此处的术语 `addon` 包括以下内容：

* 扩展
* 扩展组
* 协议

构建 TEN framework C++ 插件的方法与构建 TEN framework C++ 应用程序的方法相同。

### Golang 应用程序

要构建 TEN Golang 项目，请在 TEN Golang 项目的根目录中运行以下命令：

```shell
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```


# 云商店 (/cn/docs/ten_framework/0.11.59/cloud_store)

TEN 云商店类似于 Android 上的 Google Play 和 iOS 上的 App Store，提供各种 TEN 软件包。 可以将这些软件包下载到 TEN 应用程序中，以帮助和增强开发。

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

## 数据库中存储的 TEN 软件包的信息

1. **软件包类型**

   表示软件包类型的字符串，例如 `extension`。

2. **软件包名称**

   表示软件包名称的字符串，例如 `builtin_extension`。

3. **软件包版本**

   遵循语义版本控制 (semver) 的字符串，用于指定软件包版本，例如 `3.0.0`。

4. **依赖项**

   对象数组，每个对象包含以下字段：

   * **软件包类型**

     字符串，与顶级的 `package type` 匹配。

   * **软件包名称**

     字符串，与顶级的 `package name` 匹配。

   * **版本**

     遵循 semver 的字符串，用于指定软件包的版本要求，例如 `~3.0.0`。


# 概述 (/cn/docs/ten_framework/0.11.59/concept_overview)

![TEN framework](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_framework.png)

## TEN runtime （TEN Runtime）

TEN runtime 是专为 TEN framework 设计的应用程序运行平台。它提供了一种灵活的架构，可以无缝集成用多种编程语言开发的扩展，并处理各种类型的数据流。通过对扩展的生命周期、数据流向和线程进行统一管理，TEN runtime 使开发者能够高效构建多样化的应用和服务。

## 应用（App）

TEN framework 中的应用可以作为独立的进程运行，也可以作为现有进程中的线程运行。 这种灵活性允许根据应用程序的需求选择不同的部署选项。

## 扩展组（Extension Group）

扩展组是一种为执行指定特定线程的机制。 给定组中的所有扩展都在同一线程上执行，从而确保一致和同步的处理。

## 扩展（Extension）

扩展是 TEN framework 的基本构建块。 开发人员可以使用各种编程语言创建扩展，并将它们组合起来以构建不同的应用程序和场景。 该框架的设计支持跨语言协作，使得不同语言编写的扩展能够在同一应用程序或服务中无缝地协同工作。

例如，由于 C++ 在处理音频和视频数据方面具有性能优势，因此开发人员可能会使用 C++ 创建用于实时通信 (RTC) 的扩展，同时使用 Python 开发 AI 扩展，以利用其强大的库进行数据分析和机器学习。 然后，可以将这些扩展集成到单个应用程序中，从而利用每种语言的优势。

## 图（Graph）

在 TEN framework 中，“图” (Graph) 描绘了扩展之间的数据流转过程。它编排了数据在不同扩展之间的流动方式，明确了数据流的参与者以及它们之间的交互关系。例如，可以将语音转文本（STT）扩展的输出定向到大语言模型（LLM）扩展进行进一步处理。

TEN framework 支持扩展之间的四种主要类型的数据流：

* 命令
* 数据
* 视频帧
* 音频帧

通过在图中定义这些数据流，开发者可以创建扩展之间的通信，以及单向的数据传输通道，这在处理音频和视频数据时尤其有用。

## 应用、组和扩展之间的关系

* **应用**：

  一个应用可以执行多个图，这些图可以是静态预定义的，也可以是动态组装的。

* **图**：

  图由多个协同工作的扩展构成，旨在创建一个有意义的场景。每个图的实例在应用程序内作为一个独立的会话运行。

* **扩展组**：

  扩展组的概念类似于线程。 以相同语言编写且位于同一扩展组中的扩展在运行时在同一线程上运行。 开发人员不需要直接管理线程； 他们只需"声明"每个扩展所属的组即可。

* **扩展**：

  框架中的每个扩展都分配有一个唯一的 ID，其结构为：`app-uri/graph-name/group-name/extension-name`

![概念的层次关系](https://ten-framework-assets.s3.amazonaws.com/doc-assets/hierarchical_relationship_of_concepts.png)

## TEN 云商店

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

TEN 云商店的功能类似于 Google Play 商店或 Apple 的 App Store，为扩展提供了一个市场。 开发人员可以共享他们的扩展或下载其他人创建的扩展。 这些扩展可以集成到 TEN 应用程序中，从而促进开发并扩展功能。

## TEN 管理器

TEN 管理器是一种简化扩展管理的工具。 它处理诸如上传、共享和安装扩展之类的任务，自动管理它们及其环境之间的依赖关系。 这使得扩展的安装和发布方便高效，从而简化了在 TEN framework 内的开发过程。


# 依赖项 (/cn/docs/ten_framework/0.11.59/dependencies)

TEN framework 利用了多个第三方库。有些专门用于测试，而另一些则集成到 TEN runtime 中。以下是对这些库的描述，以及在 TEN framework 中使用它们所需的任何必要修改。

## Google gn

实例 ID

| 操作系统  | 架构    | 实例 ID                                         |
| ----- | ----- | --------------------------------------------- |
| Linux | x64   | BzX0zkfwFeUn9MaDVqm6FugmTIy-hFpgNUx43O1fN00C  |
| Linux | arm64 | rT\_12w1Iv6ug8CJ4j0VQekA0qTDq6CwoAqGWasIKFcEC |
| Win   | x64   | 1QlqF0FPVt82ba5f48HxHpv5xPqOmyaThoR3TicuJ8QC  |

直接从 [Google GN 网页](https://chrome-infra-packages.appspot.com/p/gn/gn/) 下载。

Commit ID

| 操作系统 | 架构        | Commit ID                                |
| ---- | --------- | ---------------------------------------- |
| Mac  | universal | 18602f6cf1168cf78302024043edc02e8bad2ffb |

从[Gn仓库](https://gn.googlesource.com/gn)clone编译。

## Google ninja

版本：1.12.1

直接从 [Ninja 发布页面](https://github.com/ninja-build/ninja/releases) 下载。

## yyjson

版本：0.10.0

[MIT 许可证](https://github.com/ibireme/yyjson/blob/master/LICENSE)

这在 TEN framework 核心中用于解析和生成 JSON 数据。有关详细信息，请参阅 `third_party/yyjson`。

## libuv

版本：1.50.0

[MIT 许可证](https://github.com/libuv/libuv#licensing)

这是 TEN 运行时中使用的一个事件循环库。有关详细信息，请参阅 `third_party/libuv`。

## msgpack-c

版本：6.1.0

[Boost 软件许可证，版本 1.0](https://github.com/msgpack/msgpack-c#license)

用于 C 的 MessagePack 库。有关详细信息，请参阅 `third_party/msgpack`。

## libwebsockets

版本：4.3.2

[MIT 许可证](https://github.com/warmcat/libwebsockets/blob/main/LICENSE)

规范 libwebsockets.org 网络库。有关详细信息，请参阅 `third_party/libwebsockets`。

```diff
--- /home/wei/MyData/Temp/libwebsockets-4.3.2/CMakeLists.txt
+++ libwebsockets/CMakeLists.txt
@@ -851,6 +851,7 @@
  add_definitions(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE)
  # Fail the build if any warnings
  add_compile_options(/W3 /WX)
+ add_compile_options(/Zc:preprocessor /wd4819)
  # Unbreak MSVC broken preprocessor __VA_ARGS__ behaviour
  if (MSVC_VERSION GREATER 1925)
   add_compile_options(/Zc:preprocessor /wd5105)
```

应用以下补丁。

```diff
diff --git a/third_party/libwebsockets/cmake/lws_config.h.in b/third_party/libwebsockets/cmake/lws_config.h.in
index f3f4a9d79..e8d36c3ae 100644
--- a/third_party/libwebsockets/cmake/lws_config.h.in
+++ b/third_party/libwebsockets/cmake/lws_config.h.in
@@ -238,3 +238,9 @@
 #cmakedefine LWS_WITH_PLUGINS_API
 #cmakedefine LWS_HAVE_RTA_PREF

+// NOTE (Liu): The libwebsockets library will use external variables from mbedtls
+// if 'LWS_WITH_MBEDTLS' is enabled. On Windows, variable declarations in the header
+// file must start with '__declspec(dllimport)', otherwise, the
+// symbols cannot be accessed.
+// Refer to third_party/mbedtls/include/mbedtls/export.h.
+#define USING_SHARED_MBEDTLS
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
index e8a0cb2d4..84a164e90 100755
--- a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
+++ b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
@@ -183,7 +183,12 @@ int ssl_pm_new(SSL *ssl)
         mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, version);
     } else {
         mbedtls_ssl_conf_max_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
-    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, 1);
+
+    // NOTE: mbedtls added 'ssl_conf_version_check()' since v3.1.0, and
+    // mbedtls only enables TLS 1.2 by default. So the 'min_tls_version' and
+    // 'max_tls_version' must be '0x303', or 'mbedtls_ssl_setup' will
+    // fail.
+    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
     }

     mbedtls_ssl_conf_rng(&ssl_pm->conf, mbedtls_ctr_drbg_random, &ssl_pm->ctr_drbg);
```

```diff
diff --git a/third_party/libwebsockets/CMakeLists.txt b/third_party/libwebsockets/CMakeLists.txt
index 92638143a..746f9b6a6 100644
--- a/third_party/libwebsockets/CMakeLists.txt
+++ b/third_party/libwebsockets/CMakeLists.txt
@@ -547,9 +547,12 @@ SET(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/lib")

 SET(LWS_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}")

-# Put absolute path of dynamic libraries into the object code. Some
-# architectures, notably Mac OS X, need this.
-SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
+# Commented out to avoid using absolute paths in the install_name on macOS.
+# When CMAKE_INSTALL_NAME_DIR is set to an absolute path, binaries that link
+# against libwebsockets will hardcode this absolute path, making the library
+# difficult to relocate and potentially causing "library not found" errors at
+# runtime if the library is installed in a different location.
+# SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
```

### 修复在 Windows 上链接 mbedtls

如果 CMake 版本高于 3.24，请应用以下补丁，因为 `find_package` 自 3.24 起支持 `GLOBAL` 关键字。

```diff
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
index e34151724..79b15089d 100644
--- a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
+++ b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
-        find_library(MBEDTLS_LIBRARY mbedtls)
-        find_library(MBEDX509_LIBRARY mbedx509)
-        find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+        # find_library(MBEDTLS_LIBRARY mbedtls)
+        # find_library(MBEDX509_LIBRARY mbedx509)
+        # find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+        # set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+
+        # Set the custom search path.
+        set(MBEDTLS_DIR "${MBEDTLS_BUILD_PATH}/cmake")
+
+        // NOTE (Liu): We should use 'find_package' rather than 'find_library'
+        // to search for the mbedtls libraries. 'find_library' only finds a
+        // library, shared or static, without any settings from the external
+        // library.

-        set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+        // The 'lib/tls/CMakeLists.txt' depends on the mbedtls libraries, so we set the scope to 'GLOBAL' here.
+        find_package(MbedTLS GLOBAL)
+        set(LWS_MBEDTLS_LIBRARIES MbedTLS::mbedcrypto MbedTLS::mbedx509 MbedTLS::mbedtls)
```

```diff
diff --git a/third_party/libwebsockets/BUILD.gn b/third_party/libwebsockets/BUILD.gn
index 4c89c2e2e..e6d641b9c 100644
--- a/third_party/libwebsockets/BUILD.gn
+++ b/third_party/libwebsockets/BUILD.gn
@@ -84,6 +84,10 @@ cmake_project("websockets") {
   }

   options += [
+  // The libwebsockets will use the 'MBEDTLS_BUILD_PATH' variable to set the
+  // search path of 'find_package'.
+  "MBEDTLS_BUILD_PATH=" + rebase_path("$root_gen_dir/cmake/mbedtls"),
+
```

并删除 `third_party/libwebsockets/cmake/lws_config.h.in` 中的 `#define USING_SHARED_MBEDTLS`。

## curl

版本：8.1.2

[类 MIT 许可证](https://github.com/curl/curl/blob/master/COPYING)

一种使用 URL 语法传输数据的命令行工具和库，支持 DICT、FILE、FTP、FTPS、GOPHER、GOPHERS、HTTP、HTTPS、IMAP、IMAPS、LDAP、LDAPS、MQTT、POP3、POP3S、RTMP、RTMPS、RTSP、SCP、SFTP、SMB、SMBS、SMTP、SMTPS、TELNET 和 TFTP。libcurl 提供了无数强大的功能。

有关详细信息，请参阅 `third_party/curl`。

### curl 的补丁

修补 `lib/CMakeLists.txt`，将共享库名称从 `_imp.lib` 更改为 `.lib`。

```cmake
if(WIN32)
  if(BUILD_SHARED_LIBS)
    if(MSVC)
      // Add "_imp" as a suffix before the extension to avoid conflicting with
      // the statically linked "libcurl.lib"
      //set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX "_imp.lib")
      set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX ".lib")
    endif()
  endif()
endif()
```

在 `lib/ws.h` 中导出 `Curl_ws_done`，因为需要调用此函数以防止内存泄漏。

```c
CURL_EXTERN void Curl_ws_done(struct Curl_easy *data);
// ^^^^^^^^
```

## clingo

版本：5.8.0

[MIT 许可证](https://github.com/potassco/clingo/blob/master/LICENSE.md)

一种用于逻辑程序的接地器和求解器。

有关详细信息，请参阅 `third_party/clingo`。

ten\_manager 实际上使用 clingo-sys crate 来调用 clingo，而 clingo-sys 包含了自己的 clingo 源代码。然而，这个包含的 clingo 源代码是 5.6.2 版本，在 cmake 4 环境中存在编译错误。因此，我们需要使用 cargo 的 patch 功能来使用最新的本地版本 clingo 源代码（已修复了 cmake 4 编译错误）。此外，由于 clingo-sys 已经包含了 clingo 源代码，我们实际上并不需要 third\_party/clingo 文件夹。但是，由于 clingo-sys 在 Windows 上仍然将 clingo 编译为共享库，我们目前仍然需要使用 third\_party/clingo 文件夹在 Windows 上编译 clingo 的共享库，然后使用 gn 脚本将 Windows 共享库复制到 ten\_manager 的最终输出文件夹。未来，我们应该直接使用从 clingo-sys 中包含的 clingo 源代码编译的 Windows 共享库进行复制，这样就可以移除 third\_party/clingo 文件夹。

## FFmpeg

版本：6.0

[GPL 或 LGPL 许可证](https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md)

FFmpeg 代码库主要采用 LGPL 许可，可选组件采用 GPL 许可。有关详细信息，请参阅 LICENSE 文件。

用于 ffmpeg 扩展，主要用于测试目的。有关详细信息，请参阅 `third_party/ffmpeg`。

## libbacktrace

版本：1.0

[BSD 许可证](https://github.com/ianlancetaylor/libbacktrace/blob/master/LICENSE)

一个可以链接到 C/C++ 程序中以生成符号回溯的 C 库。

> ⚠️ **注意：**
> 我们已经对 `libbacktrace` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/backtrace`。

## uthash

版本：2.3.0

[BSD 许可证](https://github.com/troydhanson/uthash/blob/master/LICENSE)

用于哈希表等的 C 宏。

> ⚠️ **注意：**
> 我们已经对 `uthash` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅文件头中提到 `uthash` 的 `core/include/ten_utils/container` 下的文件。

## uuid4

[MIT 许可证](https://github.com/gpakosz/uuid4/blob/master/LICENSE.MIT)
[WTFPLv2 许可证](https://github.com/gpakosz/uuid4/blob/master/LICENSE.WTFPLv2)

C 中的 UUID v4 生成。

> ⚠️ **注意：**
> 我们已经对 `uuid4` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/lib/sys/general/uuid.c`。

## zf\_log

[MIT 许可证](https://github.com/wonder-mice/zf_log/blob/master/LICENSE)

用于 C/ObjC/C++ 的核心日志库。

> ⚠️ **注意：**
> 我们已经对 `zf_log` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/log`。


# 消息系统 (/cn/docs/ten_framework/0.11.59/message_system)

## 消息类型

在 TEN framework 中，消息被分为四种类型：

1. 命令 (Command)
2. 数据 (Data)
3. 视频帧 (Video Frame)
4. 音频帧 (Audio Frame)

命令和非命令消息类型之间的主要区别在于，命令会产生结果，而非命令则不会。

与这些类型相对应，扩展有四个消息回调函数：

1. `OnCmd`
2. `OnData`
3. `OnVideoFrame`
4. `OnAudioFrame`

以下是 TEN framework 消息的分类图：

```text
┌── 有结果
│   └── 命令 (Command)
│       ├── TEN framework 内置命令
│       │    => 消息名称以 `ten:` 开头
│       └── 非 TEN framework 内置命令
│            => 消息名称不以 `ten:` 开头
└── 无结果
    ├── 数据 (Data)
    │   ├── TEN framework 内置数据
    │   │    => 消息名称以 `ten:` 开头
    │   └── 非 TEN framework 内置数据
    │        => 消息名称不以 `ten:` 开头
    ├── 视频帧 (Video Frame)
    │   ├── TEN framework 内置视频帧
    │   │    => 消息名称以 `ten:` 开头
    │   └── 非 TEN framework 内置视频帧
    │        => 消息名称不以 `ten:` 开头
    └── 音频帧 (Audio Frame)
        ├── TEN framework 内置音频帧
        │    => 消息名称以 `ten:` 开头
        └── 非 TEN framework 内置音频帧
             => 消息名称不以 `ten:` 开头
```

## 消息名称

在 TEN framework 中，消息名称用于区分相同类型但用途不同的消息。扩展使用不同的消息名称来决定它们需要执行的操作。

消息名称的命名规则如下：

1. 第一个字符必须是字母 (`a-z`、`A-Z`) 或下划线 (`_`)。
2. 后续字符可以是字母 (`a-z`、`A-Z`)、数字 (`0-9`) 或下划线 (`_`)。

## 消息流

![消息流](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_flow.png)

TEN framework 支持三种消息流模式：

1. **单消息，单结果：**

   适用于一个动作产生单个结果的情况。

2. **单消息，多结果：**

   适用于一个动作产生多个结果的情况。

3. **多消息：**

   适用于不需要结果的动作。

## 扩展的消息接口

扩展的 `manifest.json` 文件中的 `api` 字段定义了该扩展的外部消息接口。`api` 字段可以包含以下类型的 API 信息：

* `cmd_in`
* `cmd_out`
* `data_in`
* `data_out`
* `audio_frame_in`
* `audio_frame_out`
* `video_frame_in`
* `video_frame_out`

以下是一个 API 定义的示例：

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

TEN framework 中不同扩展之间的消息配对机制类似于传统编程语言中的函数调用。以下是一个简单的类比：

1. 消息名称类似于函数名称，扩展消息之间的配对机制类似于函数调用机制。
2. 消息属性类似于函数参数。
3. 当 TEN framework 确定扩展 A 的输出消息 `foo` 与扩展 B 的输入消息 `foo` 配对时，这类似于扩展 A 中的函数指针指向扩展 B 中的函数 `foo`。
4. 配对仅基于消息名称，而不是消息属性。这意味着 TEN framework 不支持类似于函数重载的机制。

因此，消息名称对于每个扩展的输入/输出消息至关重要，充当该扩展的控制平面接口。

尽管 TEN framework 在单消息级别提供了与语言无关的调用机制，但它还允许通过 TEN 模式对扩展之间的调用关系进行静态和动态检查。这有助于检测潜在问题，例如源扩展和目标扩展之间对属性类型的不同解释。用户可以定义自己的消息名称，并通过图确定扩展之间的调用关系。TEN framework 还提供了静态检查工具，以确保这些关系不会导致任何问题。

## TEN framework 中的消息所有权概念

在 TEN framework 中，一旦消息成功发送，发送消息的扩展将无法再使用它。此概念对于维护框架内的消息安全和线程安全至关重要。

![消息处理遵循所有权概念](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_ownership.png)

消息可以沿两个方向发送：**发送 (send)** 和 **返回 (return)**。只有命令消息具有返回方向；数据、音频帧和视频帧消息则没有。消息发送后，可能会经过各种处理步骤，允许扩展继续使用该消息可能会导致线程安全问题。例如，处理消息的另一个扩展可能在不同的线程中运行。即使初始扩展继续使用该消息没有任何明显问题，此行为也是未定义的，开发人员不应依赖它。

TEN runtime 实施了各种安全措施，以确保扩展在消息成功发送后无法使用该消息。

### 消息所有权转移至扩展

当 TEN framework 通过 `OnCmd`、`OnData`、`OnVideoFrame` 或 `OnAudioFrame` 回调函数将消息传递给扩展时，框架会将消息的所有权转移给该扩展。这种所有权的转移意味着 TEN framework 保证只有接收扩展才能访问该消息，从而确保消息安全和线程安全。该扩展可以自由使用该消息，并且在不再需要该消息时必须将所有权返回给 TEN framework。返回所有权后，该扩展不得再与该消息进行任何交互。

### 消息所有权转移至 TEN runtime

当扩展使用 `SendCmd`、`ReturnResult`、`SendData`、`SendVideoFrame` 或 `SendAudioFrame` 等 API 将消息发送回 TEN runtime 时，该扩展会将消息的所有权返回给 TEN runtime。

### 与消息相关的数据的所有权

所有权的概念不仅限于消息本身，还延伸到与该消息关联的所有资源。与消息一样，每个资源的所有权一次只能属于一个扩展，并且它反映了消息的所有权。这种设计确保了 TEN framework 中的消息和线程安全。

例如，如果扩展 A 拥有一个消息并使用该消息中的缓冲区，则它必须在不再需要它们的时候将消息和缓冲区的所有权都返回给 TEN runtime。

### 默认复制语义

为了防止多个扩展同时处理同一资源而导致数据竞争等问题，TEN framework 在跨 TEN runtime 和扩展边界传输数据时默认使用**复制语义**。复制确保了所有权的分离，从而维护安全性。

### 借用语义

TEN framework 还支持**借用语义**，允许扩展使用资源而无需取得所有权。这种方法在保持消息和线程安全的同时，提高了资源的利用率。借用 API 必须与 GiveBack API 结合使用，以确保资源的安全性和完整性。

当扩展将消息的所有权返回给 TEN runtime 时，如果所有关联的资源也已返回，则运行平台才会接受该消息。例如，如果扩展 A 借用了数据消息的缓冲区，但在返回消息本身之前没有返回缓冲区的所 有权，则 TEN runtime 将拒绝该消息（例如，`SendData` 将会失败）。

### 借用 API

扩展可以使用借用 API 直接访问消息中的资源，从而实现临时使用而无需转移所有权。

### GiveBack API

GiveBack API 用于将资源的所有权返回给 TEN runtime，确保资源得到安全管理并在不再需要时被释放。

## 命令

TEN 客户端和 TEN 扩展可以向其他 TEN 扩展发送命令。

如果命令用于不需要结果的场景，则没有问题。TEN runtime 将自动丢弃任何不必要的结果。

### 结果

结果是对命令的响应。例如，当 TEN 扩展 A 向 TEN 扩展 B 发送命令 X 时，在处理完该命令后，TEN 扩展 B 会将结果发送回 TEN 扩展 A。结果可以包含任何有意义的细节，使 TEN 扩展 A 能够获得有关执行结果的更多信息。

一个典型的结果的 JSON 表示形式可能如下所示：

```json
{
  "ten": {
    "status_code": 400
  }
}
```


# 准备工作 (/cn/docs/ten_framework/0.11.59/preparation)

TEN framework 支持 Windows、Linux 和 Mac 作为开发环境，但建议使用 Linux 或 Mac。

## 开发环境

### Linux

以 Ubuntu 为例，您必须首先安装以下软件包：

* gcc：TEN framework 支持的 C/C++ 编译器之一。
* clang：TEN framework 支持的另一种 C/C++ 编译器。
* clang-format：TEN framework 在运行时使用的代码格式化工具。
* clang-tidy：静态代码分析器。
* cmake：构建系统。版本必须大于 3.13。
* Python3：TEN framework Python 绑定所需。
* pytest：用于 TEN framework 的集成测试。

### Mac

要设置您的 Mac 开发环境，您需要以下内容：

* XCode
* CocoaPods
* 个人账户

按照以下说明准备环境：

```shell
brew install llvm googletest doxygen ninja clang-format
brew install include-what-you-use
```

由于通过 `brew` 安装的 `cmake` 版本可能已过时，那么您需要从官方网站安装 `cmake` 并设置符号链接。

```shell
ln -s /Applications/CMake.app/Contents/bin/cmake /usr/local/bin/cmake
```

如果 `cmake` 找不到 `clang-tidy`，您可以使用以下命令解决此问题：

```shell
ln -sf /usr/local/opt/llvm/bin/clang-tidy /usr/local/bin/clang-tidy
```

要构建 TEN framework 的各种语言绑定，您可以使用以下命令安装所需的语言：

```shell
brew install python golang
```

### Windows

如果您需要在 Microsoft Windows 上开发 TEN framework，您可以手动安装以下软件包：

* Visual Studio 社区版（最高至 2022）

  请务必选择与 clang 相关的工具并将 clang 二进制文件路径添加到 `PATH` 环境变量。

* cmake

* python

> ⚠️ **注意：**
> 在 Windows 上安装 `cmake` / `python` 时，安装路径不应包含空格。

## ten\_gn

TEN framework 使用 `ten_gn` 作为其构建系统。`ten_gn` 是一个基于 Google GN 的构建系统。`ten_gn` 的源代码位于 TEN framework 仓库的 `core/ten_gn` 目录中。请将 `ten_gn` 目录路径添加到您开发环境的系统 `PATH` 环境变量中。

## 使用 Docker 容器

我们提供了预先编写的 Docker 文件，允许您创建一个容器，其中包含从源代码构建 TEN framework 所需的所有必要软件包。

### Ubuntu 22.04

导航到 `tools/docker_for_building/ubuntu/22.04` 并运行以下命令来创建并进入构建环境。

```shell
docker-compose up -d
docker-compose run ten-building-ubuntu-2204
```


# 性能分析 (/cn/docs/ten_framework/0.11.59/profiling)

## TEN framework GO 项目

使用 `gperftools` 来分析您的程序的 CPU 使用率和堆内存使用率。以下步骤介绍了如何准备环境、使用 `gperftools` 来分析您的程序，以及分析性能数据。

### 安装 `gperftools`

要在 Ubuntu 上安装 `gperftools`，请运行以下命令：

```shell
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

### 使用 `gperftools` 分析 CPU 使用率

要分析您的程序的 CPU 使用率，请运行以下命令：

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/.../cpu

exec /.../program
```

程序退出时，将生成一个 CPU 性能数据文件。数据将存储在 `/.../cpu.0001.prof` 文件中。

如果您想定期生成 CPU 性能数据，可以设置 `CPUPROFILE_FREQUENCY` 环境变量。例如，要每 30 秒生成一次 CPU 性能数据，请运行以下命令：

```shell
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/.../cpu CPUPROFILE_FREQUENCY=30 exec /.../program
```

#### 分析 CPU 性能数据

要分析 CPU 性能数据，请运行以下命令：

```shell
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### 使用 `gperftools` 分析堆内存使用率

要分析您的程序的堆内存使用率，请运行以下命令：

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

建议使用 `HEAP_PROFILE_TIME_INTERVAL` 环境变量来定期生成堆内存性能数据。所有堆内存性能数据将存储在 `HEAPPROFILE` 目录中。

#### 分析堆内存性能数据

每个生成的堆内存性能数据文件都将具有 `.heap` 扩展名。要分析堆内存性能数据，您可以将 `.heap` 文件转换为人类可读的文本格式。例如，要将 `/.../heap` 中的 `.heap` 文件转换为文本格式，请运行以下命令：

```shell
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

性能数据将如下所示：

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

您还可以将 `.heap` 文件转换为其他格式，例如 PDF 或 SVG。例如，要将 `/.../heap` 中的 `.heap` 文件转换为 PDF 格式，请运行以下命令：

```shell
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

PDF 格式的性能数据将与文本输出类似，但以图形方式可视化。

此外，您可以比较两组堆内存性能数据。例如，要比较 `/.../heap.0001.heap` 和 `/.../heap.0002.heap` 中的堆内存性能数据，请运行以下命令：

```shell
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

PDF 将显示两组数据之间堆内存使用率的差异。

如果您定期生成堆内存性能数据，我们提供了一个脚本来分析目录中的所有堆内存性能数据，并生成一个 Excel 文件，显示堆内存使用率趋势。例如，要分析 `/.../heap` 中的所有堆内存性能数据，请运行以下命令：

```shell
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

该脚本将在 `/.../raw` 目录中生成带有符号的原始性能数据，在 `/.../text` 目录中生成人类可读的文本性能数据，并在 `/.../heap.xlsx` 中生成 Excel 文件。

您可以绘制折线图以显示堆内存使用率趋势。

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### 使用 `pprof` 进行分析

我们建议使用 `pprof` 来分析用 Go 编写的程序。分析 Go 程序的最简单方法是将 `pprof_app_go` 用作您的 TEN 应用程序。激活后，应用程序会读取环境变量，以确定是否激活 `pprof` 服务器和堆分析器。

```shell
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### 分析 `pprof` 性能数据

要将堆性能数据转换为文本格式，请使用以下命令：

```shell
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

要分析所有堆性能数据并生成 Excel 文件，请运行：

```shell
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

您可以绘制折线图以显示堆内存使用率趋势：

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

这将为您提供 Go 堆性能数据趋势的可视化表示。


# 版本系统 (/cn/docs/ten_framework/0.11.59/version_system)

TEN framework 遵循 semver 规范。

## TEN 包的版本信息

TEN 包的版本信息记录在其自身的 `manifest.json` 文件的 `version` 字段中。对于每个 TEN 包，此 `version` 字段都是强制性的。

```json
{
  "version": "1.0.0"
}
```

## 依赖项的版本要求

> ⚠️ **注意：**
> 如果未指定依赖项的版本要求，则默认使用最新版本。

```text
└── ten_packages/
    ├── extension/
    ├── extension_group/
    ├── protocol/
    └── system/
```

这些包通过其 `manifest.json` 文件中 `type` 字段的值来区分。

```json
{
  "type": "system" // "extension", "extension_group", "protocol"
}
```

在 TEN 包中，依赖的 TEN 包的版本要求指定如下。

```json
{
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.1"
    },
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "1.0.3"
    },
    {
      "type": "extension",
      "name": "default_http_extension_cpp",
      "version": "1.0.0"
    },
    {
      "type": "extension_group",
      "name": "default_extension_group",
      "version": "1.0.0"
    }
  ]
}
```

## 版本要求规范

版本要求字符串可以包含多个版本要求字符串，以逗号分隔。每个版本要求字符串支持以下运算符：

* `>`、`>=`、`=`

  手动指定版本范围或确切的依赖项版本。

* `~`

  用于指定具有一定更新灵活性的最低版本。如果指定了 major、minor 和 patch 版本，或者仅指定了 major 和 minor 版本，则仅允许 patch 级别的更改。如果仅指定了 major 版本，则允许 minor 和 patch 级别的更改。

* `^`

  可以省略 `^` 运算符，这意味着如果版本要求字符串不包含任何运算符，则默认为 `^`。例如，字符串 `0.1.12` 表示 `^0.1.12`。虽然 `0.1.12` 看起来像一个特定版本，但它实际上指定了一个允许与 SemVer 兼容的更新的版本要求。

  只要不修改 major、minor 或 patch 组合中最左边的非零数字，就允许更新。在这种情况下，如果您升级 TEN 包，它将更新到 `0.1.13` 版本（如果是最新的 `0.1.z` 版本），但不会更新到 `0.2.0`。如果指定的版本字符串为 `1.0`，它将更新到 `1.1`（如果是最新的 `1.y` 版本），但不会更新到 `2.0`。

  > ⚠️ **注意：**
  > 版本 `0.0.x` 被认为与任何其他版本都不兼容。


# 构建系统 (/cn/docs/ten_framework/0.11.60/build_system)

## 构建系统

### C++

TEN framework 使用 `ten_gn` 构建系统来编译 C++ 应用程序和 C++ 扩展。 `ten_gn` 是一个基于 Google GN 的构建系统。

### Golang

TEN framework 使用标准的 go 命令来构建 TEN framework Golang 项目。

## 构建

### C++ 应用程序

要构建 TEN framework C++ 应用程序，请在 TEN framework C++ 应用程序的根目录中运行以下命令。

`<os>` 可以是以下之一：

* linux
* mac
* win

`<arch>` 可以是以下之一：

* x86
* x64
* arm
* arm64

`<build_type>` 可以是以下之一：

* debug
* release

```shell
ten_gn gen <os> <arch> <build_type>
ten_gn build <os> <arch> <build_type>
```

### C++ 插件

此处的术语 `addon` 包括以下内容：

* 扩展
* 扩展组
* 协议

构建 TEN framework C++ 插件的方法与构建 TEN framework C++ 应用程序的方法相同。

### Golang 应用程序

要构建 TEN Golang 项目，请在 TEN Golang 项目的根目录中运行以下命令：

```shell
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```


# 云商店 (/cn/docs/ten_framework/0.11.60/cloud_store)

TEN 云商店类似于 Android 上的 Google Play 和 iOS 上的 App Store，提供各种 TEN 软件包。 可以将这些软件包下载到 TEN 应用程序中，以帮助和增强开发。

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

## 数据库中存储的 TEN 软件包的信息

1. **软件包类型**

   表示软件包类型的字符串，例如 `extension`。

2. **软件包名称**

   表示软件包名称的字符串，例如 `builtin_extension`。

3. **软件包版本**

   遵循语义版本控制 (semver) 的字符串，用于指定软件包版本，例如 `3.0.0`。

4. **依赖项**

   对象数组，每个对象包含以下字段：

   * **软件包类型**

     字符串，与顶级的 `package type` 匹配。

   * **软件包名称**

     字符串，与顶级的 `package name` 匹配。

   * **版本**

     遵循 semver 的字符串，用于指定软件包的版本要求，例如 `~3.0.0`。


# 概述 (/cn/docs/ten_framework/0.11.60/concept_overview)

![TEN framework](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_framework.png)

## TEN runtime （TEN Runtime）

TEN runtime 是专为 TEN framework 设计的应用程序运行平台。它提供了一种灵活的架构，可以无缝集成用多种编程语言开发的扩展，并处理各种类型的数据流。通过对扩展的生命周期、数据流向和线程进行统一管理，TEN runtime 使开发者能够高效构建多样化的应用和服务。

## 应用（App）

TEN framework 中的应用可以作为独立的进程运行，也可以作为现有进程中的线程运行。 这种灵活性允许根据应用程序的需求选择不同的部署选项。

## 扩展组（Extension Group）

扩展组是一种为执行指定特定线程的机制。 给定组中的所有扩展都在同一线程上执行，从而确保一致和同步的处理。

## 扩展（Extension）

扩展是 TEN framework 的基本构建块。 开发人员可以使用各种编程语言创建扩展，并将它们组合起来以构建不同的应用程序和场景。 该框架的设计支持跨语言协作，使得不同语言编写的扩展能够在同一应用程序或服务中无缝地协同工作。

例如，由于 C++ 在处理音频和视频数据方面具有性能优势，因此开发人员可能会使用 C++ 创建用于实时通信 (RTC) 的扩展，同时使用 Python 开发 AI 扩展，以利用其强大的库进行数据分析和机器学习。 然后，可以将这些扩展集成到单个应用程序中，从而利用每种语言的优势。

## 图（Graph）

在 TEN framework 中，“图” (Graph) 描绘了扩展之间的数据流转过程。它编排了数据在不同扩展之间的流动方式，明确了数据流的参与者以及它们之间的交互关系。例如，可以将语音转文本（STT）扩展的输出定向到大语言模型（LLM）扩展进行进一步处理。

TEN framework 支持扩展之间的四种主要类型的数据流：

* 命令
* 数据
* 视频帧
* 音频帧

通过在图中定义这些数据流，开发者可以创建扩展之间的通信，以及单向的数据传输通道，这在处理音频和视频数据时尤其有用。

## 应用、组和扩展之间的关系

* **应用**：

  一个应用可以执行多个图，这些图可以是静态预定义的，也可以是动态组装的。

* **图**：

  图由多个协同工作的扩展构成，旨在创建一个有意义的场景。每个图的实例在应用程序内作为一个独立的会话运行。

* **扩展组**：

  扩展组的概念类似于线程。 以相同语言编写且位于同一扩展组中的扩展在运行时在同一线程上运行。 开发人员不需要直接管理线程； 他们只需"声明"每个扩展所属的组即可。

* **扩展**：

  框架中的每个扩展都分配有一个唯一的 ID，其结构为：`app-uri/graph-name/group-name/extension-name`

![概念的层次关系](https://ten-framework-assets.s3.amazonaws.com/doc-assets/hierarchical_relationship_of_concepts.png)

## TEN 云商店

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_cloud_store.png)

TEN 云商店的功能类似于 Google Play 商店或 Apple 的 App Store，为扩展提供了一个市场。 开发人员可以共享他们的扩展或下载其他人创建的扩展。 这些扩展可以集成到 TEN 应用程序中，从而促进开发并扩展功能。

## TEN 管理器

TEN 管理器是一种简化扩展管理的工具。 它处理诸如上传、共享和安装扩展之类的任务，自动管理它们及其环境之间的依赖关系。 这使得扩展的安装和发布方便高效，从而简化了在 TEN framework 内的开发过程。


# 依赖项 (/cn/docs/ten_framework/0.11.60/dependencies)

TEN framework 利用了多个第三方库。有些专门用于测试，而另一些则集成到 TEN runtime 中。以下是对这些库的描述，以及在 TEN framework 中使用它们所需的任何必要修改。

## Google gn

实例 ID

| 操作系统  | 架构    | 实例 ID                                         |
| ----- | ----- | --------------------------------------------- |
| Linux | x64   | BzX0zkfwFeUn9MaDVqm6FugmTIy-hFpgNUx43O1fN00C  |
| Linux | arm64 | rT\_12w1Iv6ug8CJ4j0VQekA0qTDq6CwoAqGWasIKFcEC |
| Win   | x64   | 1QlqF0FPVt82ba5f48HxHpv5xPqOmyaThoR3TicuJ8QC  |

直接从 [Google GN 网页](https://chrome-infra-packages.appspot.com/p/gn/gn/) 下载。

Commit ID

| 操作系统 | 架构        | Commit ID                                |
| ---- | --------- | ---------------------------------------- |
| Mac  | universal | 18602f6cf1168cf78302024043edc02e8bad2ffb |

从[Gn仓库](https://gn.googlesource.com/gn)clone编译。

## Google ninja

版本：1.12.1

直接从 [Ninja 发布页面](https://github.com/ninja-build/ninja/releases) 下载。

## yyjson

版本：0.10.0

[MIT 许可证](https://github.com/ibireme/yyjson/blob/master/LICENSE)

这在 TEN framework 核心中用于解析和生成 JSON 数据。有关详细信息，请参阅 `third_party/yyjson`。

## libuv

版本：1.50.0

[MIT 许可证](https://github.com/libuv/libuv#licensing)

这是 TEN 运行时中使用的一个事件循环库。有关详细信息，请参阅 `third_party/libuv`。

## msgpack-c

版本：6.1.0

[Boost 软件许可证，版本 1.0](https://github.com/msgpack/msgpack-c#license)

用于 C 的 MessagePack 库。有关详细信息，请参阅 `third_party/msgpack`。

## libwebsockets

版本：4.3.2

[MIT 许可证](https://github.com/warmcat/libwebsockets/blob/main/LICENSE)

规范 libwebsockets.org 网络库。有关详细信息，请参阅 `third_party/libwebsockets`。

```diff
--- /home/wei/MyData/Temp/libwebsockets-4.3.2/CMakeLists.txt
+++ libwebsockets/CMakeLists.txt
@@ -851,6 +851,7 @@
  add_definitions(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE)
  # Fail the build if any warnings
  add_compile_options(/W3 /WX)
+ add_compile_options(/Zc:preprocessor /wd4819)
  # Unbreak MSVC broken preprocessor __VA_ARGS__ behaviour
  if (MSVC_VERSION GREATER 1925)
   add_compile_options(/Zc:preprocessor /wd5105)
```

应用以下补丁。

```diff
diff --git a/third_party/libwebsockets/cmake/lws_config.h.in b/third_party/libwebsockets/cmake/lws_config.h.in
index f3f4a9d79..e8d36c3ae 100644
--- a/third_party/libwebsockets/cmake/lws_config.h.in
+++ b/third_party/libwebsockets/cmake/lws_config.h.in
@@ -238,3 +238,9 @@
 #cmakedefine LWS_WITH_PLUGINS_API
 #cmakedefine LWS_HAVE_RTA_PREF

+// NOTE (Liu): The libwebsockets library will use external variables from mbedtls
+// if 'LWS_WITH_MBEDTLS' is enabled. On Windows, variable declarations in the header
+// file must start with '__declspec(dllimport)', otherwise, the
+// symbols cannot be accessed.
+// Refer to third_party/mbedtls/include/mbedtls/export.h.
+#define USING_SHARED_MBEDTLS
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
index e8a0cb2d4..84a164e90 100755
--- a/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
+++ b/third_party/libwebsockets/lib/tls/mbedtls/wrapper/platform/ssl_pm.c
@@ -183,7 +183,12 @@ int ssl_pm_new(SSL *ssl)
         mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, version);
     } else {
         mbedtls_ssl_conf_max_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
-    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, 1);
+
+    // NOTE: mbedtls added 'ssl_conf_version_check()' since v3.1.0, and
+    // mbedtls only enables TLS 1.2 by default. So the 'min_tls_version' and
+    // 'max_tls_version' must be '0x303', or 'mbedtls_ssl_setup' will
+    // fail.
+    mbedtls_ssl_conf_min_version(&ssl_pm->conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);
     }

     mbedtls_ssl_conf_rng(&ssl_pm->conf, mbedtls_ctr_drbg_random, &ssl_pm->ctr_drbg);
```

```diff
diff --git a/third_party/libwebsockets/CMakeLists.txt b/third_party/libwebsockets/CMakeLists.txt
index 92638143a..746f9b6a6 100644
--- a/third_party/libwebsockets/CMakeLists.txt
+++ b/third_party/libwebsockets/CMakeLists.txt
@@ -547,9 +547,12 @@ SET(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/lib")

 SET(LWS_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}")

-# Put absolute path of dynamic libraries into the object code. Some
-# architectures, notably Mac OS X, need this.
-SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
+# Commented out to avoid using absolute paths in the install_name on macOS.
+# When CMAKE_INSTALL_NAME_DIR is set to an absolute path, binaries that link
+# against libwebsockets will hardcode this absolute path, making the library
+# difficult to relocate and potentially causing "library not found" errors at
+# runtime if the library is installed in a different location.
+# SET(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/${LWS_INSTALL_LIB_DIR}${LIB_SUFFIX}")
```

### 修复在 Windows 上链接 mbedtls

如果 CMake 版本高于 3.24，请应用以下补丁，因为 `find_package` 自 3.24 起支持 `GLOBAL` 关键字。

```diff
diff --git a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
index e34151724..79b15089d 100644
--- a/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
+++ b/third_party/libwebsockets/lib/tls/mbedtls/CMakeLists.txt
-        find_library(MBEDTLS_LIBRARY mbedtls)
-        find_library(MBEDX509_LIBRARY mbedx509)
-        find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+        # find_library(MBEDTLS_LIBRARY mbedtls)
+        # find_library(MBEDX509_LIBRARY mbedx509)
+        # find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+        # set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+
+        # Set the custom search path.
+        set(MBEDTLS_DIR "${MBEDTLS_BUILD_PATH}/cmake")
+
+        // NOTE (Liu): We should use 'find_package' rather than 'find_library'
+        // to search for the mbedtls libraries. 'find_library' only finds a
+        // library, shared or static, without any settings from the external
+        // library.

-        set(LWS_MBEDTLS_LIBRARIES "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" "${MBEDCRYPTO_LIBRARY}")
+        // The 'lib/tls/CMakeLists.txt' depends on the mbedtls libraries, so we set the scope to 'GLOBAL' here.
+        find_package(MbedTLS GLOBAL)
+        set(LWS_MBEDTLS_LIBRARIES MbedTLS::mbedcrypto MbedTLS::mbedx509 MbedTLS::mbedtls)
```

```diff
diff --git a/third_party/libwebsockets/BUILD.gn b/third_party/libwebsockets/BUILD.gn
index 4c89c2e2e..e6d641b9c 100644
--- a/third_party/libwebsockets/BUILD.gn
+++ b/third_party/libwebsockets/BUILD.gn
@@ -84,6 +84,10 @@ cmake_project("websockets") {
   }

   options += [
+  // The libwebsockets will use the 'MBEDTLS_BUILD_PATH' variable to set the
+  // search path of 'find_package'.
+  "MBEDTLS_BUILD_PATH=" + rebase_path("$root_gen_dir/cmake/mbedtls"),
+
```

并删除 `third_party/libwebsockets/cmake/lws_config.h.in` 中的 `#define USING_SHARED_MBEDTLS`。

## curl

版本：8.1.2

[类 MIT 许可证](https://github.com/curl/curl/blob/master/COPYING)

一种使用 URL 语法传输数据的命令行工具和库，支持 DICT、FILE、FTP、FTPS、GOPHER、GOPHERS、HTTP、HTTPS、IMAP、IMAPS、LDAP、LDAPS、MQTT、POP3、POP3S、RTMP、RTMPS、RTSP、SCP、SFTP、SMB、SMBS、SMTP、SMTPS、TELNET 和 TFTP。libcurl 提供了无数强大的功能。

有关详细信息，请参阅 `third_party/curl`。

### curl 的补丁

修补 `lib/CMakeLists.txt`，将共享库名称从 `_imp.lib` 更改为 `.lib`。

```cmake
if(WIN32)
  if(BUILD_SHARED_LIBS)
    if(MSVC)
      // Add "_imp" as a suffix before the extension to avoid conflicting with
      // the statically linked "libcurl.lib"
      //set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX "_imp.lib")
      set_target_properties(${LIB_NAME} PROPERTIES IMPORT_SUFFIX ".lib")
    endif()
  endif()
endif()
```

在 `lib/ws.h` 中导出 `Curl_ws_done`，因为需要调用此函数以防止内存泄漏。

```c
CURL_EXTERN void Curl_ws_done(struct Curl_easy *data);
// ^^^^^^^^
```

## clingo

版本：5.8.0

[MIT 许可证](https://github.com/potassco/clingo/blob/master/LICENSE.md)

一种用于逻辑程序的接地器和求解器。

有关详细信息，请参阅 `third_party/clingo`。

ten\_manager 实际上使用 clingo-sys crate 来调用 clingo，而 clingo-sys 包含了自己的 clingo 源代码。然而，这个包含的 clingo 源代码是 5.6.2 版本，在 cmake 4 环境中存在编译错误。因此，我们需要使用 cargo 的 patch 功能来使用最新的本地版本 clingo 源代码（已修复了 cmake 4 编译错误）。此外，由于 clingo-sys 已经包含了 clingo 源代码，我们实际上并不需要 third\_party/clingo 文件夹。但是，由于 clingo-sys 在 Windows 上仍然将 clingo 编译为共享库，我们目前仍然需要使用 third\_party/clingo 文件夹在 Windows 上编译 clingo 的共享库，然后使用 gn 脚本将 Windows 共享库复制到 ten\_manager 的最终输出文件夹。未来，我们应该直接使用从 clingo-sys 中包含的 clingo 源代码编译的 Windows 共享库进行复制，这样就可以移除 third\_party/clingo 文件夹。

## FFmpeg

版本：6.0

[GPL 或 LGPL 许可证](https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md)

FFmpeg 代码库主要采用 LGPL 许可，可选组件采用 GPL 许可。有关详细信息，请参阅 LICENSE 文件。

用于 ffmpeg 扩展，主要用于测试目的。有关详细信息，请参阅 `third_party/ffmpeg`。

## libbacktrace

版本：1.0

[BSD 许可证](https://github.com/ianlancetaylor/libbacktrace/blob/master/LICENSE)

一个可以链接到 C/C++ 程序中以生成符号回溯的 C 库。

> ⚠️ **注意：**
> 我们已经对 `libbacktrace` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/backtrace`。

## uthash

版本：2.3.0

[BSD 许可证](https://github.com/troydhanson/uthash/blob/master/LICENSE)

用于哈希表等的 C 宏。

> ⚠️ **注意：**
> 我们已经对 `uthash` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅文件头中提到 `uthash` 的 `core/include/ten_utils/container` 下的文件。

## uuid4

[MIT 许可证](https://github.com/gpakosz/uuid4/blob/master/LICENSE.MIT)
[WTFPLv2 许可证](https://github.com/gpakosz/uuid4/blob/master/LICENSE.WTFPLv2)

C 中的 UUID v4 生成。

> ⚠️ **注意：**
> 我们已经对 `uuid4` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/lib/sys/general/uuid.c`。

## zf\_log

[MIT 许可证](https://github.com/wonder-mice/zf_log/blob/master/LICENSE)

用于 C/ObjC/C++ 的核心日志库。

> ⚠️ **注意：**
> 我们已经对 `zf_log` 进行了重大修改，以符合 TEN framework 的命名约定和文件夹结构。有关详细信息，请参阅 `core/src/ten_utils/log`。


# 消息系统 (/cn/docs/ten_framework/0.11.60/message_system)

## 消息类型

在 TEN framework 中，消息被分为四种类型：

1. 命令 (Command)
2. 数据 (Data)
3. 视频帧 (Video Frame)
4. 音频帧 (Audio Frame)

命令和非命令消息类型之间的主要区别在于，命令会产生结果，而非命令则不会。

与这些类型相对应，扩展有四个消息回调函数：

1. `OnCmd`
2. `OnData`
3. `OnVideoFrame`
4. `OnAudioFrame`

以下是 TEN framework 消息的分类图：

```text
┌── 有结果
│   └── 命令 (Command)
│       ├── TEN framework 内置命令
│       │    => 消息名称以 `ten:` 开头
│       └── 非 TEN framework 内置命令
│            => 消息名称不以 `ten:` 开头
└── 无结果
    ├── 数据 (Data)
    │   ├── TEN framework 内置数据
    │   │    => 消息名称以 `ten:` 开头
    │   └── 非 TEN framework 内置数据
    │        => 消息名称不以 `ten:` 开头
    ├── 视频帧 (Video Frame)
    │   ├── TEN framework 内置视频帧
    │   │    => 消息名称以 `ten:` 开头
    │   └── 非 TEN framework 内置视频帧
    │        => 消息名称不以 `ten:` 开头
    └── 音频帧 (Audio Frame)
        ├── TEN framework 内置音频帧
        │    => 消息名称以 `ten:` 开头
        └── 非 TEN framework 内置音频帧
             => 消息名称不以 `ten:` 开头
```

## 消息名称

在 TEN framework 中，消息名称用于区分相同类型但用途不同的消息。扩展使用不同的消息名称来决定它们需要执行的操作。

消息名称的命名规则如下：

1. 第一个字符必须是字母 (`a-z`、`A-Z`) 或下划线 (`_`)。
2. 后续字符可以是字母 (`a-z`、`A-Z`)、数字 (`0-9`) 或下划线 (`_`)。

## 消息流

![消息流](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_flow.png)

TEN framework 支持三种消息流模式：

1. **单消息，单结果：**

   适用于一个动作产生单个结果的情况。

2. **单消息，多结果：**

   适用于一个动作产生多个结果的情况。

3. **多消息：**

   适用于不需要结果的动作。

## 扩展的消息接口

扩展的 `manifest.json` 文件中的 `api` 字段定义了该扩展的外部消息接口。`api` 字段可以包含以下类型的 API 信息：

* `cmd_in`
* `cmd_out`
* `data_in`
* `data_out`
* `audio_frame_in`
* `audio_frame_out`
* `video_frame_in`
* `video_frame_out`

以下是一个 API 定义的示例：

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

TEN framework 中不同扩展之间的消息配对机制类似于传统编程语言中的函数调用。以下是一个简单的类比：

1. 消息名称类似于函数名称，扩展消息之间的配对机制类似于函数调用机制。
2. 消息属性类似于函数参数。
3. 当 TEN framework 确定扩展 A 的输出消息 `foo` 与扩展 B 的输入消息 `foo` 配对时，这类似于扩展 A 中的函数指针指向扩展 B 中的函数 `foo`。
4. 配对仅基于消息名称，而不是消息属性。这意味着 TEN framework 不支持类似于函数重载的机制。

因此，消息名称对于每个扩展的输入/输出消息至关重要，充当该扩展的控制平面接口。

尽管 TEN framework 在单消息级别提供了与语言无关的调用机制，但它还允许通过 TEN 模式对扩展之间的调用关系进行静态和动态检查。这有助于检测潜在问题，例如源扩展和目标扩展之间对属性类型的不同解释。用户可以定义自己的消息名称，并通过图确定扩展之间的调用关系。TEN framework 还提供了静态检查工具，以确保这些关系不会导致任何问题。

## TEN framework 中的消息所有权概念

在 TEN framework 中，一旦消息成功发送，发送消息的扩展将无法再使用它。此概念对于维护框架内的消息安全和线程安全至关重要。

![消息处理遵循所有权概念](https://ten-framework-assets.s3.amazonaws.com/doc-assets/message_ownership.png)

消息可以沿两个方向发送：**发送 (send)** 和 **返回 (return)**。只有命令消息具有返回方向；数据、音频帧和视频帧消息则没有。消息发送后，可能会经过各种处理步骤，允许扩展继续使用该消息可能会导致线程安全问题。例如，处理消息的另一个扩展可能在不同的线程中运行。即使初始扩展继续使用该消息没有任何明显问题，此行为也是未定义的，开发人员不应依赖它。

TEN runtime 实施了各种安全措施，以确保扩展在消息成功发送后无法使用该消息。

### 消息所有权转移至扩展

当 TEN framework 通过 `OnCmd`、`OnData`、`OnVideoFrame` 或 `OnAudioFrame` 回调函数将消息传递给扩展时，框架会将消息的所有权转移给该扩展。这种所有权的转移意味着 TEN framework 保证只有接收扩展才能访问该消息，从而确保消息安全和线程安全。该扩展可以自由使用该消息，并且在不再需要该消息时必须将所有权返回给 TEN framework。返回所有权后，该扩展不得再与该消息进行任何交互。

### 消息所有权转移至 TEN runtime

当扩展使用 `SendCmd`、`ReturnResult`、`SendData`、`SendVideoFrame` 或 `SendAudioFrame` 等 API 将消息发送回 TEN runtime 时，该扩展会将消息的所有权返回给 TEN runtime。

### 与消息相关的数据的所有权

所有权的概念不仅限于消息本身，还延伸到与该消息关联的所有资源。与消息一样，每个资源的所有权一次只能属于一个扩展，并且它反映了消息的所有权。这种设计确保了 TEN framework 中的消息和线程安全。

例如，如果扩展 A 拥有一个消息并使用该消息中的缓冲区，则它必须在不再需要它们的时候将消息和缓冲区的所有权都返回给 TEN runtime。

### 默认复制语义

为了防止多个扩展同时处理同一资源而导致数据竞争等问题，TEN framework 在跨 TEN runtime 和扩展边界传输数据时默认使用**复制语义**。复制确保了所有权的分离，从而维护安全性。

### 借用语义

TEN framework 还支持**借用语义**，允许扩展使用资源而无需取得所有权。这种方法在保持消息和线程安全的同时，提高了资源的利用率。借用 API 必须与 GiveBack API 结合使用，以确保资源的安全性和完整性。

当扩展将消息的所有权返回给 TEN runtime 时，如果所有关联的资源也已返回，则运行平台才会接受该消息。例如，如果扩展 A 借用了数据消息的缓冲区，但在返回消息本身之前没有返回缓冲区的所 有权，则 TEN runtime 将拒绝该消息（例如，`SendData` 将会失败）。

### 借用 API

扩展可以使用借用 API 直接访问消息中的资源，从而实现临时使用而无需转移所有权。

### GiveBack API

GiveBack API 用于将资源的所有权返回给 TEN runtime，确保资源得到安全管理并在不再需要时被释放。

## 命令

TEN 客户端和 TEN 扩展可以向其他 TEN 扩展发送命令。

如果命令用于不需要结果的场景，则没有问题。TEN runtime 将自动丢弃任何不必要的结果。

### 结果

结果是对命令的响应。例如，当 TEN 扩展 A 向 TEN 扩展 B 发送命令 X 时，在处理完该命令后，TEN 扩展 B 会将结果发送回 TEN 扩展 A。结果可以包含任何有意义的细节，使 TEN 扩展 A 能够获得有关执行结果的更多信息。

一个典型的结果的 JSON 表示形式可能如下所示：

```json
{
  "ten": {
    "status_code": 400
  }
}
```


# 准备工作 (/cn/docs/ten_framework/0.11.60/preparation)

TEN framework 支持 Windows、Linux 和 Mac 作为开发环境，但建议使用 Linux 或 Mac。

## 开发环境

### Linux

以 Ubuntu 为例，您必须首先安装以下软件包：

* gcc：TEN framework 支持的 C/C++ 编译器之一。
* clang：TEN framework 支持的另一种 C/C++ 编译器。
* clang-format：TEN framework 在运行时使用的代码格式化工具。
* clang-tidy：静态代码分析器。
* cmake：构建系统。版本必须大于 3.13。
* Python3：TEN framework Python 绑定所需。
* pytest：用于 TEN framework 的集成测试。

### Mac

要设置您的 Mac 开发环境，您需要以下内容：

* XCode
* CocoaPods
* 个人账户

按照以下说明准备环境：

```shell
brew install llvm googletest doxygen ninja clang-format
brew install include-what-you-use
```

由于通过 `brew` 安装的 `cmake` 版本可能已过时，那么您需要从官方网站安装 `cmake` 并设置符号链接。

```shell
ln -s /Applications/CMake.app/Contents/bin/cmake /usr/local/bin/cmake
```

如果 `cmake` 找不到 `clang-tidy`，您可以使用以下命令解决此问题：

```shell
ln -sf /usr/local/opt/llvm/bin/clang-tidy /usr/local/bin/clang-tidy
```

要构建 TEN framework 的各种语言绑定，您可以使用以下命令安装所需的语言：

```shell
brew install python golang
```

### Windows

如果您需要在 Microsoft Windows 上开发 TEN framework，您可以手动安装以下软件包：

* Visual Studio 社区版（最高至 2022）

  请务必选择与 clang 相关的工具并将 clang 二进制文件路径添加到 `PATH` 环境变量。

* cmake

* python

> ⚠️ **注意：**
> 在 Windows 上安装 `cmake` / `python` 时，安装路径不应包含空格。

## ten\_gn

TEN framework 使用 `ten_gn` 作为其构建系统。`ten_gn` 是一个基于 Google GN 的构建系统。`ten_gn` 的源代码位于 TEN framework 仓库的 `core/ten_gn` 目录中。请将 `ten_gn` 目录路径添加到您开发环境的系统 `PATH` 环境变量中。

## 使用 Docker 容器

我们提供了预先编写的 Docker 文件，允许您创建一个容器，其中包含从源代码构建 TEN framework 所需的所有必要软件包。

### Ubuntu 22.04

导航到 `tools/docker_for_building/ubuntu/22.04` 并运行以下命令来创建并进入构建环境。

```shell
docker-compose up -d
docker-compose run ten-building-ubuntu-2204
```


# 性能分析 (/cn/docs/ten_framework/0.11.60/profiling)

## TEN framework GO 项目

使用 `gperftools` 来分析您的程序的 CPU 使用率和堆内存使用率。以下步骤介绍了如何准备环境、使用 `gperftools` 来分析您的程序，以及分析性能数据。

### 安装 `gperftools`

要在 Ubuntu 上安装 `gperftools`，请运行以下命令：

```shell
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

### 使用 `gperftools` 分析 CPU 使用率

要分析您的程序的 CPU 使用率，请运行以下命令：

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/.../cpu

exec /.../program
```

程序退出时，将生成一个 CPU 性能数据文件。数据将存储在 `/.../cpu.0001.prof` 文件中。

如果您想定期生成 CPU 性能数据，可以设置 `CPUPROFILE_FREQUENCY` 环境变量。例如，要每 30 秒生成一次 CPU 性能数据，请运行以下命令：

```shell
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/.../cpu CPUPROFILE_FREQUENCY=30 exec /.../program
```

#### 分析 CPU 性能数据

要分析 CPU 性能数据，请运行以下命令：

```shell
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### 使用 `gperftools` 分析堆内存使用率

要分析您的程序的堆内存使用率，请运行以下命令：

```shell
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

建议使用 `HEAP_PROFILE_TIME_INTERVAL` 环境变量来定期生成堆内存性能数据。所有堆内存性能数据将存储在 `HEAPPROFILE` 目录中。

#### 分析堆内存性能数据

每个生成的堆内存性能数据文件都将具有 `.heap` 扩展名。要分析堆内存性能数据，您可以将 `.heap` 文件转换为人类可读的文本格式。例如，要将 `/.../heap` 中的 `.heap` 文件转换为文本格式，请运行以下命令：

```shell
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

性能数据将如下所示：

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

您还可以将 `.heap` 文件转换为其他格式，例如 PDF 或 SVG。例如，要将 `/.../heap` 中的 `.heap` 文件转换为 PDF 格式，请运行以下命令：

```shell
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

PDF 格式的性能数据将与文本输出类似，但以图形方式可视化。

此外，您可以比较两组堆内存性能数据。例如，要比较 `/.../heap.0001.heap` 和 `/.../heap.0002.heap` 中的堆内存性能数据，请运行以下命令：

```shell
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

PDF 将显示两组数据之间堆内存使用率的差异。

如果您定期生成堆内存性能数据，我们提供了一个脚本来分析目录中的所有堆内存性能数据，并生成一个 Excel 文件，显示堆内存使用率趋势。例如，要分析 `/.../heap` 中的所有堆内存性能数据，请运行以下命令：

```shell
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

该脚本将在 `/.../raw` 目录中生成带有符号的原始性能数据，在 `/.../text` 目录中生成人类可读的文本性能数据，并在 `/.../heap.xlsx` 中生成 Excel 文件。

您可以绘制折线图以显示堆内存使用率趋势。

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### 使用 `pprof` 进行分析

我们建议使用 `pprof` 来分析用 Go 编写的程序。分析 Go 程序的最简单方法是将 `pprof_app_go` 用作您的 TEN 应用程序。激活后，应用程序会读取环境变量，以确定是否激活 `pprof` 服务器和堆分析器。

```shell
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### 分析 `pprof` 性能数据

要将堆性能数据转换为文本格式，请使用以下命令：

```shell
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

要分析所有堆性能数据并生成 Excel 文件，请运行：

```shell
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

您可以绘制折线图以显示堆内存使用率趋势：

```shell
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

这将为您提供 Go 堆性能数据趋势的可视化表示。


# 版本系统 (/cn/docs/ten_framework/0.11.60/version_system)

TEN framework 遵循 semver 规范。

## TEN 包的版本信息

TEN 包的版本信息记录在其自身的 `manifest.json` 文件的 `version` 字段中。对于每个 TEN 包，此 `version` 字段都是强制性的。

```json
{
  "version": "1.0.0"
}
```

## 依赖项的版本要求

> ⚠️ **注意：**
> 如果未指定依赖项的版本要求，则默认使用最新版本。

```text
└── ten_packages/
    ├── extension/
    ├── extension_group/
    ├── protocol/
    └── system/
```

这些包通过其 `manifest.json` 文件中 `type` 字段的值来区分。

```json
{
  "type": "system" // "extension", "extension_group", "protocol"
}
```

在 TEN 包中，依赖的 TEN 包的版本要求指定如下。

```json
{
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.1"
    },
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "1.0.3"
    },
    {
      "type": "extension",
      "name": "default_http_extension_cpp",
      "version": "1.0.0"
    },
    {
      "type": "extension_group",
      "name": "default_extension_group",
      "version": "1.0.0"
    }
  ]
}
```

## 版本要求规范

版本要求字符串可以包含多个版本要求字符串，以逗号分隔。每个版本要求字符串支持以下运算符：

* `>`、`>=`、`=`

  手动指定版本范围或确切的依赖项版本。

* `~`

  用于指定具有一定更新灵活性的最低版本。如果指定了 major、minor 和 patch 版本，或者仅指定了 major 和 minor 版本，则仅允许 patch 级别的更改。如果仅指定了 major 版本，则允许 minor 和 patch 级别的更改。

* `^`

  可以省略 `^` 运算符，这意味着如果版本要求字符串不包含任何运算符，则默认为 `^`。例如，字符串 `0.1.12` 表示 `^0.1.12`。虽然 `0.1.12` 看起来像一个特定版本，但它实际上指定了一个允许与 SemVer 兼容的更新的版本要求。

  只要不修改 major、minor 或 patch 组合中最左边的非零数字，就允许更新。在这种情况下，如果您升级 TEN 包，它将更新到 `0.1.13` 版本（如果是最新的 `0.1.z` 版本），但不会更新到 `0.2.0`。如果指定的版本字符串为 `1.0`，它将更新到 `1.1`（如果是最新的 `1.y` 版本），但不会更新到 `2.0`。

  > ⚠️ **注意：**
  > 版本 `0.0.x` 被认为与任何其他版本都不兼容。


# 事件 (/cn/docs/ten_agent_examples/api-reference/events)

import { Callout } from 'fumadocs-ui/components/callout';
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';

# TEN Agent 事件

事件是 RTC 服务器、MLLM 扩展和自定义代理扩展之间的主要通信机制。TEN Agent 使用**服务器→客户端 (S2C)** 模型处理传入事件，以及**客户端→服务器 (C2S)** 原始命令处理传出命令。

## S2C 事件（服务器到客户端）

这些是**从服务器发送**的事件，您的代理扩展接收并处理。

### 用户和会话事件

#### `UserJoinedEvent`

用户加入 RTC 会话时触发。

| 属性            | 类型       | 描述             |
| ------------- | -------- | -------------- |
| **user\_id**  | `string` | 加入用户的唯一标识符     |
| **timestamp** | `float`  | 事件时间戳（Unix 时代） |

**示例：**

```python
case UserJoinedEvent():
    self._rtc_user_count += 1
    await self._greeting_if_ready()
```

#### `UserLeftEvent`

用户离开 RTC 会话时触发。

| 属性            | 类型       | 描述         |
| ------------- | -------- | ---------- |
| **user\_id**  | `string` | 离开用户的唯一标识符 |
| **timestamp** | `float`  | 事件时间戳      |

**示例：**

```python
case UserLeftEvent():
    self._rtc_user_count -= 1
```

### 转录事件

#### `InputTranscriptEvent`

ASR 扩展识别的语音（用户语音转换为文本）。

| 属性             | 类型       | 描述                      |
| -------------- | -------- | ----------------------- |
| **text**       | `string` | 识别的语音文本                 |
| **final**      | `bool`   | 是否是最终（完整）转录             |
| **metadata**   | `dict`   | 其他上下文（session\_id、置信度等） |
| **stream\_id** | `string` | 此语音段的唯一标识符              |

**示例：**

```python
case InputTranscriptEvent():
    self.current_metadata = {"session_id": event.metadata.get("session_id", "100")}
    self.session_ready = True
    await self._greeting_if_ready()
```

<Callout type="info">
  **最终与流式**: `final` 标志表示转录是否完整或仍在识别中。非最终转录不应触发处理。
</Callout>

#### `OutputTranscriptEvent`

MLLM 生成的助手语音（文本转语音将其转换为音频）。

| 属性             | 类型       | 描述         |
| -------------- | -------- | ---------- |
| **text**       | `string` | 助手的语音文本    |
| **is\_final**  | `bool`   | 是否完成响应     |
| **stream\_id** | `string` | 此语音段的唯一标识符 |

**示例：**

```python
case OutputTranscriptEvent():
    await self._send_transcript("assistant", event.text, event.is_final, event.stream_id)
```

### 工具和函数事件

#### `ToolRegisterEvent`

工具/函数变为代理可调用。

| 属性           | 类型     | 描述       |
| ------------ | ------ | -------- |
| **tool**     | `str`  | 工具/函数标识符 |
| **source**   | `str`  | 注册此工具的扩展 |
| **metadata** | `dict` | 工具架构和参数  |

**示例：**

```python
case ToolRegisterEvent():
    await self.agent.register_tool(event.tool, event.source)
```

#### `FunctionCallEvent`

MLLM 请求调用工具/函数（函数调用实际执行）。

| 属性                 | 类型       | 描述            |
| ------------------ | -------- | ------------- |
| **call\_id**       | `string` | 此函数调用请求的唯一标识符 |
| **function\_name** | `string` | 要调用的函数名称      |
| **arguments**      | `dict`   | 函数参数作为键值对     |

**示例：**

```python
case FunctionCallEvent():
    await self.agent.call_tool(event.call_id, event.function_name, event.arguments)
```

<Callout type="warning">
  处理完 `FunctionCallEvent` 后，必须通过 C2S `SendFunctionCallOutput` 原始命令发送结果以继续对话。
</Callout>

### 中断事件

#### `ServerInterruptEvent`

MLLM 检测到用户语音开始并发送中断信号。

| 属性         | 类型       | 描述                                |
| ---------- | -------- | --------------------------------- |
| **reason** | `string` | 中断原因（例如 "user\_speech\_detected"） |

**示例：**

```python
case ServerInterruptEvent():
    await self._interrupt()
```

## C2S 原始命令（客户端到服务器）

这些是您的代理扩展**发送到 MLLM 服务器**的命令。

### 设置消息上下文

使用消息历史初始化或更新对话上下文。

```python
async def _set_context_messages(self, messages: list[MLLMClientMessageItem]):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_SET_MESSAGE_CONTEXT,
        "v2v",
        MLLMClientSetMessageContext(messages=messages).model_dump(),
    )
```

**参数：**

* **messages**: 消息项列表（系统、用户、助手）

### 发送消息项

向对话添加单条消息。

```python
async def _send_message_item(self, message: MLLMClientMessageItem):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_SEND_MESSAGE_ITEM,
        "v2v",
        MLLMClientSendMessageItem(item=message).model_dump(),
    )
```

**参数：**

* **message**: 单条消息（文本、图像或音频）

### 触发响应

请求 MLLM 生成响应（思考 + 说话）。

```python
async def _send_create_response(self):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_CREATE_RESPONSE,
        "v2v",
        MLLMClientCreateResponse().model_dump(),
    )
```

<Callout type="tip">
  在设置上下文或发送消息后调用此命令以触发 MLLM 思考和说话。
</Callout>

### 发送函数调用输出

将工具/函数调用的结果返回给 MLLM。

```python
async def _send_function_call_output(self, result: str, call_id: str):
    await _send_data(
        self.ten_env,
        DATA_MLLM_IN_FUNCTION_CALL_OUTPUT,
        "v2v",
        MLLMClientFunctionCallOutput(
            output=result,
            call_id=call_id,
        ).model_dump(),
    )
```

**参数：**

* **output**: 工具结果作为字符串
* **call\_id**: 原始 `FunctionCallEvent` 中的 call\_id

## 事件流程图

<Tabs items={["实时 V2V", "级联"]}>
  <Tab>
    ```
    用户语音
        ↓
    RTC → ASR 扩展
        ↓
    InputTranscriptEvent (S2C)
        ↓
    MainExtension.on_data()
        ↓
    _set_context_messages() (C2S)
        ↓
    _send_create_response() (C2S)
        ↓
    MLLM 处理并生成响应
        ↓
    OutputTranscriptEvent (S2C)
        ↓
    TTS 扩展转换为音频
        ↓
    音频流式传输给用户
    ```
  </Tab>

  <Tab>
    ```
    用户消息（文本/语音）
        ↓
    MainExtension 处理输入
        ↓
    发送到 MLLM (C2S)
        ↓
    MLLM 用文本响应
        ↓
    OutputTranscriptEvent (S2C)
        ↓
    TTS 转换并流式传输音频
        ↓
    用户听到响应
    ```
  </Tab>
</Tabs>

## 常见模式

### 就绪时问候

```python
async def _greeting_if_ready(self):
    if self._rtc_user_count == 1 and self.config.greeting and self.session_ready:
        await self._send_message_item(
            MLLMClientMessageItem(
                role="user",
                content=self.config.greeting,
            )
        )
        await self._send_create_response()
```

### 处理工具结果

```python
case FunctionCallEvent():
    try:
        result = await call_tool(event.function_name, event.arguments)
        await self._send_function_call_output(json.dumps(result), event.call_id)
    except Exception as e:
        await self._send_function_call_output(json.dumps({"error": str(e)}), event.call_id)
```

### 优雅关闭

```python
case UserLeftEvent():
    self._rtc_user_count -= 1
    if self._rtc_user_count == 0:
        await self._cleanup()
```

## 参考资源

有关完整的事件定义和类型，请参阅 TEN Framework 中的源代码：

* [TEN Agent 事件](https://github.com/TEN-framework/TEN-Agent/tree/main/ai_agents/agents/examples)
* [MLLM 扩展 API](https://github.com/TEN-framework/ten-framework)


# 配置模式 (/cn/docs/ten_agent_examples/api-reference/schemas)

import { Callout } from 'fumadocs-ui/components/callout';
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';

# 配置模式

TEN Agent 配置在两个关键文件中定义：`property.json`（扩展运行时属性）和 `manifest.json`（元数据和依赖项）。

## property.json

`property.json` 文件定义扩展的运行时可配置属性。

### 架构结构

```json
{
  "extension_group": "ten_agent_extensions",
  "api_version": "0.1",
  "dependencies": [],
  "nodes": [
    {
      "type": "extension",
      "name": "main_control",
      "addon": "main_control.extension:MainControlExtension",
      "property": {
        "greeting": "你好！我是你的 TEN 代理。",
        "max_history": 50
      }
    }
  ]
}
```

### 按扩展类型常见属性

#### 主控制扩展

| 属性                    | 类型       | 默认值    | 描述              |
| --------------------- | -------- | ------ | --------------- |
| **greeting**          | `string` | 空      | 第一个用户加入时的初始问候消息 |
| **max\_history**      | `int`    | 100    | 要保留的最大对话历史记录消息  |
| **enable\_interrupt** | `bool`   | `true` | 允许用户语音中断助手      |
| **lang**              | `string` | "en"   | 语言代码（en、cn 等）   |

**示例：**

```json
{
  "greeting": "嗨！我能为你做什么？",
  "max_history": 50,
  "enable_interrupt": true,
  "lang": "cn"
}
```

#### LLM 扩展（Gemini 2.0 Live）

| 属性                 | 类型       | 默认值                    | 描述                  |
| ------------------ | -------- | ---------------------- | ------------------- |
| **api\_key**       | `string` | -                      | Google Cloud API 密钥 |
| **model**          | `string` | "gemini-2.0-flash-exp" | 模型标识符               |
| **system\_prompt** | `string` | -                      | 模型的系统级指令            |
| **temperature**    | `float`  | 0.7                    | 响应随机性（0.0-1.0）      |
| **max\_tokens**    | `int`    | 1024                   | 最大响应长度              |

**示例：**

```json
{
  "api_key": "${GOOGLE_API_KEY}",
  "model": "gemini-2.0-flash-exp",
  "system_prompt": "你是一个有帮助的助手。",
  "temperature": 0.7,
  "max_tokens": 1024
}
```

<Callout type="warning">
  **安全性**: 不要直接提交 API 密钥。使用环境变量（`${VAR_NAME}`）并将值存储在 `.env` 文件或部署机密中。
</Callout>

#### STT 扩展（语音转文本）

| 属性              | 类型       | 默认值      | 描述                             |
| --------------- | -------- | -------- | ------------------------------ |
| **provider**    | `string` | "google" | STT 提供商（google、deepgram、azure） |
| **api\_key**    | `string` | -        | 提供商 API 密钥                     |
| **language**    | `string` | "zh-CN"  | 语言代码                           |
| **model**       | `string` | -        | 提供商特定的模型                       |
| **chunk\_size** | `int`    | 1024     | 音频块大小（字节）                      |

**示例：**

```json
{
  "provider": "google",
  "api_key": "${GOOGLE_API_KEY}",
  "language": "zh-CN",
  "chunk_size": 1024
}
```

#### TTS 扩展（文本转语音）

| 属性           | 类型       | 默认值      | 描述                               |
| ------------ | -------- | -------- | -------------------------------- |
| **provider** | `string` | "google" | TTS 提供商（google、elevenlabs、azure） |
| **api\_key** | `string` | -        | 提供商 API 密钥                       |
| **voice**    | `string` | -        | 语音标识符（提供商特定）                     |
| **language** | `string` | "zh-CN"  | 语言代码                             |
| **speed**    | `float`  | 1.0      | 语速（0.5-2.0）                      |

**示例：**

```json
{
  "provider": "elevenlabs",
  "api_key": "${ELEVENLABS_API_KEY}",
  "voice": "bella",
  "language": "zh-CN",
  "speed": 1.0
}
```

#### RTC 扩展

| 属性                   | 类型     | 默认值     | 描述           |
| -------------------- | ------ | ------- | ------------ |
| **port**             | `int`  | 8000    | RTC 连接的服务器端口 |
| **enable\_video**    | `bool` | `false` | 启用视频流        |
| **enable\_audio**    | `bool` | `true`  | 启用音频流        |
| **max\_connections** | `int`  | 100     | 最大并发连接数      |

**示例：**

```json
{
  "port": 8000,
  "enable_video": false,
  "enable_audio": true,
  "max_connections": 100
}
```

## manifest.json

`manifest.json` 文件定义元数据、版本信息和依赖项。

### 架构结构

```json
{
  "type": "extension",
  "name": "main_control",
  "version": "0.1.0",
  "api_version": "0.1",
  "dependencies": [
    {
      "type": "extension",
      "name": "mllm_in",
      "version": ">=0.1.0"
    }
  ],
  "api": {
    "input": [
      {
        "name": "on_start_request",
        "type": "json"
      }
    ],
    "output": [
      {
        "name": "mllm_in",
        "type": "json"
      }
    ]
  }
}
```

### 核心字段

| 字段               | 类型       | 描述                          |
| ---------------- | -------- | --------------------------- |
| **type**         | `string` | 扩展类型（"extension" 或 "addon"） |
| **name**         | `string` | 唯一扩展名称                      |
| **version**      | `string` | 语义版本（主.次.补）                 |
| **api\_version** | `string` | API 兼容性版本                   |
| **description**  | `string` | 人类可读的描述                     |
| **author**       | `string` | 扩展作者/维护者                    |
| **dependencies** | `array`  | 必需的依赖项                      |
| **api**          | `object` | 输入/输出规范                     |

### 依赖项部分

```json
"dependencies": [
  {
    "type": "extension",
    "name": "extension_name",
    "version": ">=0.1.0",
    "required": true
  }
]
```

| 字段           | 类型       | 默认值    | 描述                    |
| ------------ | -------- | ------ | --------------------- |
| **type**     | `string` | -      | "extension" 或 "addon" |
| **name**     | `string` | -      | 依赖项名称                 |
| **version**  | `string` | -      | 版本约束（>=0.1.0、\~1.0 等） |
| **required** | `bool`   | `true` | 依赖项是否为强制性             |

### API 规范

#### 输入

```json
"input": [
  {
    "name": "on_data",
    "type": "json",
    "required": true
  }
]
```

#### 输出

```json
"output": [
  {
    "name": "output_topic",
    "type": "json"
  }
]
```

## 配置示例

<Tabs items={["实时 V2V", "级联", "最小化"]}>
  <Tab>
    ### 实时 V2V 配置

    ```json
    {
      "extension_group": "ten_agent_extensions",
      "api_version": "0.1",
      "nodes": [
        {
          "type": "extension",
          "name": "main",
          "addon": "main_extension:MainExtension",
          "property": {
            "greeting": "你好！我是语音助手。",
            "max_history": 50,
            "lang": "cn"
          }
        },
        {
          "type": "extension",
          "name": "mllm_in",
          "addon": "mllm_in_v2v:MLLMV2VExtension",
          "property": {
            "api_key": "${GOOGLE_API_KEY}",
            "model": "gemini-2.0-flash-exp",
            "system_prompt": "你是一个有帮助的语音助手。"
          }
        },
        {
          "type": "extension",
          "name": "rtc",
          "addon": "rtc_extension:RTCExtension",
          "property": {
            "port": 8000,
            "enable_audio": true
          }
        }
      ]
    }
    ```
  </Tab>

  <Tab>
    ### 级联配置

    ```json
    {
      "extension_group": "ten_agent_extensions",
      "api_version": "0.1",
      "nodes": [
        {
          "type": "extension",
          "name": "main",
          "addon": "main_extension:MainExtension",
          "property": {
            "greeting": "你好！我能为你做什么？",
            "max_history": 100
          }
        },
        {
          "type": "extension",
          "name": "asr",
          "addon": "asr_extension:ASRExtension",
          "property": {
            "provider": "google",
            "api_key": "${GOOGLE_API_KEY}",
            "language": "zh-CN"
          }
        },
        {
          "type": "extension",
          "name": "llm",
          "addon": "llm_extension:LLMExtension",
          "property": {
            "api_key": "${OPENAI_API_KEY}",
            "model": "gpt-4",
            "temperature": 0.7
          }
        },
        {
          "type": "extension",
          "name": "tts",
          "addon": "tts_extension:TTSExtension",
          "property": {
            "provider": "elevenlabs",
            "api_key": "${ELEVENLABS_API_KEY}",
            "voice": "bella"
          }
        }
      ]
    }
    ```
  </Tab>

  <Tab>
    ### 最小化配置

    ```json
    {
      "extension_group": "ten_agent_extensions",
      "api_version": "0.1",
      "nodes": [
        {
          "type": "extension",
          "name": "main",
          "addon": "main_extension:MainExtension",
          "property": {}
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## 环境变量

<Callout type="tip">
  使用环境变量处理敏感数据和特定于环境的设置。
</Callout>

### 常见变量

```bash
# API 密钥
export GOOGLE_API_KEY="your-google-api-key"
export OPENAI_API_KEY="your-openai-api-key"
export ELEVENLABS_API_KEY="your-elevenlabs-key"

# 配置
export TEN_AGENT_LANG="cn"
export TEN_AGENT_GREETING="你好！"
export TEN_AGENT_PORT="8000"
```

### 加载环境变量

```python
import os
from dotenv import load_dotenv

load_dotenv()  # 从 .env 文件加载

api_key = os.getenv("GOOGLE_API_KEY")
```

## 验证提示

<Callout type="warning">
  在启动代理之前始终验证配置。
</Callout>

1. **检查必需字段**存在
2. **验证 API 密钥**非空
3. **验证版本约束**兼容
4. **测试依赖项连接**存在
5. **日志配置启动**用于调试

## 参考资源

* [TEN Framework 配置指南](https://github.com/TEN-framework/ten-framework)
* [属性文件格式](https://github.com/TEN-framework/ten-framework)
* [清单规范](https://github.com/TEN-framework/ten-framework)


# 运行演示 (/cn/docs/ten_agent_examples/demo)

本指南将帮助您运行 TEN-Agent 演示。该演示是一个基于 Web 的界面，允许您与 TEN-Agent 进行交互。演示站点的编排是固定的，无法更改。

## 从预构建的 Docker 镜像运行演示

运行演示的最简单方法是使用预构建的 Docker 镜像。该镜像包含 TEN-Agent 和演示的最新版本。项目 docker compose 文件已包含预构建的镜像，因此您可以按照[入门指南](./getting_started)中的步骤开始。

成功设置后，您需要运行一个额外的命令将图表文件夹切换到演示文件夹，

```bash
task use AGENT=agents/examples/demo
```

完成后，您可以通过打开浏览器并导航到 `http://localhost:3002` 来访问演示界面。

## 从源代码运行演示

如果您想从源代码运行演示，请启动一个新终端，更改到克隆 TEN-Agent 项目的文件夹，然后按照以下步骤操作：

```bash
# 切换到演示目录
cd demo

# 安装依赖项
pnpm install

# 启动演示
pnpm dev
```

演示启动后，您可以通过打开浏览器并导航到 `http://localhost:3001` 来访问它。

> **注意：** 该演示依赖于 golang Web 服务器（位于 `server` 目录中）。在您按照[入门指南](./getting_started)中的步骤操作后，在容器中运行 `task run` 命令时，Web 服务器将启动。


# Deploy TEN Agent Example (/cn/docs/ten_agent_examples/deploy_ten_agent/deploy_agent_service)

Once you have customized your agent (either by using the playground or editing property.json directly), you can deploy it by creating a release docker image for your service.

## Make your agent service into a docker image

The Dockerfile has been prepared in every example folder. You can build the docker image by running the following command:

```shell
cd ai_agents
docker build -f agents/examples/<example-name>/Dockerfile -t example-app .
```

For Apple Silicon, use the following command:

```shell
docker build -f agents/examples/<example-name>/Dockerfile -t example-app . --platform linux/amd64
```

> Note: The above docker build commands need to be executed outside of Docker container.

## Run the docker image

After building the docker image, you can run it by running the following command:

```shell
docker run --rm -it --env-file .env -p 8080:8080 -p 3000:3000 example-app
```

The agent service will be running on [http://localhost:8080](http://localhost:8080), and a frontend will be ready for access on [http://localhost:3000](http://localhost:3000).

## Test the agent service

The service exposes a list of apis that you can use to interact with the agent. The api reference can be found [https://github.com/TEN-framework/TEN-Agent/tree/main/server](https://github.com/TEN-framework/TEN-Agent/tree/main/server).

You can also test the agent service by accessing [http://localhost:3000](http://localhost:3000).


# 创建 TTS 扩展 (/cn/docs/ten_agent_examples/extension_dev/TTS_Extension_Development_Guide)

# 创建 TTS Extension 完整指南

本教程将指导你从零开始创建一个生产级别的 TTS（Text-to-Speech） Extension，涵盖从项目创建、核心开发、测试验证到发布上线的完整流程。

## 什么是 TTS Extension

TTS Extension 是 TEN Framework 生态系统中的一个**标准扩展积木**（Standard Extension），专门用于实现文本转语音（Text-to-Speech）功能。

### 核心功能

TTS Extension 的主要职责包括：

1. **接收文字**: 从上游扩展持续接收需要转换成语音的文字（通常来自于大模型）
2. **实时合成**: 将文字实时转换成对应的音频数据流
3. **发送音频**: 将合成的音频数据传递给下游扩展进行后续处理

### 在对话流中的位置

作为标准积木，TTS Extension 在 TEN Agent 对话流中扮演着**文本到音频转换**的关键角色：

```
[上游积木]  ──文字流──>  [TTS Extension]  ──音频流──>  [下游积木]
```

**典型的上游积木**：

* **LLM Extension**: 生成对话回复文字
* **Translation Extension**: 翻译后的文字结果
* **Text Processing Extension**: 经过预处理的文字内容

**典型的下游积木**：

* **RTC Extension**: 将音频推送到 RTC 频道
* **Audio Playback Extension**: 本地播放音频
* **Audio Processing Extension**: 对音频进行后处理（如混音、音效等）

## 📚 实现模式

TTS的实现基本分为[http版](#http-模式)和[websocket版](#websocket-模式)两种（目前TTS厂商主流支持这两种）。有些厂商提供了SDK，但实际底层基本也是http或websocket，根据SDK的实际行为选择http模式或http模式。
http是比较基础的实现方法，建议初次开发TTS Extension的开发者优先实现[http版](#http-模式)。然后再尝试实现进阶的[websocket版](#websocket-模式)。

### 架构介绍

目前有`AsyncTTS2BaseExtension` 和`AsyncTTS2HttpExtension`两个基类供开发者继承，基于这两个基类可以更方便的实现TTS Extension。

### 实现方式基本区分

**路径1：直接继承AsyncTTS2BaseExtension（WebSocket模式）**

```
┌─────────────────────────────────────────────────────────────┐
│                    AsyncTTS2BaseExtension                    │
│  【通用基类】提供TTS Extension的基础设施                      │
│  - 消息队列管理                                               │
│  - 生命周期管理                                               │
│  - 音频数据发送                                               │
│  - 指标上报                                                   │
│  - 错误处理                                                   │
└─────────────────────────────────────────────────────────────┘
                              ↑ 继承（路径1）
┌─────────────────────────────────────────────────────────────┐
│        VendorTTSExtension (WebSocket/SDK模式子类)             │
│  【子类实现】实现厂商特定逻辑                                   │
│  - 完整的request_tts()实现                                     │
│  - WebSocket/SDK客户端实现                                     │
│  - 配置类实现                                                  │
│  - 供应商信息                                                  │
│  - 采样率配置                                                  │
└─────────────────────────────────────────────────────────────┘
```

**路径2：继承AsyncTTS2HttpExtension（HTTP模式）**

```
┌─────────────────────────────────────────────────────────────┐
│                    AsyncTTS2BaseExtension                    │
│  【通用基类】提供TTS Extension的基础设施                      │
│  - 消息队列管理                                               │
│  - 生命周期管理                                               │
│  - 音频数据发送                                               │
│  - 指标上报                                                   │
│  - 错误处理                                                   │
└─────────────────────────────────────────────────────────────┘
                              ↑ 继承
┌─────────────────────────────────────────────────────────────┐
│               AsyncTTS2HttpExtension (HTTP模式)              │
│  【模式基类】提供HTTP模式的完整实现                             │
│  - 配置加载和验证                                              │
│  - 客户端管理                                                  │
│  - 请求处理逻辑（完整的request_tts()实现）                     │
│  - TTFB计算和上报                                              │
│  - PCMWriter管理                                              │
└─────────────────────────────────────────────────────────────┘
                              ↑ 继承（路径2）
┌─────────────────────────────────────────────────────────────┐
│              VendorTTSExtension (HTTP模式子类)                 │
│  【子类实现】实现厂商特定逻辑                                   │
│  - 配置类实现（create_config()）                               │
│  - 客户端实现（create_client()）                               │
│  - 供应商信息（vendor()）                                      │
│  - 采样率配置（synthesize_audio_sample_rate()）               │
└─────────────────────────────────────────────────────────────┘
```

详细的基类、子类职责可参考 [TTS 基类、子类各自功能](#tts-基类子类各自功能)

## 🚀 项目初始化

### 创建扩展项目

使用 TMan 的 TTS 专用模板快速创建项目骨架：

```bash title="Terminal"
# 进入扩展目录
cd ten-framework/ai_agents/agents/ten_packages/extension

# 创建TTS扩展项目
tman create extension my_tts_extension --template default_tts_python --template-data class_name_prefix=MyTts
```

创建成功后会显示：

```bash title="输出信息"
Package 'extension:my_tts_extension' created successfully in 'my_tts_extension' in 2 seconds.
```

### 安装项目依赖

#### 添加第三方库依赖

首先在 `requirements.txt` 中添加所需的第三方库依赖：

```text title="requirements.txt"
websockets~=14.0
pydantic
requests
httpx
aiofiles
```

#### 安装 TEN 依赖

进入创建的扩展目录并安装依赖：

```bash title="Terminal"
cd my_tts_extension
tman install --standalone
```

这会根据 `manifest.json` 中声明的依赖构建依赖树，并安装到 `.ten` 目录下。

## 项目结构概览

```
my_tts_extension/
├── .vscode/               # VS Code 调试配置
│   └── launch.json       # 调试启动配置
├── manifest.json          # 扩展元数据和依赖声明
├── property.json          # 默认配置参数，可参考[property.json内容](#property.json内容)
├── requirements.txt       # Python 依赖
├── config.py              # 配置管理类，可参考 [配置管理设计](#配置管理设计)
├── {vendor}_tts.py        # TTS客户端核心实现
├── extension.py           # 主要实现文件
└── tests/                 # 测试文件
    ├── bin/start          # 测试启动脚本
    ├── test_basic.py      # 单元测试
    └── configs/           # 测试配置
```

## HTTP 模式

### 模式特点

HTTP模式使用标准的HTTP流式请求，适合传统的REST API TTS服务。实现简单，维护成本低，但延迟相对较高。典型实现如Rime TTS。

### 核心架构

```
┌─────────────────┐    HTTP Request  ┌─────────────────┐
│   Extension     │─────────────────►│  TTS Provider   │
│                 │  POST请求         │                 │
│ - HTTP客户端     │                  │ - REST API      │
│ - 流式响应处理    │◄─────────────────│ - 流式音频响应    │
│ - 错误重试       │  HTTP Response    │ - 错误处理       │
└─────────────────┘                  └─────────────────┘
```

### 实现规范

HTTP模式基于`AsyncTTS2HttpExtension`基类实现，该基类已经提供了完整的请求处理逻辑、TTFB计算、音频数据处理等功能。开发者只需要实现客户端接口和配置类即可。

<div className="fd-steps">
  <div className="fd-step">
    #### HTTP客户端接口实现

    **客户端必须实现`AsyncTTS2HttpClient`接口：**

    ```python
    from ten_ai_base.tts2_http import AsyncTTS2HttpClient
    from ten_ai_base.struct import TTS2HttpResponseEventType
    from httpx import AsyncClient, Timeout, Limits

    class VendorTTSClient(AsyncTTS2HttpClient):
        """HTTP TTS Client, implements AsyncTTS2HttpClient interface"""
        
        def __init__(self, config: VendorTTSConfig, ten_env: AsyncTenEnv):

            # API endpoint configuration (using abstract method)
            self.endpoint = self._get_api_endpoint()
            
            # Request headers configuration (using abstract method)
            self.headers = self._create_headers()
            
            # HTTP client configuration
            self.client = AsyncClient(
                http2=True,  # Enable HTTP/2
                follow_redirects=True,
            )
        
        def _get_api_endpoint(self) -> str:
            """Get API endpoint - to be implemented by subclass"""
            raise NotImplementedError("Subclasses must implement _get_api_endpoint")
        
        def _create_headers(self) -> dict:
            """Create request headers - to be implemented by subclass"""
            raise NotImplementedError("Subclasses must implement _create_headers")
    ```

    **资源管理方法：**

    ```python
    async def clean(self) -> None:
        """Clean up resources - required by AsyncTTS2HttpClient interface"""
        if self.client:
            self.ten_env.log_debug("Cleaning HTTP client")
            # Note: Depending on actual implementation, may only need to set client to None
            # Or call await self.client.aclose()
            self.client = None

    async def cancel(self) -> None:
        """Cancel current request - required by AsyncTTS2HttpClient interface"""
        self.ten_env.log_debug("VendorTTS: cancel() called.")
        self._is_cancelled = True
    ```
  </div>

  <div className="fd-step">
    #### 请求处理

    **TTS请求处理（实现AsyncTTS2HttpClient接口）：**

    ```python
    from typing import AsyncIterator, Tuple
    from ten_ai_base.struct import TTS2HttpResponseEventType

    async def get(
        self, text: str, request_id: str
    ) -> AsyncIterator[Tuple[bytes | None, TTS2HttpResponseEventType]]:
        """Process single TTS request - required by AsyncTTS2HttpClient interface
        
        Note: Return type uses TTS2HttpResponseEventType enum, not integer
        """
        self._is_cancelled = False
        
        if not self.client:
            return
        
        try:
            # Build request data (using abstract method)
            request_data = self._create_request_data(text)
            
            # Send streaming request
            async with self.client.stream(
                "POST",
                self.endpoint,
                headers=self.headers,
                json=request_data,
            ) as response:
                # Handle response
                async for chunk in response.aiter_bytes(chunk_size=4096):
                    # Process response, prepare audio
                    pass
                
                # Request completed
                if not self._is_cancelled:
                    yield None, TTS2HttpResponseEventType.END
        
        except Exception as e:
            # Error handling
            pass

    def _create_request_data(self, text: str) -> dict:
        """Create request data - to be implemented by subclass"""
        raise NotImplementedError("Subclasses must implement _create_request_data")

    ```
  </div>

  <div className="fd-step">
    #### Extension实现

    **Extension必须继承`AsyncTTS2HttpExtension`：**

    ```python
    from ten_ai_base.tts2_http import AsyncTTS2HttpExtension, AsyncTTS2HttpConfig, AsyncTTS2HttpClient

    class VendorTTSExtension(AsyncTTS2HttpExtension):
        """TTS Extension implementation, inherits from AsyncTTS2HttpExtension base class"""
        
        async def create_config(self, config_json_str: str) -> AsyncTTS2HttpConfig:
            """Create config object - required by AsyncTTS2HttpExtension interface"""
            return VendorTTSConfig.model_validate_json(config_json_str)
        
        async def create_client(
            self, config: AsyncTTS2HttpConfig, ten_env: AsyncTenEnv
        ) -> AsyncTTS2HttpClient:
            """Create client object - required by AsyncTTS2HttpExtension interface"""
            return VendorTTSClient(config=config, ten_env=ten_env)
        
        def vendor(self) -> str:
            """Return vendor name"""
            return "vendor_name"
        
        def synthesize_audio_sample_rate(self) -> int:
            """Return audio sample rate"""
            return self.config.sample_rate if self.config else 16000
    ```

    **注意**：`AsyncTTS2HttpExtension`基类已经实现了完整的`request_tts()`方法，包括：

    * 请求处理逻辑
    * TTFB计算和上报
    * 音频数据处理
    * 错误处理
    * PCM文件写入

    开发者只需要实现上述抽象方法即可。
  </div>

  <div className="fd-step">
    #### 请求优化

    http请求有一些可以优化的参数，可以根据TTS厂商情况设置。可参考 [HTTP 请求优化](#http-请求优化)
  </div>
</div>

### 最佳实践

1. **使用基类**：继承`AsyncTTS2HttpExtension`基类，避免重复实现请求处理逻辑
2. **接口实现**：严格按照`AsyncTTS2HttpClient`接口实现客户端
3. **事件类型**：使用`TTS2HttpResponseEventType`枚举，确保类型安全
4. **抽象方法**：使用抽象方法（`_get_api_endpoint()`, `_create_headers()`等）分离厂商特定逻辑
5. **连接池管理**：使用HTTP连接池，减少连接建立开销
6. **超时控制**：设置合理的超时时间
7. **错误处理**：正确返回事件类型，让Extension基类统一处理错误
8. **元数据上报**：实现`get_extra_metadata()`方法，支持TTFB指标上报
9. **日志记录**：实现必要的日志记录，包括供应商错误、状态变化、请求响应等。详细要求请参考 [日志规范](#日志规范)

## websocket 模式

### 模式特点

WebSocket模式支持websocket通信，允许流式音频响应，实现最低延迟的TTS响应。典型实现如ElevenLabs TTS2。

### 核心架构

```
┌─────────────────┐    WebSocket     ┌─────────────────┐
│   Extension     │─────────────────►│  TTS Provider   │
│                 │  发送请求         │                 │
│ - 请求管理       │                  │ - 流式音频响应    │
│ - 响应处理       │◄─────────────────│ - 任务状态管理    │
│ - 连接复用       │  接收响应          │ - 错误处理       │
└─────────────────┘                  └─────────────────┘
```

### 实现规范

<div className="fd-steps">
  <div className="fd-step">
    #### 连接管理策略

    **连接生命周期管理：**

    ```python
    class VendorTTS2Synthesizer:
        def __init__(self, config, ten_env, error_callback, response_msgs):
            # Connection state management
            self._session_closing = False
            self._connect_exp_cnt = 0
            self.websocket_task = None
            self.channel_tasks = []
            self._session_started = False
            
            # Event synchronization mechanism
            self._connection_event = asyncio.Event()
            self._connection_success = False
            self._receive_ready_event = asyncio.Event()
            
            # Start WebSocket connection monitoring
            self.websocket_task = asyncio.create_task(self._process_websocket())
    ```

    **自动重连机制：**

    ```python
    async def _process_websocket(self) -> None:
        """Main WebSocket connection monitoring and reconnection logic"""
        try:
            # Use websockets.connect's automatic reconnection mechanism
            async for ws in websockets.connect(
                uri=self.uri,
                ...
            ):
                self.ws = ws
                try:
                    # Start send and receive tasks
                    self.channel_tasks = [
                        # For vendors supporting bidirectional streams, can use both send and receive loops.
                        # If vendor only supports one request at a time, don't use loop for send functionality
                        asyncio.create_task(self._send_loop(ws)),
                        asyncio.create_task(self._receive_loop(ws)),
                    ]
                    
                    # Wait for receive loop to be ready
                    await self._receive_ready_event.wait()
                    await self.start_connection()
                    
                    await self._await_channel_tasks()
                    
                except websockets.ConnectionClosed as e:
                    if not self._session_closing:
                        # Reset all event states
                        self._receive_ready_event.clear()
                        self._connection_event.clear()
                        self._connection_success = False
                        self._session_started = False
                        continue
        except Exception as e:
            self.ten_env.log_error(f"WebSocket connection error: {e}")
    ```
  </div>

  <div className="fd-step">
    #### 消息队列处理

    **文本输入队列：**

    ```python
    async def _send_loop(self, ws: ClientConnection) -> None:
        """Text sending loop"""
        try:
            # Send initialization message
            init_msg = ...
            await ws.send(json.dumps(init_msg))
            
            while not self._session_closing:
                # Send text
                pass
                    
        except asyncio.CancelledError:
            raise
        except Exception as e:
            self.ten_env.log_error(f"Exception in send_loop: {e}")
            raise
    ```

    **音频接收处理：**

    ```python
    async def _receive_loop(self, ws: ClientConnection) -> None:
        """Message receiving loop"""
        try:
            self._receive_ready_event.set()
            
            while self._session_closing == False:
                message = await ws.recv()
                # Parse response
                pass
                    
        except asyncio.CancelledError:
            raise
        except Exception as e:
            self.ten_env.log_error(f"Exception in receive_loop: {e}")
            raise
    ```
  </div>

  <div className="fd-step">
    #### 错误处理和重连机制

    **异常处理策略：**

    ```python
    def _process_ws_exception(self, exp) -> None | Exception:
        """Handle WebSocket connection exception and decide whether to reconnect"""
        self.ten_env.log_debug(f"Websocket internal error: {exp}")
        self._connect_exp_cnt += 1
        
        if self._connect_exp_cnt > 5:  # Maximum retry count
            self.ten_env.log_error(f"Max retries exceeded: {str(exp)}")
            return exp
        return None  # Continue reconnecting
    ```

    **错误回调处理：**

    ```python
    if self.error_callback:
        module_error = ModuleError(
            message=data["error"],
            module=ModuleType.TTS,
            code=error_code,
            vendor_info=error_info,
        )
        await self.error_callback("", module_error)
    ```
  </div>

  <div className="fd-step">
    #### 资源管理和清理

    **连接取消和清理：**

    ```python
    def cancel(self) -> None:
        """Cancel current connection, used for flush scenarios"""

    def _clear_queues(self) -> None:
        """Clear all queues to prevent processing old data, used after receiving flush"""

    ```
  </div>
</div>

### 最佳实践

1. **连接预热**：在初始化时建立WebSocket连接，减少首次请求延迟
2. **自动重连**：实现指数退避的重连策略
3. **资源清理**：及时清理取消的合成器，避免内存泄漏
4. **错误分类**：区分网络错误、认证错误和业务错误
5. **队列管理**：使用有界队列防止内存溢出
6. **超时处理**：设置合理的超时时间，避免长时间阻塞
7. **日志记录**：实现必要的日志记录，包括供应商错误、状态变化、请求响应等。详细要求请参考 [日志规范](#日志规范)

## Extension 结构及其他文件实现

Extension目录的结构可参考 [项目结构概览](#项目结构概览)

配置Config文件实现可参考 [配置管理设计](#配置管理设计)

必须实现的功能可参考 [必须实现的功能](#必须实现的功能-1)

不同模式的特殊要求可参考 [不同模式的特殊要求](#不同模式的特殊要求)

## 🧪 单元测试

### 测试文件结构

TTS Extension的测试目录应包含完整的测试套件，确保代码质量和功能正确性。

```
tests/
├── __init__.py                    # 测试包初始化
├── conftest.py                   # pytest配置和fixtures
├── test_basic.py                 # 基础功能测试
├── test_error_msg.py             # 错误处理测试，主要检测出现错误时error message的生成
├── test_params.py                # 参数配置测试，主要检测配置参数的检查功能
├── test_robustness.py            # 健壮性测试，主要检测对于异常情况的处理
├── test_metrics.py               # 指标测试
└── configs/                      # 测试配置文件
    ├── test_config.json          # 测试配置
    ├── invalid_config.json       # 无效配置测试
    └── mock_config.json          # Mock测试配置
```

### 测试最佳实践

1. **测试覆盖**：确保所有主要功能都有测试覆盖
2. **Mock使用**：合理使用Mock避免依赖外部服务
3. **异步测试**：正确处理异步测试和等待
4. **资源清理**：确保测试后正确清理资源
5. **错误场景**：测试各种错误和异常情况
6. **边界测试**：测试边界条件和极限情况
7. **并发测试**：测试并发请求处理能力
8. **配置测试**：测试各种配置参数组合

## 🔗 集成测试（Guarder）

### 环境变量配置

创建 `.env` 文件配置真实 API 密钥：

```bash title=".env"
# TTS Vendor Services API Key
VENDOR_TTS_API_KEY=your_api_key_here
# 例如：
ELEVENLABS_TTS_API_KEY=your_elevenlabs_api_key
```

### 测试配置

在 `tests/configs/` 目录下创建以下测试配置文件，Guarder 测试会使用这些配置：

<div className="fd-steps">
  <div className="fd-step">
    #### 基础音频设置配置

    **`property_basic_audio_setting1.json`** - 用于基础音频设置测试、边界输入测试、指标测试、无效文本处理测试：

    ```json title="tests/configs/property_basic_audio_setting1.json"
    {
      "dump": true,
      "dump_path": "./tests/keep_dump_output/",
      "params": {
        "output_format": "pcm_44100",
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```

    **`property_basic_audio_setting2.json`** - 用于基础音频设置测试（对比不同的采样率配置）：

    ```json title="tests/configs/property_basic_audio_setting2.json"
    {
      "dump": true,
      "dump_path": "./tests/keep_dump_output/",
      "params": {
        "output_format": "pcm_44100",
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```

    **注意**：根据你的 TTS 服务商，可能需要调整参数名称（如 `key`、`api_key`、`sample_rate`、`output_format` 等）。
  </div>

  <div className="fd-step">
    #### Dump 功能测试配置

    **`property_dump.json`** - 用于 dump 功能测试、flush 测试、按请求ID导出测试：

    ```json title="tests/configs/property_dump.json"
    {
      "dump": true,
      "dump_path": "./tests/dump_output/",
      "params": {
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```
  </div>

  <div className="fd-step">
    #### 错误处理测试配置

    **`property_invalid.json`** - 用于无效必需参数测试（测试无效的 API Key）：

    ```json title="tests/configs/property_invalid.json"
    {
      "params": {
        "key": "invalid"
      }
    }
    ```

    **`property_miss_required.json`** - 用于缺少必需参数测试（测试缺少 API Key）：

    ```json title="tests/configs/property_miss_required.json"
    {
      "params": {
        "key": ""
      }
    }
    ```
  </div>

  <div className="fd-step">
    #### 完整配置示例（可选）

    **`property_bechmarl.json`** - 完整配置示例，包含多个参数，可用于基准测试：

    ```json title="tests/configs/property_bechmarl.json"
    {
      "dump": true,
      "dump_path": "./tests/keep_dump_output/",
      "params": {
        "output_format": "pcm_44100",
        "language": "en",
        "voice_id": "your_voice_id",
        "model_id": "your_model_id",
        "key": "${env:VENDOR_TTS_API_KEY}"
      }
    }
    ```
  </div>
</div>

### 配置文件说明

**关键配置项**：

* **`params.key`** 或 **`params.api_key`**：TTS 服务商的 API Key，使用 `${env:VENDOR_TTS_API_KEY}` 从环境变量读取
* **`dump`**：是否启用音频 dump 功能（`true` 或 `false`）
* **`dump_path`**：音频 dump 文件的保存路径
* **`params.sample_rate`** 或 **`params.output_format`**：音频采样率配置，根据服务商不同而不同

**环境变量支持**：

配置文件中可以使用 `${env:VARIABLE_NAME}` 格式从环境变量读取值，这样可以在不修改配置文件的情况下切换不同的 API Key。

### 运行 Guarder 测试

使用真实 API 密钥运行完整集成测试：

```bash title="Terminal"
cd ai_agents
task tts-guarder-test EXTENSION=your_extension_name CONFIG_DIR=tests/configs
```

### Guarder测试点说明

Guarder集成测试包含以下测试点，确保Extension符合TEN Framework的标准：

#### 1. **基础音频设置测试** (`test_basic_audio_setting.py`)

**测试目标**：验证Extension能够根据配置正确设置音频参数

**测试点**：

* 验证不同配置文件的采样率设置是否正确
* 验证Extension能够正确读取和响应不同的采样率配置
* 验证音频帧的采样率与实际配置一致

**预期结果**：

* 不同配置文件产生不同的采样率输出
* 所有音频帧的采样率保持一致
* 没有错误或异常

#### 2. **边界输入测试** (`test_corner_input.py`)

**测试目标**：验证Extension对边界输入的处理

**测试点**：

* 验证Extension能够处理边界输入
* 验证Extension能够正确上报metrics数据
* 验证metrics在`tts_audio_end`之前上报

**预期结果**：

* Extension能够处理边界输入并生成音频
* 能够接收到metrics数据
* metrics在`tts_audio_end`之前发送

#### 3. **PCM Dump功能测试** (`test_dump.py`)

**测试目标**：验证PCM音频文件导出功能

**测试点**：

* 验证启用dump功能时，能够生成PCM文件
* 验证PCM文件保存在指定路径
* 验证PCM文件不为空

**预期结果**：

* 当`dump=true`时，在`dump_path`目录下生成PCM文件
* PCM文件包含音频数据
* 文件命名符合规范

#### 4. **按请求ID导出测试** (`test_dump_each_request_id.py`)

**测试目标**：验证每个请求ID都能生成独立的dump文件

**测试点**：

* 验证多个请求能够生成多个独立的dump文件
* 验证每个请求ID对应一个dump文件
* 验证dump文件数量正确

**预期结果**：

* 发送N个请求，生成N个dump文件
* 每个dump文件对应一个request\_id
* 文件命名包含request\_id信息

#### 5. **Flush功能测试** (`test_flush.py`)

**测试目标**：验证flush请求的处理

**测试点**：

* 验证收到flush请求时，能够正确中断当前请求
* 验证flush时发送的`tts_audio_end`事件，`reason`应为`2`（INTERRUPTED）
* 验证flush完成后发送`tts_flush_end`事件
* 验证事件顺序：`tts_audio_end`在`tts_flush_end`之前

**预期结果**：

* 收到flush请求后，能够中断当前TTS合成
* 发送`tts_audio_end`事件，`reason=2`（INTERRUPTED）
* 发送`tts_flush_end`事件
* 事件顺序正确

#### 6. **指标上报测试** (`test_metrics.py`)

**测试目标**：验证指标数据的正确上报

**测试点**：

* 验证Extension能够上报metrics数据
* 验证metrics在`tts_audio_end`之前发送
* 验证metrics包含必需字段（module、vendor、metrics等）

**预期结果**：

* 能够接收到metrics数据
* metrics在`tts_audio_end`之前发送
* metrics数据格式正确，包含必需字段

#### 7. **无效必需参数测试** (`test_invalid_required_params.py`)

**测试目标**：验证对无效必需参数的错误处理

**测试点**：

* 验证当配置参数无效时（如API key格式错误），能够正确返回错误
* 验证错误代码为`FATAL_ERROR`（-1000）
* 验证错误消息清晰，便于用户定位问题

**预期结果**：

* Extension能够检测到无效的必需参数
* 返回错误，错误代码为`-1000`（FATAL\_ERROR）
* 错误消息包含问题描述

#### 8. **无效文本处理测试** (`test_invalid_text_handling.py`)

**测试目标**：验证Extension对无效文本的处理能力

**测试点**：

* 验证当文本无效时（如空字符串、纯标点、纯空格等），能够正确处理
* 验证返回错误代码为`NON_FATAL_ERROR`（1000）
* 验证错误包含`vendor_info`字段
* 验证在错误后，发送有效文本能够正常工作

**测试的无效文本类型**：

* 空字符串
* 纯空格、制表符、换行符
* 纯标点符号（中英文）
* 表情符号和特殊字符
* 数学公式、化学方程式
* 混合无效字符

**预期结果**：

* 无效文本返回错误，错误代码为`1000`（NON\_FATAL\_ERROR）
* 错误包含`vendor_info`，包含供应商名称、错误代码和消息
* 错误后，发送有效文本能够正常生成音频

#### 9. **缺少必需参数测试** (`test_miss_required_params.py`)

**测试目标**：验证对缺少必需参数的错误处理

**测试点**：

* 验证当缺少必需参数时（如缺少API key），能够正确返回错误
* 验证错误代码为`FATAL_ERROR`（-1000）
* 验证错误消息提示缺少的参数

**预期结果**：

* Extension能够检测到缺少的必需参数
* 返回错误，错误代码为`-1000`（FATAL\_ERROR）
* 错误消息明确提示缺少的参数

### 测试注意事项

1. **环境准备**：确保设置正确的环境变量（API key等）
2. **配置准备**：确保测试配置文件正确，包含有效的API凭证
3. **网络连接**：确保能够访问TTS供应商的API服务
4. **测试隔离**：每次测试运行前清理dump目录，确保测试结果准确
5. **错误分析**：如果测试失败，仔细查看日志输出，定位问题原因
6. **测试顺序**：某些测试可能有依赖关系，建议按顺序运行
7. **资源清理**：测试完成后清理临时文件和资源

## 🌐 端到端测试

完成开发后，可以使用 TMan Designer 快速替换 TEN Agent 对话图中的 TTS 扩展，验证在实际对话场景下的效果。

### 使用 TMan Designer 替换 TTS 扩展

```bash title="Terminal"
# 在 TEN Agent 项目目录下启动
cd /path/to/your/ten-agent-project
tman designer
```

TMan Designer 会打开可视化界面，你可以：

1. **选择 TTS 节点**: 点击现有的 TTS 扩展积木
2. **替换为你的扩展**: 选择 `my_tts_extension`
3. **配置参数**: 设置 API Key、语音ID等参数
4. **一键应用**: 完成替换并启动测试

替换完成后，通过真实对话验证扩展的音频质量、响应速度和稳定性。

## 提Pull Request前检查清单

在提交Pull Request之前，请确保已完成以下所有检查项：

<div className="fd-steps">
  <div className="fd-step">
    ### 功能实现 ✅

    **要求**：完成所有核心功能的实现

    **检查项**：

    * [ ] 实现`request_tts()`方法，能够正确处理文本输入并生成音频
    * [ ] 实现`vendor()`方法，返回正确的供应商名称
    * [ ] 实现`synthesize_audio_sample_rate()`方法，返回正确的采样率
    * [ ] 如果使用HTTP模式，实现`create_config()`和`create_client()`方法
    * [ ] 如果使用HTTP模式，实现配置类和客户端类的所有必需方法
    * [ ] 如果是websocket模式，要有自动重连机制
    * [ ] 正确实现错误处理，区分`FATAL_ERROR`和`NON_FATAL_ERROR`
    * [ ] 正确处理flush请求，实现`cancel_tts()`方法
    * [ ] 正确发送`tts_audio_start`和`tts_audio_end`事件
    * [ ] 正确计算和上报TTFB指标
    * [ ] 正确计算和上报音频时长
    * [ ] 正确发送metrics数据
    * [ ] 日志记录符合要求
  </div>

  <div className="fd-step">
    ### 提交的文件 ✅

    * [ ] **厂商交互功代码**：通常为xxx\_tts.py
    * [ ] **主控制代码**：通常为extension.py
    * [ ] **单元测试代码**：通常在tests文件夹下
    * [ ] **Guarder测试配置文件**：通常在tests/configs文件夹下
    * [ ] **最小启动参数文件**：property.json
    * [ ] **版本、依赖、接口定义文件**：manifest.json
  </div>

  <div className="fd-step">
    ### 单元测试（UT）✅

    **要求**：完成所有单元测试，确保代码质量

    **检查项**：

    * [ ] 所有单元测试通过
    * [ ] 测试所有主要功能路径
    * [ ] 测试错误处理逻辑
    * [ ] 测试边界条件
    * [ ] 测试参数验证逻辑
    * [ ] 测试配置加载和验证

    **运行命令**：

    ```bash
    # 方式1：使用task命令（推荐）
    # 在项目根目录（ten-framework/ai_agents）下运行
    task test-extension EXTENSION=agents/ten_packages/extension/your_extension_name

    # 方式2：手动运行
    cd agents/ten_packages/extension/your_extension_name
    tman -y install --standalone
    ./tests/bin/start

    # 方式3：运行所有Extension的测试
    task test-agent-extensions
    ```

    **注意**：

    * 确保在项目根目录（`ten-framework/ai_agents`）下运行task命令
    * 如果Extension没有安装依赖，task命令会自动安装
    * 测试配置文件应在`tests/configs/`目录下
  </div>

  <div className="fd-step">
    ### Ten Agent自测 ✅

    **要求**：在Ten Agent中完成自测，确保Extension能够正常工作

    **检查项**：

    * [ ] 在Ten Agent中成功加载Extension
    * [ ] 能够听到Agent声音
    * [ ] 能够正常多轮对话
    * [ ] 能够正常打断对话
  </div>

  <div className="fd-step">
    ### Guarder集成测试 ✅

    **要求**：通过所有Guarder集成测试，并将测试结果贴在PR评论中

    **测试位置**：`ten-framework/ai_agents/agents/integration_tests/tts_guarder`

    **运行命令**：

    ```bash
    # 方式1：使用task命令（推荐）
    # 在项目根目录（ten-framework/ai_agents）下运行
    task tts-guarder-test EXTENSION=your_extension_name CONFIG_DIR=tests/configs

    # 方式2：手动运行
    cd agents/integration_tests/tts_guarder
    ./scripts/install_deps_and_build.sh linux x64
    ./tests/bin/start --extension_name your_extension_name --config_dir agents/ten_packages/extension/your_extension_name/tests/configs

    # 运行单个测试文件
    ./tests/bin/start --extension_name your_extension_name --config_dir agents/ten_packages/extension/your_extension_name/tests/configs tests/test_basic_audio_setting.py
    ```

    **环境变量设置**：

    ```bash
    # 在项目根目录创建.env文件，或设置环境变量
    # 根据实际供应商设置API Key
    export VENDOR_TTS_API_KEY=your_api_key_here
    # 例如：
    export ELEVENLABS_TTS_API_KEY=your_elevenlabs_api_key
    # 或在.env文件中：
    # ELEVENLABS_TTS_API_KEY=your_elevenlabs_api_key
    ```

    **注意**：

    * 确保在项目根目录（`ten-framework/ai_agents`）下运行task命令
    * 测试配置文件应在Extension目录下的`tests/configs/`目录中
    * 如果使用task命令，会自动从`.env`文件读取环境变量
    * 确保Extension已正确安装依赖

    **测试结果要求**：

    * [ ] 所有Guarder测试通过
    * [ ] 在PR评论中粘贴guarder 测试结果
    * [ ] 如有测试失败，说明原因并提供解决方案
  </div>
</div>

### PR提交检查清单总结

在提交PR之前，请确认以下所有项目已完成：

* [ ] **功能实现**：所有核心功能已实现并通过自测
* [ ] **所有必备文件都已提交**：代码运行的必备文件
* [ ] **单元测试**：所有UT通过，覆盖率达标
* [ ] **Ten Agent自测**：在Ten Agent中验证功能正常
* [ ] **Guarder测试**：所有Guarder测试通过
* [ ] **测试结果**：在PR评论中粘贴Guarder测试结果
* [ ] **代码审查**：代码已自检，符合编码规范
* [ ] **提交信息**：PR描述清晰，包含变更说明和测试结果

## Spec

### HTTP 请求优化

**连接复用：**

```python
class VendorTTSClient:
    def __init__(self, config, ten_env):
        # Use connection pool
        self.client = AsyncClient(
            timeout=Timeout(
                connect=10.0,  # Connection timeout
                read=30.0,     # Read timeout
                write=10.0,    # Write timeout
                pool=5.0       # Connection pool timeout
            ),
            limits=Limits(
                max_connections=100,
                max_keepalive_connections=20,
                keepalive_expiry=600.0,
            ),
            http2=True,
            follow_redirects=True,
        )
```

**请求压缩：**

```python
def _get_headers(self) -> dict:
    """Get optimized request headers"""
    return {
        "Authorization": f"Bearer {self.api_key}",
        "Content-Type": "application/json",
        "Accept": "audio/pcm",
        "Accept-Encoding": "gzip, deflate",  # Enable compression
        "User-Agent": "TEN-Framework-TTS/1.0",
    }
```

### TTS 基类、子类各自功能

#### 基类 `AsyncTTS2BaseExtension` 核心功能

TTS Extension基于`AsyncTTS2BaseExtension`基类构建，该基类继承自`AsyncExtension`，提供了TTS Extension的完整基础设施。

```python
class AsyncTTS2BaseExtension(AsyncExtension, ABC):
    """TTS Extension基类"""
    
    # 抽象方法 - 子类必须实现
    @abstractmethod
    async def request_tts(self, t: TTSTextInput) -> None:
        """处理TTS请求 - 子类必须实现
        - 从队列中接收文本输入
        - 调用TTS服务生成音频
        - 使用send_tts_audio_data()发送音频数据
        """
        
    @abstractmethod
    def vendor(self) -> str:
        """返回供应商名称 - 子类必须实现
        - 用于指标上报和错误追踪
        """
        
    @abstractmethod
    def synthesize_audio_sample_rate(self) -> int:
        """返回音频采样率 - 子类必须实现
        - 用于音频帧格式化和时长计算
        """
    
    # 可选重写方法
    async def cancel_tts(self) -> None:
        """取消TTS请求 - 子类可选重写
        - 收到flush请求时调用
        - 用于实现TTS特定的取消逻辑
        - 应该快速执行，避免阻塞主线程
        """
    
    # 已实现的完整功能
    
    # 生命周期管理
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        """Extension初始化 - 基类已实现
        - 初始化消息队列
        - 初始化指标计数器
        """
        
    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        """Extension启动 - 基类已实现
        - 启动消息队列处理任务
        """
        
    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        """Extension停止 - 基类已实现
        - 发送最后一批指标
        - 清空消息队列
        - 取消处理任务
        """
        
    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        """Extension销毁 - 基类已实现"""
    
    # 消息队列处理
    async def on_data(self, ten_env: AsyncTenEnv, data: Data) -> None:
        """处理接收到的数据 - 基类已实现
        - 处理tts_text_input：放入消息队列
        - 处理tts_flush：清空队列并取消当前任务
        - 使用锁机制防止并发问题
        """
        
    async def _process_input_queue(self, ten_env: AsyncTenEnv) -> None:
        """异步处理队列中的消息 - 基类已实现
        - 从队列中逐个取出消息
        - 调用request_tts()处理每个请求
        """
        
    async def _flush_input_items(self) -> None:
        """清空消息队列 - 基类已实现
        - 清空输入队列
        - 取消当前处理任务
        - 调用cancel_tts()执行TTS特定取消逻辑
        """
    
    # 音频发送方法
    async def send_tts_audio_data(self, audio_data: bytes, timestamp: int = 0) -> None:
        """发送音频数据 - 基类已实现
        - 自动处理不完整的音频帧（leftover_bytes）
        - 格式化音频帧（采样率、声道数、采样宽度）
        - 发送到下游Extension
        """
        
    async def send_tts_audio_start(self, request_id: str, turn_id: int = -1, 
                                   extra_metadata: dict | None = None) -> None:
        """发送音频开始事件 - 基类已实现
        - 通知下游Extension音频开始
        - 支持元数据传递
        """
        
    async def send_tts_audio_end(self, request_id: str, request_event_interval_ms: int, 
                                  request_total_audio_duration_ms: int, turn_id: int = -1, 
                                  reason: TTSAudioEndReason = TTSAudioEndReason.REQUEST_END,
                                  extra_metadata: dict | None = None) -> None:
        """发送音频结束事件 - 基类已实现
        - 通知下游Extension音频结束
        - 包含事件间隔时长和音频总时长
        - 支持不同的结束原因（REQUEST_END, INTERRUPTED等）
        - 清理请求元数据
        """
    
    # 指标上报方法
    async def send_tts_ttfb_metrics(self, request_id: str, ttfb_ms: int, 
                                    turn_id: int = -1, extra_metadata: dict | None = None) -> None:
        """发送TTFB指标 - 基类已实现
        - TTFB（Time To First Byte）指标上报
        - 支持额外元数据
        """
        
    async def send_usage_metrics(self, request_id: str = "", 
                                 extra_metadata: dict | None = None) -> None:
        """发送使用量指标 - 基类已实现
        - 输入字符数、输出字符数
        - 接收的音频时长
        - 总使用量统计
        """
        
    async def send_metrics(self, metrics: ModuleMetrics, request_id: str = "") -> None:
        """发送通用指标 - 基类已实现
        - 发送ModuleMetrics对象
        """
    
    # 错误处理
    async def send_tts_error(self, request_id: str | None, error: ModuleError,
                             turn_id: int = -1, extra_metadata: dict | None = None) -> None:
        """发送错误信息 - 基类已实现
        - 标准化的错误报告格式
        - 包含供应商信息、错误代码、错误消息
        - 支持元数据传递
        """
    
    # 辅助方法
    def synthesize_audio_channels(self) -> int:
        """返回音频声道数 - 基类已实现（默认1）"""
        return 1
    
    def synthesize_audio_sample_width(self) -> int:
        """返回音频采样宽度（字节） - 基类已实现（默认2，16-bit）"""
        return 2
    
    def get_uuid(self) -> str:
        """生成唯一标识符 - 基类已实现"""
    
    def update_metadata(self, request_id: str | None, metadata: dict | None) -> dict:
        """更新元数据 - 基类已实现
        - 合并请求元数据和额外元数据
        """
    
    # 指标辅助方法
    def metrics_add_output_characters(self, characters: int) -> None:
        """添加输出字符数 - 基类已实现"""
    
    def metrics_add_input_characters(self, characters: int) -> None:
        """添加输入字符数 - 基类已实现"""
    
    def metrics_add_recv_audio_chunks(self, chunks: bytes) -> None:
        """添加接收的音频块 - 基类已实现"""
    
    async def metrics_calculate_duration(self) -> None:
        """计算音频时长 - 基类已实现"""
    
    def metrics_reset(self) -> None:
        """重置指标计数器 - 基类已实现"""
```

##### 基类提供的完整功能

`AsyncTTS2BaseExtension`基类已经完整实现了以下功能，开发者无需重复实现：

1. **生命周期管理**
   * `on_init()`: 初始化消息队列和指标计数器
   * `on_start()`: 启动消息队列处理任务
   * `on_stop()`: 发送最后一批指标、清空队列、取消任务
   * `on_deinit()`: 销毁处理

2. **异步消息队列处理**
   * `on_data()`: 处理上游发送的`tts_text_input`和`tts_flush`数据
   * `_process_input_queue()`: 异步处理队列中的消息，调用`request_tts()`
   * `_flush_input_items()`: 收到flush后清空队列并取消当前任务
   * 使用锁机制防止并发问题，确保flush时不会继续接收新请求

3. **音频数据管理**
   * `send_tts_audio_data()`: 自动处理不完整音频帧，格式化并发送
   * `send_tts_audio_start()`: 发送音频开始事件
   * `send_tts_audio_end()`: 发送音频结束事件，包含时长和原因

4. **指标上报**
   * `send_tts_ttfb_metrics()`: TTFB指标上报
   * `send_usage_metrics()`: 使用量指标上报（字符数、音频时长等）
   * `send_metrics()`: 通用指标上报
   * 自动计算音频时长和统计使用量

5. **错误处理**
   * `send_tts_error()`: 标准化的错误报告机制
   * 包含供应商信息、错误代码、错误消息

6. **辅助功能**
   * 音频参数获取（声道数、采样宽度）
   * 元数据管理
   * UUID生成
   * 指标统计和重置

##### 子类需要实现的方法

开发者只需要实现以下3个抽象方法（以及可选的`cancel_tts()`）：

1. **`async def request_tts(t: TTSTextInput) -> None`**
   * 处理TTS请求的核心逻辑
   * 从配置或服务获取音频数据
   * 使用`send_tts_audio_data()`发送音频
   * 使用`send_tts_audio_start()`和`send_tts_audio_end()`发送事件
   * 使用`send_tts_ttfb_metrics()`上报TTFB指标
   * 使用`send_tts_error()`报告错误

2. **`def vendor() -> str`**
   * 返回供应商名称，用于指标和错误追踪

3. **`def synthesize_audio_sample_rate() -> int`**
   * 返回音频采样率，用于音频帧格式化

4. **`async def cancel_tts() -> None`**（可选）
   * 实现TTS特定的取消逻辑
   * 例如：关闭连接、停止请求等
   * 应该快速执行，避免阻塞

#### 基类 `AsyncTTS2HttpExtension` 核心功能

HTTP模式基于`AsyncTTS2HttpExtension`基类实现，该基类继承自`AsyncTTS2BaseExtension`，并提供了HTTP模式的完整实现。

```python
class AsyncTTS2HttpExtension(AsyncTTS2BaseExtension):
    """HTTP模式TTS Extension基类"""
    
    # 抽象方法 - 子类必须实现
    @abstractmethod
    async def create_config(self, config_json_str: str) -> AsyncTTS2HttpConfig:
        """从JSON字符串创建配置对象 - 子类必须实现"""
        
    @abstractmethod
    async def create_client(self, config: AsyncTTS2HttpConfig, ten_env: AsyncTenEnv) -> AsyncTTS2HttpClient:
        """创建客户端对象 - 子类必须实现"""
    
    @abstractmethod
    def vendor(self) -> str:
        """返回供应商名称 - 子类必须实现（继承自基类）"""
        
    @abstractmethod
    def synthesize_audio_sample_rate(self) -> int:
        """返回音频采样率 - 子类必须实现（继承自基类）"""

    async def request_tts(self, t: TTSTextInput) -> None:
        """处理TTS请求 - 基类已完整实现
        包含完整的请求处理逻辑：
        - 请求状态管理（request_id、turn_id等）
        - 客户端自动重连机制
        - PCMWriter管理（创建、清理）
        - 调用client.get()获取音频流
        - 音频数据处理和发送
        - TTFB计算和上报（首次音频块时）
        - 音频时长计算
        - 音频开始/结束事件发送
        - 错误处理和上报
        """
        
    def _calculate_audio_duration_ms(self) -> int:
        """计算音频时长（毫秒）- 基类已实现
        根据音频字节数、采样率、声道数计算
        """
```

##### 基类提供的完整功能

`AsyncTTS2HttpExtension`基类已经完整实现了以下功能，开发者无需重复实现：

1. **生命周期管理**
   * `on_init()`: 配置加载、验证和客户端创建
   * `on_stop()`: 资源清理
   * `on_deinit()`: 销毁处理

2. **请求处理逻辑**（在`request_tts()`中实现）
   * 请求状态跟踪（request\_id、turn\_id、first\_chunk等）
   * 客户端自动重连（client为None时自动重新创建）
   * PCMWriter生命周期管理（创建、清理）
   * 调用客户端获取音频流
   * 音频数据流式处理
   * TTFB指标计算和上报（首次音频块时）
   * 音频时长计算
   * 音频开始/结束事件发送
   * 错误处理和上报

3. **资源管理**
   * PCMWriter自动管理（按request\_id）
   * 客户端资源清理
   * 状态重置

##### 子类需要实现的方法

开发者只需要实现以下4个抽象方法：

1. **`create_config(config_json_str: str) -> AsyncTTS2HttpConfig`**
   * 从JSON字符串创建配置对象

2. **`create_client(config: AsyncTTS2HttpConfig, ten_env: AsyncTenEnv) -> AsyncTTS2HttpClient`**
   * 创建客户端对象

3. **`vendor() -> str`**
   * 返回供应商名称

4. **`synthesize_audio_sample_rate() -> int`**
   * 返回音频采样率

##### 配置接口 `AsyncTTS2HttpConfig`

配置类必须继承`AsyncTTS2HttpConfig`并实现以下抽象方法：

```python
class AsyncTTS2HttpConfig(BaseModel):
    """HTTP模式配置基类"""
    
    dump: bool = False
    dump_path: str = "/tmp"
    
    @abstractmethod
    def update_params(self) -> None:
        """更新配置参数 - 子类必须实现
        - 从params字典中提取配置项
        - 处理参数映射和转换
        - 删除黑名单参数
        """
        
    @abstractmethod
    def to_str(self, sensitive_handling: bool = True) -> str:
        """转换为字符串 - 子类必须实现
        - 支持敏感信息加密
        - 用于日志记录
        """
        
    @abstractmethod
    def validate(self) -> None:
        """验证配置 - 子类必须实现
        - 检查必需参数
        - 验证参数范围和格式
        """
```

##### 客户端接口 `AsyncTTS2HttpClient`

客户端类必须实现`AsyncTTS2HttpClient`接口：

```python
class AsyncTTS2HttpClient:
    """HTTP模式客户端接口"""
    
    @abstractmethod
    async def clean(self) -> None:
        """清理资源 - 子类必须实现
        - 清理HTTP客户端连接
        - 释放相关资源
        """
        
    @abstractmethod
    async def cancel(self) -> None:
        """取消当前请求 - 子类必须实现
        - 设置取消标志
        - 中断正在进行的请求
        """
        
    @abstractmethod
    async def get(
        self, text: str, request_id: str
    ) -> AsyncIterator[Tuple[bytes | None, TTS2HttpResponseEventType]]:
        """获取音频流 - 子类必须实现
        - 发送HTTP POST请求
        - 处理流式响应
        - 返回音频块和事件类型
        - 使用TTS2HttpResponseEventType枚举
        """
        
    @abstractmethod
    def get_extra_metadata(self) -> dict[str, Any]:
        """获取额外元数据 - 子类必须实现
        - 返回除透传metadata之外额外需要传递的内容
        - 用于TTFB指标上报
        - 如voice_id、model_id等
        """
```

#### 基类与子类职责

##### 详细说明

**1. AsyncTTS2BaseExtension（通用基类）**

**职责**：提供TTS Extension的基础设施，所有TTS Extension共享的功能。

**负责**：

* ✅ 消息队列的完整管理（接收、处理、flush）
* ✅ Extension生命周期管理（init、start、stop、deinit）
* ✅ 音频数据发送和格式化
* ✅ 指标上报（TTFB、使用量等）
* ✅ 错误处理和上报
* ✅ 元数据管理

**不负责**：

* ❌ 具体的TTS服务调用
* ❌ 配置加载（由模式基类或子类负责）
* ❌ 客户端管理（由模式基类负责）
* ❌ 厂商特定的逻辑

**2. AsyncTTS2HttpExtension（HTTP模式基类）**

**职责**：提供HTTP模式的完整实现，所有HTTP模式TTS Extension共享的逻辑。

**负责**：

* ✅ 配置加载和验证（调用子类的`create_config()`）
* ✅ 客户端创建和管理（调用子类的`create_client()`）
* ✅ HTTP请求处理逻辑（完整的`request_tts()`实现）
* ✅ TTFB计算和上报
* ✅ **PCMWriter生命周期管理（如启用dump功能）**
  * **初始化**：基类在`request_tts()`中自动检测新请求ID时初始化
  * **写入**：基类在处理音频流时自动写入
  * **销毁/刷新**：基类在请求完成时自动刷新
* ✅ 请求状态跟踪（request\_id、turn\_id等）
* ✅ 音频流处理（调用子类的`client.get()`）
* ✅ 错误处理和事件发送

**不负责**：

* ❌ 厂商特定的配置结构（由子类配置类实现）
* ❌ 厂商特定的HTTP通信逻辑（由子类客户端类实现）
* ❌ 供应商信息（由子类实现）

**3. VendorTTSExtension（子类实现）**

**职责**：实现厂商特定的逻辑，与具体TTS服务提供商交互。

根据继承路径的不同，子类的职责也不同：

**路径1：直接继承AsyncTTS2BaseExtension（WebSocket）**

**负责**：

* ✅ **完整的`request_tts()`实现**
  * 请求状态管理（request\_id、turn\_id等）
  * WebSocket连接管理或SDK调用
  * 音频流处理和发送
  * TTFB计算和上报
  * 音频开始/结束事件发送
  * 错误处理和上报
  * **PCMWriter管理（如启用dump功能）**
    * **初始化**：在`request_tts()`中检测到新的`request_id`时，创建`PCMWriter`实例并保存到字典中（按`request_id`索引），同时清理旧的PCMWriter
    * **写入**：每次收到音频数据块时，在调用`send_tts_audio_data()`之前，使用`PCMWriter.write()`写入音频数据
    * **销毁/刷新**：请求完成时（包括正常完成和中断），在发送`audio_end`事件之前，使用`PCMWriter.flush()`刷新缓冲区
* ✅ **客户端类实现**
  * WebSocket连接管理
  * 或SDK客户端封装
  * 资源清理
  * 请求取消
* ✅ **配置类实现**
  * 参数提取和映射
  * 日志输出格式
  * 参数验证
* ✅ **Extension类实现**
  * `vendor()`: 返回供应商名称
  * `synthesize_audio_sample_rate()`: 返回采样率
  * `cancel_tts()`: 可选实现取消逻辑

**不负责**：

* ❌ 消息队列管理（基类已实现）
* ❌ 音频数据发送（基类已实现）
* ❌ 指标上报（基类已实现）
* ❌ 错误处理格式（基类已实现）

**路径2：继承AsyncTTS2HttpExtension（HTTP模式）**

**负责**：

* ✅ **配置类实现（`VendorTTSConfig`）**
  * `update_params()`: 参数提取和映射
  * `to_str()`: 日志输出格式
  * `validate()`: 参数验证
* ✅ **客户端类实现（`VendorTTSClient`）**
  * `get()`: HTTP请求发送和响应处理
  * `clean()`: 资源清理
  * `cancel()`: 请求取消
  * `get_extra_metadata()`: 元数据获取
* ✅ **Extension类实现**
  * `create_config()`: 创建配置对象
  * `create_client()`: 创建客户端对象
  * `vendor()`: 返回供应商名称
  * `synthesize_audio_sample_rate()`: 返回采样率

**不负责**：

* ❌ 消息队列管理（基类已实现）
* ❌ 音频数据发送（基类已实现）
* ❌ 指标上报（基类已实现）
* ❌ 错误处理格式（基类已实现）
* ❌ **请求处理流程（HTTP模式基类已实现）**
* ❌ **PCMWriter管理（HTTP模式基类已实现）**

**注意**：两种路径都需要在配置类中支持`dump`和`dump_path`参数来启用PCM功能。

### TTS Extension 接口

#### 输入输出数据格式

TTS 标准接口（`tts-interface.json`）中除了属性配置外，还定义了输入和输出的数据格式规范：

**输入数据**：

* **文本输入** (`tts_text_input`): 从上游接收的文本数据流
* **刷新请求** (`tts_flush`): 取消当前请求并清空队列

**输出数据**：

* **文本结果** (`tts_text_result`): 包含时间戳的文本结果（当`enable_words=true`时）
* **音频开始** (`tts_audio_start`): 音频开始事件
* **音频结束** (`tts_audio_end`): 音频结束事件，包含时长统计
* **刷新完成** (`tts_flush_end`): 刷新处理完成的通知
* **错误信息** (`error`): 发生错误时的错误详情
* **性能指标** (`metrics`): TTFB、音频时长等性能数据
* **PCM音频帧** (`pcm_frame`): 向下游发送的音频数据流

详细的数据结构定义和字段说明请参考 `tts-interface.json` 文件。

#### 上游输入接口（data\_in）

<div className="fd-steps">
  <div className="fd-step">
    ##### tts\_text\_input - TTS文本输入

    **用途**：接收来自上游Extension（通常是LLM Extension）的文本输入，用于生成语音。

    **必需字段**：

    * `request_id` (string): 请求唯一标识符，用于追踪和关联请求
    * `text` (string): 需要转换为语音的文本内容

    **可选字段**：

    * `text_input_end` (bool): 标识当前文本输入是否结束
      * `true`: 表示当前轮次文本输入已完成。
      * `false`或不设置: 表示还有后续文本输入
    * `metadata` (object): 元数据信息
      * `session_id` (string): 会话ID，用于关联同一会话的多个请求
      * `turn_id` (int64): 轮次ID，用于标识对话轮次
      * 其他需要TTS Extension 透传给下游积木的内容

    **处理说明**：

    * Extension需要将接收到的文本放入请求队列
    * 支持流式文本输入，可以接收多个`tts_text_input`消息
    * 当`text_input_end=true`时，开始处理完整的文本输入进行TTS合成
  </div>

  <div className="fd-step">
    ##### tts\_flush - 刷新请求

    **用途**：取消当前正在处理的TTS请求，清空请求队列。

    **字段**：

    * `flush_id` (string): 刷新请求的唯一标识符
    * `metadata` (object): 元数据信息
      * `session_id` (string): 会话ID
      * 其他需要TTS Extension 在处理完flush之后透传给下游的内容

    **处理说明**：

    * 收到flush请求时，Extension必须：
      1. 清空输入队列中的所有待处理请求
      2. 取消当前正在进行的TTS合成请求
      3. 调用客户端的`cancel()`方法中断TTS服务调用
      4. 发送`tts_flush_end`响应，通知下游Extension刷新完成
  </div>
</div>

#### 下游输出接口（data\_out）

<div className="fd-steps">
  <div className="fd-step">
    ##### tts\_text\_result - TTS文本结果（字幕对齐用）

    **用途**：当`enable_words=true`时，输出包含时间戳的文本结果。

    **必需字段**：

    * `request_id` (string): 请求ID，与输入的`tts_text_input.request_id`对应
    * `text` (string): 合成音频对应的文本内容
    * `start_ms` (int64): 文本开始时间戳（毫秒）
    * `duration_ms` (int64): 文本持续时长（毫秒）
    * `words` (array): 含时间戳的文本数组
      * `word` (string): 词语内容
      * `start_ms` (int64): 词语开始时间戳（毫秒）
      * `duration_ms` (int64): 词语持续时长（毫秒）

    **可选字段**：

    * `text_result_end` (bool): 标识文本结果是否结束
    * `metadata` (object): 元数据信息
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他上游传入需要透传的内容

    **发送说明**：

    * 仅在`enable_words=true`时需要发送此消息
    * 如果供应商TTS服务支持词级别时间戳，需要在收到供应商响应时提取并发送
    * 如果不支持词级别时间戳，可以不发送此消息
  </div>

  <div className="fd-step">
    ##### tts\_flush\_end - 刷新结束

    **用途**：响应`tts_flush`请求，通知下游Extension刷新操作已完成。

    **字段**：

    * `flush_id` (string): 与输入的`tts_flush.flush_id`对应
    * `metadata` (object): 元数据信息, tts\_flush 带入的内容透传出去

    **发送说明**：

    * 在完成flush操作后必须发送，确保下游Extension知道刷新已完成
  </div>

  <div className="fd-step">
    ##### tts\_audio\_start - 音频开始事件

    **用途**：通知下游Extension音频数据开始发送。

    **字段**：

    * `request_id` (string): 请求ID，与输入的`tts_text_input.request_id`对应
    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他透传数据

    **发送说明**：

    * 在开始发送第一个音频数据块之前发送
    * 用于下游Extension进行音频播放准备
  </div>

  <div className="fd-step">
    ##### tts\_audio\_end - 音频结束事件

    **用途**：通知下游Extension音频数据发送完成。

    **必需字段**：

    * `request_id` (string): 请求ID，与输入的`tts_text_input.request_id`对应
    * `request_event_interval_ms` (int64): 请求事件间隔时长（毫秒）
      * 从接收第一个`tts_text_input`到发送第一个音频数据块的时间间隔
    * `request_total_audio_duration_ms` (int64): 请求总音频时长（毫秒）
      * 整个请求生成的音频总时长
    * `reason` (int64): 结束原因
      * `1`: 正常结束（REQUEST\_END）
      * `2`: 中断（INTERRUPTED）
      * 其他值根据实际需求定义

    **可选字段**：

    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID

    **发送说明**：

    * 在所有音频数据块发送完成后发送
    * 必须准确计算`request_event_interval_ms`和`request_total_audio_duration_ms`
    * 如果请求被中断（收到flush），`reason`应设置为中断状态
  </div>

  <div className="fd-step">
    ##### error - 错误信息

    **用途**：向上游或下游Extension报告错误信息。

    **必需字段**：

    * `module` (string): 模块名称，必须为`"tts"`
    * `code` (int64): 错误代码
      * `-1000`: 致命错误（FATAL\_ERROR）
      * `1000`: 非致命错误（NON\_FATAL\_ERROR）
    * `message` (string): 错误描述信息

    **可选字段**：

    * `id` (string): 错误唯一标识符
    * `vendor_info` (object): 供应商特定的错误信息
      * `vendor` (string): 供应商名称
      * `code` (string): 供应商错误代码
      * `message` (string): 供应商错误消息
    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他需要透传的参数

    **发送说明**：

    * 当TTS合成过程中出现错误时必须发送
    * `NON_FATAL_ERROR` 是指无法恢复的错误，通常是基本参数问题。
    * `FATAL_ERROR` 是指可以恢复的错误，通常表示当前请求有问题，但不影响后续请求
  </div>

  <div className="fd-step">
    ##### metrics - 指标数据

    **用途**：上报TTS Extension的性能和使用量指标。

    **必需字段**：

    * `module` (string): 模块名称，通常为`"tts"`
    * `vendor` (string): 供应商名称
    * `metrics` (object): 指标数据对象
      * 可以包含TTFB、音频时长、字符数、请求数等指标

    **可选字段**：

    * `id` (string): 指标唯一标识符
    * `metadata` (object): 元数据信息，来自于tts\_text\_input（每个reqeust id共用一个）
      * `session_id` (string): 会话ID
      * `turn_id` (int64): 轮次ID
      * 其他需要透传的内容

    **发送说明**：

    * 用于监控和统计TTS Extension的使用情况
    * 通常在请求完成时上报
  </div>
</div>

#### 音频输出接口（audio\_frame\_out）

##### pcm\_frame - PCM音频帧

**用途**：向下游Extension发送PCM格式的音频数据。

**字段**：

* `metadata` (object): 元数据信息
  * `session_id` (string): 会话ID
  * `turn_id` (int64): 轮次ID

**音频格式要求**：

* 格式：PCM（脉冲编码调制）
* 采样率：由`synthesize_audio_sample_rate()`方法返回（常见值：16000、24000、44100、48000 Hz）
* 声道数：由`synthesize_audio_channels()`方法返回（默认1，单声道）
* 采样宽度：由`synthesize_audio_sample_width()`方法返回（默认2字节，16-bit）
* 字节序：小端序（Little-Endian）

**发送说明**：

* 音频数据以帧为单位流式发送
* 基类会自动处理不完整的音频帧（leftover\_bytes）
* 如果启用了`dump`功能，音频数据会同时保存到文件

#### 接口调用流程示例

##### 正常请求流程

```
上游Extension                     TTS Extension                    下游Extension                         TTS厂商服务器
     |                                  |                                 |                                 |
     |---- tts_text_input ------------->|                                 |                                 |
     |    (text="Hello")                |                                 |                                 |
     |                                  |-------------------------------调用TTS服务 ------------------------->|
     |                                  |                                 |                                 |
     |                                  |<------------------------------接收音频数据--------------------------|
     |                                  |                                 |                                 |
     |                                  |-----tts_ttfb(metrics)---------->|                                 |
     |                                  |                                 |                                 |
     |                                  |---- tts_audio_start  ---------->|                                 |
     |                                  |                                 |                                 |
     |                                  |---- pcm_frame ----------------->|                                 |
     |                                  |---- pcm_frame ----------------->|                                 |
     |                                  |---- pcm_frame ----------------->|                                 |
     |                                  |    ...                          |                                 |
     |                                  |---- tts_audio_end ------------->|                                 |
     |                                  |                                 |                                 |
     |                                  |---- input,output metrics ------>|                                 |
```

##### Flush请求流程

```
上游Extension                     TTS Extension                    下游Extension
     |                                  |                                 |
     |---- tts_flush ------------------>|                                 |
     |                                  | [内部处理]                       |
     |                                  |   1. 取消当前请求                 |
     |                                  |   2. 清空队列                    |
     |                                  |   3. 调用客户端cancel()           |
     |                                  |                                 |
     |                                  |---- tts_flush_end ------------->|
```

### manifest.json 内容

`manifest.json` 文件定义了 TTS Extension 的元数据、依赖关系、API 接口和属性声明。

#### 基本结构

```json title="manifest.json"
{
  "type": "extension",
  "name": "vendor_tts_python",
  "version": "0.1.0"
}
```

**关键字段说明**：

* `type`: 必须是 `"extension"`
* `name`: Extension 的唯一标识符，建议使用 `{vendor}_tts_python` 格式
* `version`: Extension 版本号，遵循语义化版本规范，每次更新代码都需要更新版本号，这样才能发不到TEN Extension Store

#### 依赖声明

```json
"dependencies": [
  {
    "type": "system",
    "name": "ten_runtime_python",
    "version": "0.11"
  },
  {
    "type": "system",
    "name": "ten_ai_base",
    "version": "0.7"
  }
]
```

**必需依赖**：

* `ten_runtime_python`: TEN Framework Python 运行时环境
* `ten_ai_base`: TEN AI Base 系统包，提供 TTS 基类和接口定义

#### 文件包含配置

```json
"package": {
  "include": [
    "manifest.json",
    "property.json",
    "BUILD.gn",
    "**.tent",
    "**.py",
    "README.md",
    "requirements.txt"
  ]
}
```

定义 Extension 打包时需要包含的文件，使用 glob 模式匹配。

#### API 接口配置

**1. Interface 继承**

```json
"api": {
  "interface": [
    {
      "import_uri": "../../system/ten_ai_base/api/tts-interface.json"
    }
  ]
}
```

**说明**：

* 必须继承 `ten_ai_base` 系统包下的标准 TTS 接口
* 该接口定义了所有 TTS Extension 必须遵循的标准属性：
  * `dump`: 布尔值，配置是否开启音频 dump
  * `dump_path`: 字符串，音频 dump 的存储路径

**2. Property 声明**

```json
"api": {
  "property": {
    "properties": {
      "dump": {
          "type": "bool"
      },
      "dump_path": {
          "type": "string"
      },
      "params": {
        "type": "object",
        "properties": {
             ...
        }
      }
    }
  }
}
```

**说明**：

* 在 `api.property.properties` 中声明 Extension 特有的配置属性
* `params` TTS 厂商使用的参数，罗列TTS工作的最小集参数，不需要全部罗列：

**Elvenlabs TTS完整示例**：

```json title="manifest.json"
{
  "type": "extension",
  "name": "vendor_tts_python",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "0.11"
    },
    {
      "type": "system",
      "name": "ten_ai_base",
      "version": "0.7"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "BUILD.gn",
      "**.tent",
      "**.py",
      "README.md",
      "requirements.txt"
    ]
  },
  "api": {
    "interface": [
      {
        "import_uri": "../../system/ten_ai_base/api/tts-interface.json"
      }
    ],
    "property": {
      "properties": {
        "params": {
          "type": "object",
          "properties": {
            "key": {
              "type": "string"
            },
            "model_id": {
              "type": "string"
            },
            "voice_id": {
              "type": "string"
            },
            "output_format": {
              "type": "string"
            }
          }
        },
        "dump": {
          "type": "bool"
        },
        "dump_path": {
          "type": "string"
        }
      }
    }
  }
}
```

### property.json内容

在 `property.json` 中提供可以使TTS工作的最小集默认配置：

```json title="property.json"
{
  "params": {
    "api_key": "your_tts_api_key_here",
    "voice_id": "default_voice",
    "model": "default_model",
    "sample_rate": "24000"
  },
  "extra_params": {
    "extra_key": "extra_value"
  },
  "dump": false,
  "dump_path": "/tmp/tts_audio_dump"
}
```

其中dump是指开启音频dump功能，TTS生成的所有音频都讲保存在文件里，同一个request id 的文本将按顺序保存在同一个文件。不同的request id文本保存在不同的文件。dump\_path是文件保存的位置。
params里存放的是可以透传给TTS厂商的参数。
具体内容和参数命名需要参考TTS厂商官网。
如果有其他一些定制化参数但又不是TTS厂商官方定义的，可以统一放在extra\_params里面。extra\_params的值跟params的值一样是json对象

### 配置管理设计

#### 设计配置类

创建灵活的配置类，支持必填参数和可选透传参数：

```python title="config.py"
from pydantic import BaseModel
from typing import Dict, Optional

class MyTTSConfig(BaseModel):
    # All TTS Vendor parameters are in params, including required and optional parameters
    params: Dict[str, Optional[str]] = {}

    #Not TTS Vendor parameters, but used in this TTS Extension
    extra_params: Dict[str, Optional[str]] = {}
    
    # Audio dump related configuration - standard implementation for all TTS extensions
    dump: bool = False
    dump_path: Optional[str] = None
```

#### 读取扩展配置

在 `on_init` 阶段读取和初始化配置：

```python title="extension.py"
from ten_ai_base.const import LOG_CATEGORY_KEY_POINT, LOG_CATEGORY_VENDOR
from ten_ai_base.message import ModuleError, ModuleErrorCode

@override
async def on_init(self, ten_env: AsyncTenEnv) -> None:
    await super().on_init(ten_env)
    
    # Read complete extension configuration
    config_json, _ = await ten_env.get_property_to_json("")
    
    try:
        # Deserialize configuration to config class instance
        self.config = MyTTSConfig.model_validate_json(config_json)
        
        # Print configuration information (sensitive information masked)
        ten_env.log_info(
            f"config: {self.config.to_json(sensitive_handling=True)}",
            category=LOG_CATEGORY_KEY_POINT,
        )
            
    except Exception as e:
        ten_env.log_error(
            f"invalid property: {e}",
            category=LOG_CATEGORY_KEY_POINT
        )
        # Use default configuration when configuration error occurs
        self.config = MyTTSConfig.model_validate_json("{}")
        # Send fatal error
        await self.send_tts_error(
            ModuleError(
                module=MODULE_NAME_TTS,
                code=ModuleErrorCode.FATAL_ERROR.value,
                message=str(e),
            ),
        )
```

#### 配置敏感信息脱敏

为配置类添加脱敏方法，保护敏感信息：

```python title="config.py"
from ten_ai_base.utils import encrypt

class MyTTSConfig(BaseModel):
    params: Dict[str, Optional[str]] = {}
    dump: bool = False
    dump_path: Optional[str] = None
    
    def to_json(self, sensitive_handling: bool = False) -> str:
        """
        Serialize configuration to JSON, support sensitive information masking
        
        Args:
            sensitive_handling: Whether to mask sensitive information
        """
        if not sensitive_handling:
            return self.model_dump_json()
        
        # Deep copy configuration object
        config = self.model_copy(deep=True)
        
        # Mask sensitive fields in params
        if config.params:
            encrypted_params = {}
            for key, value in config.params.items():
                # Encrypt fields containing sensitive words like 'key', 'token', 'secret'
                if (key in ['api_key', 'key', 'token', 'secret', 'password'] 
                    and isinstance(value, str) and value):
                    encrypted_params[key] = encrypt(value)
                else:
                    encrypted_params[key] = value
            config.params = encrypted_params
            
        return config.model_dump_json()
```

#### 配置功能最佳实践

1. **类型安全**：使用pydantic的类型验证，确保参数类型正确
2. **必需参数检查**：在`validate()`方法中检查必需参数
3. **参数范围验证**：验证采样率、声道数等参数的有效范围
4. **敏感信息保护**：使用`to_str()`方法的安全模式处理敏感信息
5. **默认值设置**：为所有最小集参数提供合理的默认值
6. **错误信息清晰**：提供清晰的错误信息，便于调试

### 必须实现的功能

**注意**：除了以下功能实现外，还需要实现必要的日志记录。详细要求请参考 [日志规范](#日志规范)。

<div className="fd-steps">
  <div className="fd-step">
    #### 初始化方法

    * **`__init__(config, ten_env, error_callback)`**：初始化客户端，接收配置对象、环境对象和错误回调函数
    * **`_initialize_client()`**：初始化具体的客户端连接（WebSocket连接、HTTP客户端、SDK客户端等）
  </div>

  <div className="fd-step">
    #### 核心请求处理

    **WebSocket模式：**

    * **`async def get(text, request_id) -> AsyncIterator[tuple[bytes | None, int, int | None]]`**：处理TTS请求，返回异步迭代器，每次yield音频数据块、事件类型和TTFB时间（首次音频块时）
      * 返回值：`(audio_data, event_type, ttfb_ms)`
      * 事件类型：`EVENT_TTS_RESPONSE`（音频数据）、`EVENT_TTS_REQUEST_END`（请求结束）、`EVENT_TTS_ERROR`（错误）、`EVENT_TTS_INVALID_KEY_ERROR`（认证错误）

    **HTTP模式：**

    * **`async def get(text, request_id) -> AsyncIterator[Tuple[bytes | None, TTS2HttpResponseEventType]]`**：处理TTS请求，返回异步迭代器，使用`TTS2HttpResponseEventType`枚举
      * 返回值：`(audio_data, event_type)`
      * 事件类型：`TTS2HttpResponseEventType.RESPONSE`、`TTS2HttpResponseEventType.END`、`TTS2HttpResponseEventType.ERROR`、`TTS2HttpResponseEventType.INVALID_KEY_ERROR`、`TTS2HttpResponseEventType.FLUSH`
  </div>

  <div className="fd-step">
    #### 抽象方法（用于解耦厂商特定逻辑）

    **WebSocket模式：**

    * **`_get_websocket_uri() -> str`**：获取WebSocket连接URI
    * **`_create_request_data(text) -> dict`**：创建请求数据格式
    * **`_parse_response(data) -> tuple`**：解析响应数据，返回音频数据和事件类型
    * **`_receive_responses() -> AsyncIterator`**：接收WebSocket响应流

    **HTTP模式（继承`AsyncTTS2HttpClient`时）：**

    * **`_get_api_endpoint() -> str`**：获取API端点URL
    * **`_create_headers() -> dict`**：创建HTTP请求头（包含认证信息）
    * **`_create_request_data(text) -> dict`**：创建HTTP请求体数据
    * **`_is_authentication_error(error_message) -> bool`**：判断是否为认证错误

    **SDK模式：**

    * **`_parse_credentials()`**：解析认证信息（证书文件、服务账号等）
    * **`_create_streaming_config()`**：创建流式配置对象
    * **`_create_request_generator(text)`**：创建请求生成器（第一个请求包含配置，后续发送文本）
    * **`_call_streaming_api(request_generator)`**：调用SDK的流式API
    * **`_extract_audio_content(response)`**：从SDK响应中提取音频数据
  </div>

  <div className="fd-step">
    #### 资源管理方法

    * **`async def stop()`**：停止客户端，关闭连接和清理资源
    * **`def cancel()`**：取消当前请求，设置取消标志，清理队列和关闭连接
    * **`async def clean()`**（HTTP模式）：清理HTTP客户端资源
    * **`async def reset()`**（WebSocket/SDK模式，可选）：重置客户端连接
  </div>

  <div className="fd-step">
    #### 辅助方法

    **HTTP模式：**

    * **`def get_extra_metadata() -> dict[str, Any]`**：返回供应商特定的元数据（如voice\_id、model\_id），用于TTFB指标上报

    **WebSocket/SDK模式（可选）：**

    * **`async def start()`**：启动客户端连接（WebSocket单向流模式中，连接已在`__init__`中启动）
    * **`_process_ws_exception(exception)`**：处理WebSocket异常
    * **`_await_connection_tasks()`**：等待连接任务完成
  </div>
</div>

### 日志规范

TTS Extension 必须实现以下日志，用于调试、监控和问题排查。所有日志应使用 `ten_env` 提供的日志方法，并指定正确的日志级别和分类。

#### 日志分类

TTS Extension 使用以下日志分类：

* **`LOG_CATEGORY_KEY_POINT`**：Extension 本身的关键事件，如配置加载、文本过滤等
* **`LOG_CATEGORY_VENDOR`**：与供应商相关的日志，如供应商错误、状态变化、请求响应等

#### 基类已实现的日志

以下日志已在基类 `AsyncTTS2BaseExtension` 中实现，子类**不需要**重复实现：

<div className="fd-steps">
  <div className="fd-step">
    ##### 文本输入相关日志

    **收到 TTS 文本输入** (`get_tts_text_input`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`get tts_text_input: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"get tts_text_input: {tts_text_input_object}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### Flush 相关日志

    **收到 flush** (`receive tts flush`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`receive tts flush`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        "receive tts flush ",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **发送 flush\_end** (`send tts_flush_end`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`send tts_flush_end`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        "send tts_flush_end",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### 错误信息日志

    **发送错误信息** (`tts_error`)

    **日志级别**：`error`

    **日志分类**：`key_point`

    **日志格式**：`tts_error: {}`

    **基类实现**：

    ```python
    self.ten_env.log_error(
        f"tts_error: msg: {error_msg}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### 音频事件日志

    **音频开始** (`tts_audio_start`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_audio_start: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_audio_start: {tts_audio_start} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **音频结束** (`tts_audio_end`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_audio_end: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_audio_end: {tts_audio_end} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```
  </div>

  <div className="fd-step">
    ##### 指标日志

    **TTFB 指标** (`tts_ttfb`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_ttfb: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_ttfb: {ttfb} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **Metrics 指标** (`tts_metrics`)

    **日志级别**：`info`

    **日志分类**：`key_point`

    **日志格式**：`tts_metrics: {}`

    **基类实现**：

    ```python
    self.ten_env.log_info(
        f"tts_metrics: {tts_metrics} of request_id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **注意**：以上日志均由基类 `AsyncTTS2BaseExtension` 自动处理，子类无需重复实现。开发者只需关注下面"必须实现的日志"部分。
  </div>
</div>

#### 必须实现的日志

<div className="fd-steps">
  <div className="fd-step">
    ##### 配置参数日志

    **类型**：配置参数

    **打印时机及内容**：在 `on_init` 回调里读取 `property` 并反序列化成 `config` 结构体后，加密打印 `config`

    **日志级别**：`Info`

    **日志分类**：`key_point`

    **日志格式**：`config: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_KEY_POINT

    ten_env.log_info(
        f"config: {self.config.to_str(sensitive_handling=True)}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **说明**：

    * 必须使用 `sensitive_handling=True` 对敏感信息进行脱敏处理
    * 配置信息应在 Extension 初始化时记录，便于问题排查
  </div>

  <div className="fd-step">
    ##### 供应商相关日志

    ###### 2.1 vendor\_error - 供应商错误日志

    **类型**：供应商相关

    **打印时机及内容**：当收到供应商返回的错误结果时，打印原始错误内容

    **日志级别**：`error`

    **日志分类**：`vendor`

    **日志格式**：`vendor_error: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_error(
        f"vendor_error: code: {code} reason: {cancellation_details.reason}, error_details: {cancellation_details.error_details}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 当供应商返回错误时，必须记录详细的错误信息
    * 包括错误代码、原因和详细错误信息

    ###### 2.2 vendor\_status - 供应商状态变化日志

    **类型**：供应商相关

    **打印时机及内容**：客户端与供应商服务器之间的状态变化

    **日志级别**：`debug`

    **日志分类**：`vendor`

    **日志格式**：`vendor_status: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_debug(
        f"vendor_status: connected to: {url}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 记录与供应商服务器的连接状态变化
    * 包括连接建立、断开、重连等状态

    ###### 2.3 send\_text\_to\_tts\_server - 发送文本到TTS服务器日志

    **类型**：供应商相关

    **打印时机及内容**：发送到 TTS 服务器的文本

    **日志级别**：`debug`

    **日志分类**：`vendor`

    **日志格式**：`send_text_to_tts_server`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_debug(
        f"send_text_to_tts_server: {text} of request_id: {request_id}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 记录发送到 TTS 服务器的文本内容和对应的 `request_id`
    * 用于追踪请求流程

    ###### 2.4 receive\_audio - 接收音频日志

    **类型**：供应商相关

    **打印时机及内容**：接收到的音频

    **日志级别**：`debug`

    **日志分类**：`vendor`

    **日志格式**：`receive_audio: {}`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_VENDOR

    self.ten_env.log_debug(
        f"receive_audio: duration: {ms} of request id: {request_id}",
        category=LOG_CATEGORY_VENDOR,
    )
    ```

    **说明**：

    * 记录接收到的音频信息，包括音频时长和对应的 `request_id`
    * 用于监控音频接收情况
  </div>

  <div className="fd-step">
    ##### Extension 本身的关键日志

    ###### 3.1 skip\_tts\_text\_input - 跳过文本输入日志

    **类型**：过滤不发送的内容

    **打印时机及内容**：当某些文本输入被过滤不发送时

    **日志级别**：`debug`

    **日志分类**：`key_point`

    **日志格式**：`skip_tts_text_input`

    **示例代码**：

    ```python
    from ten_ai_base.const import LOG_CATEGORY_KEY_POINT

    self.ten_env.log_debug(
        f"skip_tts_text_input: {text} of request id: {request_id}",
        category=LOG_CATEGORY_KEY_POINT,
    )
    ```

    **说明**：

    * 当 TTS 厂商接收某些内容会有特殊行为需要避免发送时，应记录此日志
    * 记录被跳过的文本内容和对应的 `request_id`
  </div>
</div>

#### 日志最佳实践

1. **敏感信息保护**：配置参数等包含敏感信息的日志必须使用 `sensitive_handling=True` 进行脱敏
2. **日志级别选择**：
   * `error`：用于错误信息，如供应商返回的错误
   * `info`：用于关键信息，如配置加载
   * `debug`：用于调试信息，如状态变化、请求响应等
3. **日志分类使用**：
   * `LOG_CATEGORY_KEY_POINT`：Extension 本身的关键事件
   * `LOG_CATEGORY_VENDOR`：与供应商相关的所有日志
4. **日志格式统一**：使用统一的日志格式，便于日志分析和问题排查
5. **包含关键信息**：日志中应包含 `request_id` 等关键信息，便于追踪请求流程


# 创建 Hello World 扩展 (/cn/docs/ten_agent_examples/extension_dev/create_a_hello_world_extension)

在本章中，我们将一步步创建一个 `Hello World` 扩展，支持 **Python、Go 和 C++**。你可以根据喜好选择任意语言进行实践。现在，让我们开始吧。

## 前置条件

在开始之前，你需要熟悉[前文的基础内容](/docs/ten_agent/getting_started)。特别是，需要掌握如何使用 `docker compose up` 并了解后台启动的服务。

<div className="fd-steps">
  <div className="fd-step">
    ## 启动服务器

    首先，运行以下命令来启动相关服务：

    <Callout>
      如果标题为 `Terminal`，表示命令在本地运行。\
      如果标题为 `Bash`，表示命令在 Docker 容器内运行。
    </Callout>

    ```bash title="Terminal"
    docker compose up -d
    ```

    执行后，你应当能看到类似如下的输出：

    ```bash title="Terminal"
    Attaching to ten_agent_dev, ten_agent_playground
    ten_agent_dev         | cd agents && tman designer
    ten_agent_dev         | :-)  Starting server at http://0.0.0.0:49483
    ten_agent_playground  |   ▲ Next.js 14.2.4
    ten_agent_playground  |   - Local:        http://localhost:3000
    ten_agent_playground  |   - Network:      http://0.0.0.0:3000
    ten_agent_playground  |
    ten_agent_playground  |  ✓ Starting...
    ten_agent_playground  |  ✓ Ready in 429ms
    ...
    ```

    此时，以下服务已启动并运行：

    * `ten_agent_dev`：`http://0.0.0.0:49483`（TMAN Designer）
    * `ten_agent_playground`：`http://localhost:3000`（TEN Agent Playground）
  </div>

  <div className="fd-step">
    ## 进入 Docker 容器

    为了在隔离环境中操作，执行以下命令：

    ```bash title="Terminal"
    docker exec -it ten_agent_dev bash
    ```
  </div>

  <div className="fd-step">
    ## 创建 hello world 扩展

    运行如下命令，即可在 Python、Go 或 C++ 中创建一个名为 `hello_world` 的扩展。

    import { Tab, Tabs } from 'fumadocs-ui/components/tabs'

    <Tabs items={['Python', 'Go', 'C++']} defaultIndex={0}>
      <Tab>
        ```bash title="Bash"
        cd /app/agents/ten_packages/extension
        tman create extension hello_world --template=default_async_extension_python@0.10 --template-data class_name_prefix=HelloWorld
        ```
      </Tab>

      <Tab>
        ```bash title="Bash"
        cd /app/agents/ten_packages/extension
        tman create extension hello_world --template=default_extension_go@0.10 --template-data class_name_prefix=HelloWorld
        ```
      </Tab>

      <Tab>
        ```bash title="Bash"
        cd /app/agents/ten_packages/extension
        tman create extension hello_world --template=default_extension_cpp@0.10 --template-data class_name_prefix=HelloWorld
        ```
      </Tab>
    </Tabs>

    执行成功后，你将看到类似日志：

    ```bash title="Bash"
    Package 'extension:hello_world' created successfully in '/app' in 3 seconds.
    ```
  </div>

  <div className="fd-step">
    ## 将扩展节点添加到图中

    打开 **TMAN Designer** 并选择需要修改的图：

    1. 在画布上右键，选择 **Add Node**。
    2. 在弹窗的 **Addon** 字段中，选择刚创建的 `hello_world` 扩展。
    3. 点击 **Add Node** 完成操作。
    4. 画布中应当出现 `hello_world` 节点。
  </div>

  <div className="fd-step">
    ## 在图中尝试消息连接

    扩展可以在图中与其他节点进行消息收发。我们来创建一个简单的消息连接：
    让 `hello_world` 节点周期性发送 `"Hello, World!"`（消息名为 `output_text`），并接收名为 `input_text` 的消息。

    ### 1). 在 `manifest.json` 中声明消息协议

    在扩展的 `manifest.json` 文件中，声明该扩展所使用的消息协议。在 `api` 字段添加如下内容：

    ```json
    ...
      "api": {
        "data_in": [
          {
            "name": "input_text",
            "property": {
              "properties": {
                "text": {
                  "type": "string"
                }
              }
            }
          }
        ],
        "data_out": [
          {
            "name": "output_text",
            "property": {
              "properties": {
                "text": {
                  "type": "string"
                }
              }
            }
          }
        ]
      },
    ...
    ```

    在 `extension.py` 中编写一个异步循环，每隔 5 秒发送一次消息：

    ```python
        async def on_start(self, ten_env: AsyncTenEnv) -> None:
            ten_env.log(LogLevel.DEBUG, "on_start")
            asyncio.create_task(self._send_hello(ten_env))

        async def _send_hello(self, ten_env: AsyncTenEnv) -> None:
            while True:
                await asyncio.sleep(5)
                data = Data.create("output_text")
                data.set_property_string("text", "Hello, World!")
                await ten_env.send_data(data)
    ```

    保存后，在 Designer 菜单中选择 **App -> Reload All Apps** 重新加载扩展配置。

    <Callout title="注意">
      修改 `manifest.json` 后必须执行重新加载操作，否则更改不会生效。
    </Callout>

    ### 2). 添加从 `hello_world` 到 `main` 的连接

    在 Designer 中右键点击 `hello_world` 节点，选择 **Add Connection from hello\_world**。
    在对话框中设置消息类型为 `data`，消息名为 `output_text`，目标节点选择 `main`，最后点击 **Add Connection**。
    此时，`hello_world` 与 `main` 的连接已建立。

    ### 3). 在 `main` 中添加代码处理输入

    打开 `main.py`，新增如下代码：

    ```python
        async def on_data(self, ten_env: AsyncTenEnv, data: Data):
            name = data.get_name()
            if name == "output_text":
                text, _ = data.get_property_string("text")
                if text:
                    ten_env.log_info(f"[MainControlExtension] Received input_text: {text}")
                    return

            await self.agent.on_data(data)
    ```

    ### 4). 从 `main` 回复消息给 `hello_world`

    建议在 `main` 中使用 `setDests` API 发送消息（因其负责整体会话上下文，最适合掌握目标节点）。

    ```python
        async def on_data(self, ten_env: AsyncTenEnv, data: Data):
            name = data.get_name()
            if name == "output_text":
                text, _ = data.get_property_string("text")
                if text:
                    source_name = data.get_source().extension_name
                    ten_env.log_info(f"[MainControlExtension] Received input_text: {text} from: {source_name}")

                    reply = Data.create("input_text")
                    reply.set_property_string("text", f"Echo: {text}")
                    reply.set_dests([Loc(
                        app_uri="",
                        graph_id="",
                        extension_name=source_name
                    )])
                    await ten_env.send_data(reply)
                    return

            await self.agent.on_data(data)
    ```

    修改 `hello_world`，使其接收回复并打印到日志：

    ```python
        async def on_data(self, ten_env: AsyncTenEnv, data: Data) -> None:
            name = data.get_name()
            if name == "input_text":
                text, _ = data.get_property_string("text")
                if text:
                    ten_env.log_info(f"[HelloWorldExtension] Received input_text: {text}")
                    return
    ```
  </div>

  <div className="fd-step">
    ## 验证扩展

    使用 TMAN Designer 启动应用，并在日志中确认是否打印以下内容：

    ```
    [MainControlExtension] Received input_text: Hello, World!
    [HelloWorldExtension] Received input_text: Echo: Hello, World!
    ```

    至此，你已成功完成第一个 `hello_world` 扩展的创建。 🎉
  </div>
</div>


# ASR Extension 开发完整指南 (/cn/docs/ten_agent_examples/extension_dev/create_asr_extension)

# ASR Extension 开发完整指南

本教程涵盖 ASR Extension 从创建、开发、测试到发布的完整流程。

## 使用指南

* **Basic 部分**：实现基本功能，让 ASR Extension 能正常工作并通过端到端验证
* **Advanced 部分**：实现生产级特性（自动重连、finalize机制、规范化日志、音频dump等），完善测试并发布到商店

根据你的需求选择阅读相应部分。

## 前置条件

* 了解 TEN Extension 的开发、测试流程，参考 [TEN 扩展开发完整指南](/docs/ten_framework/development/how_to_develop_with_ext)
* 掌握 Python 异步编程（`asyncio`、`async/await`）
* 安装好 tman 命令行工具，并熟悉其基本使用
* 准备好 ASR 服务商的 API 密钥（用于测试）

## 目录

### Part 1: Basic - 实现基础功能

1. [ASR Extension 简介](#1-asr-extension-简介)
2. [创建项目](#2-创建项目)
3. [理解接口规范](#3-理解接口规范)
4. [实现核心功能](#4-实现核心功能)
5. [测试](#5-测试)
6. [Basic 开发自检清单](#6-basic-开发自检清单)

### Part 2: Advanced - 生产级质量

7. [自动重连机制](#7-自动重连机制)
8. [优化 Finalize 机制](#8-优化-finalize-机制)
9. [规范化日志](#9-规范化日志)
10. [错误上报规范](#10-错误上报规范)
11. [音频 Dump 功能](#11-音频-dump-功能)
12. [音频缓冲策略](#12-音频缓冲策略)
13. [完善测试](#13-完善测试)
14. [Advanced 开发自检清单](#14-advanced-开发自检清单)

### Part 3: 发布与贡献

15. [发布到 TEN Store](#15-发布到-ten-store)
16. [适配其他 ASR 服务](#16-适配其他-asr-服务)

### 附录

* [A. ASR Interface 规范](#a-asr-interface-规范)
* [B. 基类方法参考](#b-基类方法参考)
* [C. 完整示例代码](#c-完整示例代码)
* [D. 调试工具](#d-调试工具)

***

# Part 1: Basic - 实现基础功能

<div className="fd-steps">
  <div className="fd-step">
    ## ASR Extension 简介

    ASR Extension 是 TEN Framework 的标准积木组件之一，负责**实时地**将音频流转写为对应语言的文字。

    ### 在对话流中的位置

    ```
    [RTC Extension] ──音频流──> [ASR Extension] ──文字流──> [LLM Extension]
    ```
  </div>

  <div className="fd-step">
    ## 创建项目

    ### 2.0 准备工作

    <Callout type="info">
      如果你还没有 TEN Framework 仓库，请先克隆：

      ```bash
      git clone https://github.com/TEN-framework/ten-framework.git
      cd ten-framework
      ```

      本教程假设你在 TEN Framework 仓库根目录下进行开发。
    </Callout>

    ### 2.1 使用模板创建

    使用 tman 命令行工具，从 ASR 扩展模板创建项目：

    ```bash
    cd ai_agents/agents/ten_packages/extension
    tman create extension my_asr_extension --template default_asr_python --template-data class_name_prefix=MyAsr
    ```

    **命令参数说明**：

    * `extension my_asr_extension`：创建一个 extension，目录名和插件名为 `my_asr_extension`
    * `--template default_asr_python`：使用 ASR Python 扩展模板
    * `--template-data class_name_prefix=MyAsr`：设置 Python Extension class 的类名前缀为 `MyAsr`（生成的类名为 `MyAsrExtension`）

    ### 2.2 安装TEN packages的依赖

    ```bash
    cd my_asr_extension
    tman install --standalone
    ```

    <Callout type="tip">
      这条命令使用 tman 工具的依赖计算和下载能力，根据 `manifest.json` 中声明的依赖计算出依赖树。由于是开发态的依赖，需要使用 `--standalone` 参数，依赖会被安装在 `.ten` 目录下，包括开发依赖的 Python module 和独立测试时依赖的系统包。
    </Callout>

    ### 2.3 项目结构

    ```
    my_asr_extension/
    ├── manifest.json       # 扩展元数据
    ├── property.json       # 默认配置
    ├── requirements.txt    # Python 依赖
    ├── extension.py        # 主实现文件
    ├── addon.py            # 扩展入口
    ├── __init__.py         # Python 包初始化
    ├── docs/               # 文档目录
    ├── .vscode/            # VS Code 调试配置
    └── tests/              # 测试文件
    ```

    #### 文件说明

    **manifest.json 和 property.json** 是 TEN Extension 的标准 metadata 文件：

    * `manifest.json`：包含插件的**名称、版本、介绍、依赖以及 schema 定义**
    * `property.json`：定义插件的**默认属性值**

    <Callout type="info">
      详细了解 TEN Framework 的元数据系统，请参考 [元数据系统文档](/docs/ten_framework/metadata_system/metadata_system)。
    </Callout>

    **requirements.txt** 是 Python 插件独有的文件，用于声明插件本身对**第三方 pip 包的依赖**。

    **extension.py** 是 extension 的**核心源代码**，包含所有业务逻辑实现。

    **tests/** 文件夹用于**插件独立测试**，包含单元测试和测试配置。
  </div>

  <div className="fd-step">
    ## 理解接口规范

    ASR Extension 继承自 `AsyncASRBaseExtension`，需实现以下方法：

    ### 必须实现的方法

    ```python
    vendor() -> str                                          
        # 返回服务商名称（如 "deepgram", "azure"）

    start_connection() -> None                               
        # 建立与服务商的连接

    stop_connection() -> None                                
        # 停止连接

    send_audio(frame: AudioFrame, session_id: str | None) -> bool  
        # 发送音频帧给服务商
        # session_id: 音频来源标识（单人场景可忽略）
        # 返回 True 表示发送成功

    finalize(session_id: str | None) -> None                 
        # VAD 检测到说话结束后调用，触发最终结果

    is_connected() -> bool                                   
        # 返回当前连接状态

    input_audio_sample_rate() -> int                         
        # 返回期望的音频采样率（如 16000）
    ```

    ### 基类自动处理

    基类 `AsyncASRBaseExtension` 已经帮你处理了：

    * 音频帧接收和队列管理
    * 性能指标计算（TTFW、TTLW）
    * 会话管理和元数据传递

    你只需要专注于对接具体的 ASR 服务商。
  </div>

  <div className="fd-step">
    ## 实现核心功能

    ### 4.1 配置管理

    #### 配置模型设计

    ```python
    from pydantic import BaseModel
    from typing import Dict, Optional

    class MyAsrConfig(BaseModel):
        # 服务商参数（透传设计）
        params: Dict[str, Optional[str]] = {}
        
        # 音频 dump 功能
        dump: bool = False
        dump_path: Optional[str] = None
    ```

    #### params 透传设计的优势

    `params` 是一个字典，用于存放**所有服务商相关的参数**。这些参数会**直接透传给服务商 SDK**，而不需要在 Extension 代码中枚举。

    **为什么这样设计？**

    ✅ **灵活性**：使用者可以通过 `property.json` 配置任何服务商支持的参数，不受 Extension 实现限制

    ✅ **可扩展**：服务商新增参数时，Extension 代码无需修改

    ✅ **简洁性**：避免为每个参数都定义一个配置字段

    **示例**：

    ```json title="property.json"
    {
      "params": {
        "api_key": "your_api_key_here",
        "language": "zh-CN",
        "model": "nova-2",
        "punctuate": "true",
        "custom_param": "any_value"  // 任何服务商支持的参数
      },
      "dump": false
    }
    ```

    使用时直接透传给服务商：

    ```python
    # 从 params 中读取并传给服务商
    api_key = self.config.params.get("api_key")
    language = self.config.params.get("language", "en-US")  # 带默认值
    ```

    <Callout type="warning">
      **注意**：模板生成的 `property.json` 文件内容为空 `{}`，你需要手动添加配置。
    </Callout>

    ### 4.2 读取配置

    ```python
    @override
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        await super().on_init(ten_env)
        
        # 读取配置
        config_json, _ = await ten_env.get_property_to_json("")
        self.config = MyAsrConfig.model_validate_json(config_json)
        
        ten_env.log_info(f"Config loaded: {self.config.model_dump_json()}")
    ```

    ### 4.3 实现基础方法

    ```python
    class MyAsrExtension(AsyncASRBaseExtension):
        def __init__(self, name: str):
            super().__init__(name)
            self.config: MyAsrConfig = MyAsrConfig()
            self.client = None  # 服务商 SDK 客户端
            self.is_connected_flag = False
            
        @override
        def vendor(self) -> str:
            return "my_vendor"  # 改为你的服务商名称
        
        @override
        def input_audio_sample_rate(self) -> int:
            # 从 params 中读取，带默认值
            return int(self.config.params.get("sample_rate", "16000"))
        
        @override
        def is_connected(self) -> bool:
            return self.is_connected_flag
    ```

    ### 4.4 实现连接管理

    <Callout type="warning">
      **重要提示**：以下代码仅为示例，实际实现需要根据你的服务商而定。

      不同服务商的连接方式差异很大：

      * 有些提供封装好的 **SDK**（如 Azure、Deepgram）
      * 有些需要直接使用 **WebSocket** 连接
      * 有些使用 **HTTP 流式 API**

      请参考你的服务商文档，使用对应的连接方式。
    </Callout>

    #### 建立连接

    ```python
    @override
    async def start_connection(self) -> None:
        """建立与服务商的连接"""
        try:
            await self.stop_connection()  # 先停止已有连接
            
            # 1. 初始化服务商客户端
            # 从 params 透传参数给服务商 SDK
            # 示例（需根据实际服务商调整）：
            self.client = VendorClient(
                api_key=self.config.params.get("api_key"),
                language=self.config.params.get("language", "en-US"),
                # ... 其他参数直接从 params 中读取并透传
            )
            
            # 2. 注册事件处理器
            # 不同服务商的事件机制不同，这里仅为示例
            self.client.on("connected", self._on_open)
            self.client.on("result", self._on_transcript)
            self.client.on("error", self._on_error)
            
            # 3. 启动连接
            await self.client.connect()
            
        except Exception as e:
            self.ten_env.log_error(f"Failed to connect: {e}")

    @override
    async def stop_connection(self) -> None:
        """停止连接"""
        if self.client:
            await self.client.disconnect()
            self.client = None
            self.is_connected_flag = False
    ```

    <Callout type="tip">
      **params 透传的价值**：所有参数都从 `params` 字典中读取并透传给服务商 SDK，这样用户可以灵活配置任何服务商支持的参数，不需要修改代码。

      **参考现有实现**：

      * `azure_asr_python` - 使用 Azure SDK
      * `deepgram_asr_python` - 使用 Deepgram SDK
    </Callout>

    ### 4.5 发送音频

    ```python
    @override
    async def send_audio(self, audio_frame: AudioFrame, session_id: str | None) -> bool:
        """发送音频数据
        
        Args:
            audio_frame: 音频帧
            session_id: 会话ID，用于标识不同的对话轮次
            
        Returns:
            bool: 发送成功返回 True，否则返回 False
        """
        if not self.is_connected() or not self.client:
            return False
            
        try:
            audio_buf = audio_frame.get_buf()
            if audio_buf:
                await self.client.send(bytes(audio_buf))
                return True
        except Exception as e:
            self.ten_env.log_error(f"Failed to send audio: {e}")
        
        return False
    ```

    ### 4.6 处理识别结果

    服务商会通过回调函数返回识别结果，你需要转换为标准格式并发送：

    ```python
    async def _on_transcript(self, result):
        """处理识别结果"""
        # 1. 提取文本
        text = result.text.strip()
        if not text:
            return
        
        # 2. 转换为标准 ASR 结果
        asr_result = ASRResult(
            text=text,
            final=result.is_final,           # 是否为最终结果
            start_ms=result.start_time_ms,   # 开始时间
            duration_ms=result.duration_ms   # 持续时间
        )
        
        # 3. 发送给下游
        await self.send_asr_result(asr_result)
    ```

    其他必需的事件处理：

    ```python
    async def _on_open(self):
        """连接建立"""
        self.is_connected_flag = True
        self.ten_env.log_info("Connection opened")

    async def _on_error(self, error):
        """处理错误"""
        self.ten_env.log_error(f"Vendor error: {error}")
    ```

    ### 4.7 实现 finalize

    当 VAD 检测到用户说话结束时，会调用 `finalize()` 方法，通知服务商尽快返回最终结果：

    ```python
    @override
    async def finalize(self, session_id: str | None) -> None:
        """触发最终结果"""
        if self.client:
            # 不同服务商有不同的 finalize 方式：
            # 1. 调用 finalize API（推荐）
            await self.client.finalize()
            
            # 2. 或发送静音包
            # await self.client.send_silence()
            
            # 3. 或断连重连
            # await self.stop_connection()
            # await self.start_connection()
            
            # 通知完成
            await self.send_asr_finalize_end()
    ```

    <Callout type="info">
      不同服务商的 finalize 机制不同，选择最适合你的服务商的方式。详细说明见 [Advanced 部分](#8-优化-finalize-机制)。
    </Callout>

    到这里，基础功能已经实现完成！接下来可以进行测试。
  </div>

  <div className="fd-step">
    ## 测试

    ### 5.1 单元测试

    单元测试用于验证 ASR Extension 的基础功能是否正常工作。

    #### 测试目标

    基础功能至少需要验证：

    1. **配置加载**：能正确读取 `property.json` 中的配置
    2. **连接建立**：能成功连接到服务商
    3. **音频处理**：能接收并发送音频帧
    4. **结果输出**：能输出标准的 `ASRResult` 格式结果

    #### 测试流程示例

    一个典型的单元测试会：

    1. **准备测试音频**：从文件加载 PCM 音频数据
    2. **逐帧发送**：将音频按帧发送给 ASR Extension
    3. **验证结果**：检查是否收到标准格式的 `ASRResult`

    **ASRResult 标准格式**：

    ```python
    ASRResult(
        text="识别出的文本",          # 必需：识别文本
        final=True,                   # 必需：是否为最终结果
        start_ms=0,                   # 可选：开始时间（毫秒）
        duration_ms=1000,             # 可选：持续时间（毫秒）
        language="zh-CN",             # 可选：语言
        words=[]                      # 可选：词级别信息
    )
    ```

    #### 运行测试

    ```bash
    cd my_asr_extension
    ./tests/bin/start
    ```

    <Callout type="tip">
      模板已经包含基础测试用例，你可以在 `tests/test_basic.py` 中添加更多测试。

      参考现有实现的测试：

      * `azure_asr_python/tests/test_asr_result.py` - 测试结果输出
      * `deepgram_asr_python/tests/test_basic.py` - 基础功能测试
    </Callout>

    ### 5.2 端到端测试

    在 TEN Agent 项目中使用 TMan Designer 替换 ASR 扩展：

    ```bash
    cd /path/to/your/ten-agent-project
    tman designer
    ```

    通过可视化界面：

    1. 选择现有 ASR 节点
    2. 替换为你的 `my_asr_extension`
    3. 配置 API Key 等参数
    4. 启动并进行真实对话测试
  </div>

  <div className="fd-step">
    ## Basic 开发自检清单

    完成以下检查项，确保基础功能正常：

    * [ ] **项目创建**：使用模板创建项目，安装依赖成功
    * [ ] **配置管理**：能从property中正确读取配置，并正确透传给服务商SDK
    * [ ] **连接建立**：`start_connection` 能成功连接到服务商
    * [ ] **音频发送**：能通过 `send_audio` 发送音频帧给服务商
    * [ ] **结果接收**：能正确接收服务商返回的识别结果，并转换为标准格式
    * [ ] **结果发送**：能通过 `send_asr_result` 发送标准化结果
    * [ ] **Finalize**：实现 `finalize` 方法（即使是简单实现）
    * [ ] **连接清理**：`stop_connection` 能正确关闭连接
    * [ ] **单元测试**：基础测试用例能通过
    * [ ] **端到端测试**：在 TEN Agent 中能完成基本对话

    <Callout type="success">
      完成以上检查项后，你的 ASR Extension 已具备基本功能，可以在实际场景中使用。
    </Callout>

    ***
  </div>
</div>

# Part 2: Advanced - 生产级质量

进阶部分将帮助你实现生产级 ASR Extension，包括稳定性、可维护性和可调试性。

<div className="fd-steps">
  <div className="fd-step">
    ## 自动重连机制

    ### 7.1 为什么需要重连

    网络环境复杂，ASR 服务可能出现：

    * 临时网络抖动
    * 服务端主动断连
    * 超时错误

    实现自动重连确保服务稳定性。

    ### 7.2 使用 ReconnectManager

    参考现有 ASR 扩展（如 `azure_asr_python`）中的 `reconnect_manager.py`。

    ```python
    from .reconnect_manager import ReconnectManager

    class MyAsrExtension(AsyncASRBaseExtension):
        def __init__(self, name: str):
            super().__init__(name)
            # 最多重连 5 次，基础延迟 0.5 秒（指数退避）
            self.reconnect_manager = ReconnectManager(max_attempts=5, base_delay=0.5)
    ```

    ### 7.3 在连接成功时重置

    ```python
    async def _on_open(self, *args, **kwargs) -> None:
        self.is_connected_flag = True
        self.ten_env.log_info("Connection opened", category=LOG_CATEGORY_VENDOR)
        
        if self.reconnect_manager:
            self.reconnect_manager.mark_connection_successful()
    ```

    ### 7.4 处理断连和错误

    ```python
    async def _on_close(self, *args, **kwargs) -> None:
        self.is_connected_flag = False
        self.ten_env.log_warn("Connection closed", category=LOG_CATEGORY_VENDOR)
        
        if self.client:  # 意外断连
            await self._handle_reconnect()

    async def _on_error(self, *args, **kwargs) -> None:
        error = args[1] if len(args) > 1 else None
        self.ten_env.log_error(f"Vendor error: {error}", category=LOG_CATEGORY_VENDOR)
        
        await self.send_asr_error(
            ModuleError(module=MODULE_NAME_ASR, code=ModuleErrorCode.NON_FATAL_ERROR.value, message=str(error))
        )
        await self._handle_reconnect()
    ```

    ### 7.5 实现重连逻辑

    ```python
    async def _handle_reconnect(self) -> None:
        if not self.reconnect_manager or not self.reconnect_manager.can_retry():
            self.ten_env.log_error("Max reconnection attempts reached", category=LOG_CATEGORY_VENDOR)
            await self.send_asr_error(
                ModuleError(module=MODULE_NAME_ASR, code=ModuleErrorCode.FATAL_ERROR.value, message="Reconnection failed")
            )
            return
        
        self.ten_env.log_info(
            f"Attempting reconnection {self.reconnect_manager.current_attempts + 1}/{self.reconnect_manager.max_attempts}",
            category=LOG_CATEGORY_VENDOR
        )
        
        success = await self.reconnect_manager.handle_reconnect(connect_func=self.start_connection)
        
        if success:
            self.ten_env.log_info("Reconnection successful", category=LOG_CATEGORY_VENDOR)
        else:
            self.ten_env.log_error("Reconnection failed", category=LOG_CATEGORY_VENDOR)
    ```
  </div>

  <div className="fd-step">
    ## 优化 Finalize 机制

    ### 8.1 Finalize 的作用

    当 VAD 检测到人声结束后，立即触发 ASR 返回 final 结果，降低对话延迟。

    ### 8.2 三种实现方式

    不同服务商支持的方式不同：

    1. **调用 API**（推荐，如 Deepgram）
    2. **断连重连**（需处理好重连逻辑）
    3. **发送静音包**（需注意时间戳计算）

    ### 8.3 通知完成

    ```python
    @override
    async def finalize(self, session_id: str | None) -> None:
        """触发最终结果
        
        Args:
            session_id: 会话ID，用于标识不同的对话轮次
        """
        if self.client:
            # 记录时间戳用于延迟计算
            self.last_finalize_timestamp = asyncio.get_event_loop().time() * 1000
            
            # 方式1: 调用 API
            await self.client.finalize()
            
            # 通知完成
            await self.send_asr_finalize_end()
    ```
  </div>

  <div className="fd-step">
    ## 规范化日志

    ### 9.1 日志分类

    使用 `category` 参数让日志清晰：

    * `LOG_CATEGORY_KEY_POINT`: 关键节点（配置、初始化）
    * `LOG_CATEGORY_VENDOR`: 服务商相关（连接、结果、错误）

    ```python
    from ten_ai_base.const import LOG_CATEGORY_KEY_POINT, LOG_CATEGORY_VENDOR

    self.ten_env.log_info("Config loaded", category=LOG_CATEGORY_KEY_POINT)
    self.ten_env.log_error("Connection failed", category=LOG_CATEGORY_VENDOR)
    ```

    ### 9.2 关键日志点

    必须打印的日志：

    ```python
    # 配置加载
    self.ten_env.log_info(f"Config: {self.config.to_json(sensitive_handling=True)}", category=LOG_CATEGORY_KEY_POINT)

    # 连接状态变化
    self.ten_env.log_info("Connection opened", category=LOG_CATEGORY_VENDOR)
    self.ten_env.log_warn("Connection closed", category=LOG_CATEGORY_VENDOR)

    # 错误信息
    self.ten_env.log_error(f"Vendor error: {error}", category=LOG_CATEGORY_VENDOR)

    # 重连尝试
    self.ten_env.log_info(f"Reconnecting {attempt}/{max_attempts}", category=LOG_CATEGORY_VENDOR)
    ```

    ### 9.3 敏感信息脱敏

    ```python
    from ten_ai_base.utils import encrypt

    class MyAsrConfig(BaseModel):
        params: Dict[str, Optional[str]] = {}
        
        def to_json(self, sensitive_handling: bool = False) -> str:
            if not sensitive_handling:
                return self.model_dump_json()
            
            config = self.model_copy(deep=True)
            if config.params:
                for key in ['api_key', 'key', 'token', 'secret']:
                    if key in config.params and config.params[key]:
                        config.params[key] = encrypt(config.params[key])
            return config.model_dump_json()
    ```
  </div>

  <div className="fd-step">
    ## 错误上报规范

    ### 10.1 错误分类

    **致命错误 (FATAL\_ERROR)**：

    * 配置解析失败
    * 无效的 API Key
    * 初始连接失败
    * 达到最大重连次数

    **非致命错误 (NON\_FATAL\_ERROR)**：

    * 临时网络问题
    * 服务暂时不可用
    * 音频处理错误

    ### 10.2 包含供应商信息

    ```python
    from ten_ai_base.message import ModuleError, ModuleErrorCode, ModuleErrorVendorInfo

    await self.send_asr_error(
        ModuleError(
            module=MODULE_NAME_ASR,
            code=ModuleErrorCode.NON_FATAL_ERROR.value,
            message=f"Vendor error: {str(error)}"
        ),
        ModuleErrorVendorInfo(
            vendor="deepgram",
            code=getattr(error, 'code', 'unknown'),
            message=str(error)
        )
    )
    ```
  </div>

  <div className="fd-step">
    ## 音频 Dump 功能

    ### 11.1 为什么需要 Dump

    在出现识别问题时，保存原始音频用于：

    * 复现问题
    * 分析音频质量
    * 对比不同服务商

    ### 11.2 实现 Dump

    ```python
    import os
    from ten_ai_base.dumper import Dumper

    DUMP_FILE_NAME = "my_asr_in.pcm"

    class MyAsrExtension(AsyncASRBaseExtension):
        def __init__(self, name: str):
            super().__init__(name)
            self.audio_dumper: Optional[Dumper] = None
        
        @override
        async def on_init(self, ten_env: AsyncTenEnv) -> None:
            await super().on_init(ten_env)
            
            if self.config.dump:
                dump_file_path = os.path.join(self.config.dump_path, DUMP_FILE_NAME)
                self.audio_dumper = Dumper(dump_file_path)
                await self.audio_dumper.start()
        
        @override
        async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
            await super().on_deinit(ten_env)
            if self.audio_dumper:
                await self.audio_dumper.stop()
                self.audio_dumper = None
        
        @override
        async def send_audio(self, audio_frame: AudioFrame) -> bool:
            buf = audio_frame.get_buf()
            
            # Dump 音频
            if self.audio_dumper and buf:
                await self.audio_dumper.push_bytes(bytes(buf))
            
            # 发送音频
            if self.is_connected() and self.client:
                await self.client.send(bytes(buf))
                return True
            
            return False
    ```
  </div>

  <div className="fd-step">
    ## 音频缓冲策略

    ### 12.1 为什么推荐保持模式

    默认使用**丢弃模式**：连接断开时丢弃音频帧。

    推荐使用**保持模式**：缓存音频帧，连接恢复后发送。

    **原因**：保证时间戳准确性。

    如果丢弃音频：

    * 时间：0-10s（发送）→ 10-15s（断连丢弃）→ 15-20s（发送）
    * ASR 只收到 15s 音频，但实际时间跨度 20s
    * 时间戳偏差 5s，影响对话同步和打断检测

    ### 12.2 配置保持模式

    ```python
    from ten_ai_base.asr import ASRBufferConfig, ASRBufferConfigModeKeep

    @override
    def buffer_strategy(self) -> ASRBufferConfig:
        return ASRBufferConfig(
            mode=ASRBufferConfigModeKeep(byte_limit=10 * 1024 * 1024)  # 10MB 缓存上限
        )
    ```
  </div>

  <div className="fd-step">
    ## 完善测试

    ### 13.1 单元测试覆盖

    Advanced 部分的单元测试需要覆盖生产级特性。使用 Mock 避免真实 API 调用，确保测试快速、稳定、可重复。

    #### 测试用例设计

    参考 `azure_asr_python/tests` 的实现，高级测试应覆盖：

    **1. 重连能力测试** (`test_reconnect.py`)

    **测试目标**：验证 Extension 能够自动重连并恢复服务

    **用例设计**：

    ```python
    # 模拟供应商断连场景
    def test_reconnect():
        # 1. Mock 供应商：前3次连接会断开，第4次成功
        # 2. 验证 Extension 会自动重试
        # 3. 验证重连成功后能正常工作
        # 4. 检查错误上报次数（应该有3次 NON_FATAL_ERROR）
    ```

    **验证点**：

    * ✅ 断连后自动触发重连
    * ✅ 使用指数退避策略
    * ✅ 达到重连次数上限时上报 FATAL\_ERROR
    * ✅ 重连成功后重置计数器

    ***

    **2. 非法参数测试** (`test_invalid_params.py`)

    **测试目标**：验证配置错误时的错误处理

    **用例设计**：

    ```python
    # 使用非法参数启动 Extension
    def test_invalid_params():
        # 1. 提供空的或无效的 params（如缺少 api_key）
        # 2. 启动 Extension
        # 3. 验证收到 FATAL_ERROR
        # 4. 检查错误信息是否包含有用信息
    ```

    **验证点**：

    * ✅ 配置验证失败时上报 FATAL\_ERROR
    * ✅ 错误消息清晰，便于排查
    * ✅ Extension 不会崩溃

    ***

    **3. 音频 Dump 测试** (`test_dump.py`)

    **测试目标**：验证音频 Dump 功能的完整性

    **用例设计**：

    ```python
    # 开启 dump 后发送音频
    def test_dump():
        # 1. 配置 dump=True 和 dump_path
        # 2. 发送 N 帧音频（每帧有特定的字节模式）
        # 3. 测试结束后检查 dump 文件
        # 4. 验证文件大小 = N * 帧大小
        # 5. 验证每一帧的内容完全一致
    ```

    **验证点**：

    * ✅ Dump 文件被创建
    * ✅ 所有发送的音频都被完整 dump
    * ✅ Dump 内容与发送内容完全一致（逐字节验证）
    * ✅ 帧顺序正确

    ***

    **4. Finalize 延迟测试** (`test_finalize.py`)

    **测试目标**：验证 Extension 能快速输出 final 结果

    **用例设计**：

    ```python
    # 测试 finalize 响应速度
    def test_finalize():
        # 1. 持续发送音频
        # 2. 1.5秒后发送 asr_finalize 事件
        # 3. Mock 供应商在收到 finalize 后快速返回 final 结果
        # 4. 验证收到 asr_finalize_end 事件
        # 5. 检查 finalize_id 和 metadata 是否正确传递
    ```

    **验证点**：

    * ✅ 收到 `asr_finalize` 后触发供应商的 finalize
    * ✅ 快速收到 final 结果（\< 300ms 典型）
    * ✅ 发送 `asr_finalize_end` 通知下游
    * ✅ `finalize_id` 和 `session_id` 正确传递

    ***

    **5. 结果格式测试** (`test_asr_result.py`)

    **测试目标**：验证 ASR 结果的标准格式

    **用例设计**：

    ```python
    # 验证输出结果的数据结构
    def test_asr_result():
        # 1. Mock 供应商返回识别结果
        # 2. 验证 ASRResult 包含所有必需字段
        # 3. 验证 interim 和 final 结果都正确
        # 4. 验证 metadata 正确传递（如 session_id）
    ```

    **验证点**：

    * ✅ 包含必需字段：`text`, `final`, `start_ms`, `duration_ms`, `language`
    * ✅ 可选字段正确填充：`words`, `metadata`
    * ✅ `session_id` 从输入正确传递到输出

    ***

    **6. 错误上报测试** (`test_vendor_error.py`)

    **测试目标**：验证错误分类和供应商信息上报

    **用例设计**：

    ```python
    # 模拟供应商返回错误
    def test_vendor_error():
        # 1. Mock 供应商返回不同类型的错误
        # 2. 验证 Extension 上报正确的错误类型
        # 3. 检查是否包含 ModuleErrorVendorInfo
    ```

    **验证点**：

    * ✅ 临时错误上报 NON\_FATAL\_ERROR
    * ✅ 严重错误上报 FATAL\_ERROR
    * ✅ 包含供应商错误码和消息
    * ✅ 错误信息便于调试

    ***

    **7. 性能指标测试** (`test_metrics.py`)

    **测试目标**：验证性能指标正确计算和上报

    **用例设计**：

    ```python
    # 测试 TTFW、TTLW 指标
    def test_metrics():
        # 1. 发送音频
        # 2. Mock 供应商在特定时间返回首字和最后一字
        # 3. 验证计算出的 TTFW 和 TTLW 指标
    ```

    **验证点**：

    * ✅ TTFW（首字延迟）正确计算
    * ✅ TTLW（末字延迟）正确计算
    * ✅ 指标通过 `metrics` 消息上报

    ***

    #### 运行测试

    ```bash
    cd my_asr_extension
    ./tests/bin/start
    ```

    <Callout type="info">
      **Mock 的重要性**：使用 Mock 而不是真实 API 的原因：

      * 🚀 **速度快**：测试在几秒内完成
      * 💰 **零成本**：不消耗 API 配额
      * 🎯 **可控性**：可以精确模拟各种场景（断连、错误、延迟）
      * 🔁 **可重复**：结果稳定，适合 CI/CD

      参考 `azure_asr_python/tests/mock.py` 了解如何实现 Mock。
    </Callout>

    ### 13.2 集成测试（Guarder）

    使用真实 API Key 运行：

    ```bash
    cd ai_agents
    task asr-guarder-test EXTENSION=my_asr_extension
    ```

    测试内容：

    * ASR 识别准确性
    * Finalize 延迟
    * 多语言支持
    * 错误处理
    * 性能指标

    ### 13.3 VS Code 调试

    使用预置的 `.vscode/launch.json`：

    1. 在代码中设置断点
    2. 按 `F5` 启动调试
    3. 选择 "Python: Test Extension"
  </div>

  <div className="fd-step">
    ## Advanced 开发自检清单

    完成以下检查项，确保生产级质量：

    **稳定性**：

    * [ ] 实现 `ReconnectManager` 自动重连
    * [ ] 指数退避策略（避免频繁重连）
    * [ ] 最大重连次数限制
    * [ ] 连接成功后重置重连计数器

    **Finalize 机制**：

    * [ ] 实现 VAD 触发后的快速 finalize
    * [ ] 调用 `send_asr_finalize_end()` 通知完成
    * [ ] 如使用断连方式，处理好重连逻辑
    * [ ] 如使用静音包，正确计算时间戳

    **日志规范**：

    * [ ] 使用 `LOG_CATEGORY_KEY_POINT` 和 `LOG_CATEGORY_VENDOR` 分类
    * [ ] 记录连接状态变化
    * [ ] 记录所有错误信息
    * [ ] 敏感信息脱敏（API Key 等）

    **错误上报**：

    * [ ] 正确区分 FATAL\_ERROR 和 NON\_FATAL\_ERROR
    * [ ] 包含 `ModuleErrorVendorInfo` 供应商信息
    * [ ] 配置错误上报 FATAL\_ERROR
    * [ ] 临时网络错误上报 NON\_FATAL\_ERROR

    **音频 Dump**：

    * [ ] 实现 `Dumper` 集成
    * [ ] 支持 `dump` 和 `dump_path` 配置
    * [ ] 在 `on_init` 初始化，`on_deinit` 清理

    **音频缓冲**：

    * [ ] 实现 `buffer_strategy()` 返回保持模式
    * [ ] 设置合理的缓存上限（如 10MB）

    **测试覆盖**：

    * [ ] 单元测试覆盖所有核心功能
    * [ ] 使用 Mock 避免真实 API 调用
    * [ ] Guarder 测试通过（真实 API）
    * [ ] 端到端测试验证稳定性

    **代码质量**：

    * [ ] 遵循项目代码风格
    * [ ] 添加必要的注释
    * [ ] 提供清晰的 README

    <Callout type="success">
      完成以上检查项后，你的 ASR Extension 达到生产级质量，可以贡献到社区并发布到商店。
    </Callout>

    ***
  </div>
</div>

# Part 3: 发布与贡献

<div className="fd-steps">
  <div className="fd-step">
    ## 发布到 TEN Store

    ### 15.1 提交到主仓库

    ```bash
    # Fork TEN Framework 仓库
    git clone https://github.com/your-username/ten-framework.git
    cd ten-framework

    # 复制扩展到正确位置
    cp -r /path/to/your/my_asr_extension ai_agents/agents/ten_packages/extension/

    # 创建分支
    git checkout -b feat/add-my-asr-extension

    # 提交
    git add ai_agents/agents/ten_packages/extension/my_asr_extension/
    git commit -m "feat: add my_asr_extension for [供应商名称] ASR service"
    git push origin feat/add-my-asr-extension
    ```

    ### 15.2 创建 Pull Request

    1. 访问你的 fork 仓库
    2. 点击 "Compare & pull request"
    3. 填写标题和描述（功能、特性、测试情况）
    4. 提交等待审查

    ### 15.3 自动发布

    PR 合并后：

    * ✅ 自动上传到 TEN Store
    * ✅ 自动处理版本号
    * ✅ 全球开发者可下载使用
  </div>

  <div className="fd-step">
    ## 适配其他 ASR 服务

    参考 TEN Framework 中的其他成品 ASR 扩展：

    ```bash
    ten-framework/ai_agents/agents/ten_packages/extension/
    ├── azure_asr_python/          # Azure Speech Services
    ├── deepgram_asr_python/       # Deepgram ASR
    ├── google_asr_python/         # Google Cloud Speech
    ├── xfyun_asr_python/          # 科大讯飞
    └── ...                        # 更多扩展
    ```

    所有扩展都遵循相同架构，可作为适配新服务的参考。

    ***
  </div>
</div>

# 附录

## A. ASR Interface 规范

### A.1 Interface 继承

在 `manifest.json` 中声明：

```json
{
  "api": {
    "interface": [
      {
        "import_uri": "../../system/ten_ai_base/api/asr-interface.json"
      }
    ]
  }
}
```

### A.2 标准属性

`asr-interface.json` 定义的标准属性：

* `dump`: 布尔值，是否开启音频 dump
* `dump_path`: 字符串，音频 dump 存储路径

### A.3 扩展属性

在 `api.property` 中声明特有属性：

```json
{
  "api": {
    "property": {
      "properties": {
        "params": {
          "type": "object",
          "properties": {
            "key": { "type": "string" },
            "language": { "type": "string" }
          }
        }
      }
    }
  }
}
```

### A.4 输入输出数据格式

**输入**：

* `pcm_frame`: PCM 音频帧
* `asr_finalize`: VAD 检测到人声结束事件

**输出**：

* `asr_result`: 识别结果
* `asr_finalize_end`: Finalize 完成通知
* `error`: 错误信息
* `metrics`: 性能指标

详见 `asr-interface.json` 文件。

## B. 基类方法参考

### B.1 AsyncASRBaseExtension

**必须实现**：

* `vendor() -> str`
* `start_connection() -> None`
* `stop_connection() -> None`
* `send_audio(frame: AudioFrame, session_id: str | None) -> bool`
* `finalize(session_id: str | None) -> None`
* `is_connected() -> bool`
* `input_audio_sample_rate() -> int`

**可选实现**：

* `input_audio_channels() -> int`
* `input_audio_sample_width() -> int`
* `buffer_strategy() -> ASRBufferConfig`
* `audio_actual_send_metrics_interval() -> int`

**工具方法**：

* `send_asr_result(asr_result: ASRResult)`
* `send_asr_error(error: ModuleError, vendor_info: ModuleErrorVendorInfo | None)`
* `send_asr_finalize_end()`
* `send_connect_delay_metrics(connect_delay: int)`
* `send_vendor_metrics(vendor_metrics: dict)`

## C. 完整示例代码

参考 `deepgram_asr_python` 或 `azure_asr_python` 扩展的完整实现：

```
ten-framework/ai_agents/agents/ten_packages/extension/
├── deepgram_asr_python/
│   ├── extension.py           # 完整实现
│   ├── reconnect_manager.py   # 重连管理器
│   ├── manifest.json
│   ├── property.json
│   └── tests/                 # 完整测试用例
```

## D. 调试工具

### D.1 VS Code 调试配置

`.vscode/launch.json`：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Test Extension",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/tests/bin/start",
      "console": "integratedTerminal",
      "cwd": "${workspaceFolder}",
      "env": {
        "PYTHONPATH": "${workspaceFolder}:..."
      }
    }
  ]
}
```

### D.2 调试特定测试

修改 `args` 参数：

```json
{
  "args": [
    "tests/test_basic.py::test_asr_basic_functionality",
    "-v"
  ]
}
```

***

## 总结

本教程分为 Basic 和 Advanced 两部分：

**Basic 部分**：

* 快速创建项目
* 实现核心功能
* 通过基础测试
* 完成端到端验证

**Advanced 部分**：

* 自动重连机制
* 优化 finalize
* 规范化日志
* 错误上报
* 音频 dump
* 完善测试

按照自检清单逐项完成，即可开发出生产级 ASR Extension。

<Callout type="success">
  **开发愉快！** 遇到问题欢迎在 [TEN Framework GitHub](https://github.com/TEN-framework/ten-framework) 提 Issue。
</Callout>


# 切换语言 (/cn/docs/ten_agent_examples/playground/change_language)

当我们谈论智能体的语言时，我们指的是以下几个方面：

* 智能体理解用户音频输入的语言。即 STT 语言。
* 智能体回复用户的语言。即 TTS 语言。

您需要根据您的要求配置上述扩展的不同属性。

## 更改 STT + LLM + TTS 智能体的 STT 语言

要更改智能体理解用户音频输入的语言，您需要更改 STT 扩展的属性。您可以按照以下步骤操作：

1. 打开 [localhost:3000](http://localhost:3000) 上的 playground 以配置您的智能体。
2. 选择“voice\_assistant”图表类型。
3. 单击图表选择右侧的按钮以打开属性配置。
4. 从下拉列表中选择“STT”扩展。
5. 将语言属性更改为所需的语言。
6. 单击“保存更改”以将语言应用于 STT 扩展。

> 注意：STT 扩展的语言属性根据 STT 服务提供商而异。有关支持的选项列表，请参阅 STT 服务提供商的文档。

## 更改 STT + LLM + TTS 智能体的 STT 语言（使用 RTC 集成的 Azure STT）

要更改智能体理解用户音频输入的语言，您需要更改 RTC 扩展的属性，因为 STT 扩展集成在其中。您可以按照以下步骤操作：

1. 打开 [localhost:3000](http://localhost:3000) 上的 playground 以配置您的智能体。
2. 选择“voice\_assistant\_stt\_integrated”图表类型。
3. 单击图表选择右侧的按钮以打开属性配置。
4. 从下拉列表中选择“RTC”扩展。
5. 将语言属性更改为所需的语言。对于 Azure，“en-US”代表英语。您可以在 Azure 文档中找到更多语言选项。
6. 单击“保存更改”以将语言应用于 RTC 扩展。

## 更改智能体说话的语言

> 注意：TTS 支持的语言通常由 TTS 的声音决定。有些 TTS 支持多语言声音，而有些只支持一种语言。

要更改智能体回复用户的语言，您需要更改 TTS 扩展的属性。您可以按照以下步骤操作：

1. 打开 [localhost:3000](http://localhost:3000) 上的 playground 以配置您的智能体。
2. 选择“voice\_assistant”图表类型。
3. 单击图表选择右侧的按钮以打开属性配置。
4. 从下拉列表中选择“TTS”扩展。
5. 将声音属性更改为所需语言的声音。
6. 单击“保存更改”以将语言应用于 TTS 扩展。

> 注意：TTS 扩展的声音属性根据 TTS 服务提供商而异。有关支持的选项列表，请参阅 TTS 服务提供商的文档。


# 配置模块 (/cn/docs/ten_agent_examples/playground/configure_modules)

本指南将帮助您在 TEN-Agent Playground 中配置模块。

## 配置模块

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择一种图表类型（例如，语音智能体、实时对话智能体）。
3. 单击图表选择右侧的按钮以打开模块选择。
4. 根据图表类型，您可以从下拉列表中选择可用的模块。
5. 单击“保存更改”以将模块应用于图表。
6. 如果您看到成功提示，则表示该模块已成功应用于图表。

## 可用模块

以下模块类型可用于 TEN-Agent Playground：

### 语音识别 (STT)

语音识别模块将语音转换为文本。

### 文本转语音 (TTS)

文本转语音模块将文本转换为语音。

### 大型语言模型 (LLM)

大型语言模型模块基于相关输入文本生成文本。

### 语音转语音模型 (V2V)

语音转语音模型模块基于相关输入语音生成语音。

### 工具 (TOOL)

工具模块提供一组供智能体使用的工具。这些工具可以绑定到 `LLM` 模块或 `V2V` 模块。

## 绑定工具模块

您可以在 TEN-Agent Playground 中将工具模块绑定到 `LLM` 或 `V2V` 模块。
工具可以为智能体提供其他功能，例如天气检查、新闻更新等。如果需要，您也可以编写自己的工具扩展。

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择一种图表类型（例如，语音智能体、实时智能体）。
3. 单击图表选择右侧的按钮以打开模块选择。
4. 根据图表类型，您将在模块选择器中看到 `LLM` 或 `V2V` 模块。
5. 单击模块右侧的按钮以打开工具选择。
6. 选择可绑定到模块的可用工具。
7. 单击“保存更改”以将工具应用于模块。
8. 如果您看到成功提示，则表示该工具已成功应用于模块。


# 配置属性 (/cn/docs/ten_agent_examples/playground/configure_properties)

本指南将帮助您在 TEN-Agent Playground 中配置属性。

## 配置属性

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择一种图表类型（例如，语音智能体、实时智能体）。
3. 单击图表选择右侧的按钮以打开属性配置。
4. 从下拉列表中，您可以找到图中使用的所有扩展节点。
5. 选择一个扩展节点以配置其属性。例如：stt / llm / tts / v2v / tool。
6. 您将看到可以为所选扩展节点配置的属性列表。
7. 您可以通过单击输入字段或开关（如果是布尔值）来更改属性的值。
8. 单击“保存更改”以将属性应用于扩展节点。
9. 如果您看到成功提示，则表示该属性已成功应用于扩展节点。

## 添加更多属性

有些属性在扩展节点中定义，但目前没有值。这些属性不会显示在属性配置中。您可以按照以下步骤添加更多属性：

1. 在属性列表抽屉中，单击“添加属性”按钮。
2. 将打开一个新抽屉，其中包含可用属性的下拉列表。
3. 从下拉列表中选择一个属性。
4. 单击“添加”以将该属性添加到扩展节点。
5. 配置属性的值。
6. 单击“保存更改”以将属性应用于扩展节点。
7. 如果您看到成功提示，则表示该属性已成功应用于扩展节点。


# 运行 Playground (/cn/docs/ten_agent_examples/playground)

本指南将帮助您运行 TEN-Agent Playground。Playground 是一个基于 Web 的界面，允许您与 TEN-Agent 进行交互。您可以在受控环境中替换模块、更改配置并测试智能体的行为。

## 从预构建的 Docker 镜像运行 Playground

运行 Playground 的最简单方法是使用预构建的 Docker 镜像。该镜像包含 TEN-Agent 和 Playground 的最新版本。项目 docker compose 文件已包含预构建的镜像，因此您可以按照[入门指南](./getting_started)中的步骤开始。

完成后，您可以通过打开浏览器并导航到 `http://localhost:3000` 来访问 Playground。

## 从源代码运行 Playground

如果您想从源代码运行 Playground，请启动一个新终端，切换到克隆 TEN-Agent 项目的文件夹，然后按照以下步骤操作：

```bash
# 切换到 Playground 目录
cd playground

# 安装依赖项
pnpm install

# 启动 Playground
pnpm dev
```

Playground 启动后，您可以通过打开浏览器并导航到 `http://localhost:3001` 来访问它。

> **注意：** Playground 依赖于 golang Web 服务器（位于 `server` 目录中）和 ten-dev-server（由 ten framework cli 提供）。默认情况下，当您使用 `docker compose up -d` 命令启动时，会提供 ten-dev-server。在您按照[入门指南](./getting_started)中的步骤操作后，在容器中运行 `task run` 命令时，Web 服务器将启动。


# 理解扩展文件夹 (/cn/docs/ten_agent_examples/project_structure/extension)

`ten_packages/extension` 文件夹包含扩展模块。每个扩展模块都是一个独立的 Python/Golang/C++ 包。

扩展文件夹名称通常与扩展模块名称相同，你也可以在扩展文件夹下的 `manifest.json` 文件中找到扩展模块名称。模块名称应在 `property.json` 文件的 `addon` 属性中使用。

以下是扩展文件夹的示例结构：

![扩展文件夹结构](https://ten-framework-assets.s3.amazonaws.com/doc-assets/extension_folder_struct.png?raw=true)

## 扩展通用文件

* manifest.json: 此文件包含扩展的元数据。它包括扩展名称、版本、属性和 API（data、audio\_frame、video\_frame）。
* property.json: 此文件包含扩展的默认属性。它用于定义扩展的默认配置。

## Python 扩展

* extension.py: 此文件包含扩展的主要逻辑。它通常包含扩展的核心实现。
* requirements.txt: 此文件包含扩展所需的 Python 依赖项。当你运行 `task use` 时，将自动安装此文件中指定的依赖项。

## Golang 扩展

* \_extension.go: 此文件包含扩展的主要逻辑。它通常包含扩展的核心实现。
* go.mod: 此文件包含 Go 模块定义。它指定模块名称和扩展的依赖项。

## C++ 扩展

* src/main.cc: 此文件包含扩展的主要逻辑。它通常包含扩展的核心实现。
* BUILD.gn: 此文件包含扩展的构建配置。它指定扩展的目标名称和依赖项。


# 项目结构 (/cn/docs/ten_agent_examples/project_structure)

TEN Agent 项目构建于 TEN framework 之上。有关 TEN framework 项目基本概念，请参考 [TEN framework 项目概览](../ten_framework/concept_overview)。

以下是 TEN Agent 项目的文件夹结构：

![项目结构](https://ten-framework-assets.s3.amazonaws.com/doc-assets/folder_structure.png)

它包含以下重要的文件夹和文件：

* `property.json`: 此文件包含扩展的编排配置。它是主要的运行时配置文件。
* `ten_packages/extension`: 此文件夹包含扩展模块。每个扩展模块都是一个独立的 Python/Golang/C++ 包。
* `server`: 此文件夹包含 Web 服务器代码。它负责处理传入的请求以及 agent 进程的启动/停止。
* `playground`: 此文件夹包含 playground 的 UI 代码。它是一个基于 Web 的界面，用于与 agent 交互。


# 理解 property.json (/cn/docs/ten_agent_examples/project_structure/property_json)

此文件包含扩展的编排配置。它是主要的运行时配置文件。

![图表概览](https://ten-framework-assets.s3.amazonaws.com/doc-assets/graph_at_a_glance.png?raw=true)

`property.json` 文件包含以下编排信息：

* 图 (Graphs)
* 节点 (Nodes)
* 连接 (Connections)

## 图 (Graphs)

`predefined_graphs` 属性包含了可用图的列表。每个图定义了 agent 在特定场景中的行为方式。每个图都有一个 `name` 属性，用于标识图。

每个图包含 `nodes` 和 `connections` 属性。

## 节点 (Nodes)

`nodes` 部分包含图中包含的扩展列表。每个节点都有一个 `name` 属性，用于标识节点，以及一个 `addon` 属性，用于指定它将使用的扩展模块。你可以在一个图中拥有多个相同扩展类型的节点。例如，你可以拥有多个具有相同 `addon` 属性的 `chatgpt_openai_python` 扩展，而每个节点将具有不同的 `name` 属性。它的工作方式类似于同一扩展的多个实例。

#### 节点属性

节点的 `property` 部分包含扩展的配置。每个扩展都有其特定的属性，这些可用的属性在扩展文件夹下的 `manifest.json` 文件中定义。你可以使用定义扩展的运行时属性来自定义其行为。以下是如何配置 `chatgpt_openai_python` 扩展的示例：

```json
{
  "name": "chatgpt_openai_python",
  "addon": "chatgpt_openai_python",
  "property": {
    "api_key": "${env:OPENAI_API_KEY|}",
    "model": "gpt-3.5-turbo",
    "temperature": 0.5,
    "max_tokens": 100,
    "prompt": "You are a helpful assistant"
  }
}
```

![Property JSON 节点](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_json_connections.png?raw=true)

#### 读取环境变量

扩展通常需要 `api_key` 才能工作。如果你不想在 `property.json` 文件中硬编码 `api_key`，可以使用环境变量。你可以使用 `${env:<env_var_name>|<default_value>}` 语法来读取环境变量。以下是如何读取 `OPENAI_API_KEY` 环境变量的示例：

```json
{
  "name": "chatgpt_openai_python",
  "addon": "chatgpt_openai_python",
  "property": {
    "api_key": "${env:OPENAI_API_KEY|}"
  }
}
```

## 通信链路 (Connections)

> `connections`部分包含节点之间通信链路的列表。每个通信链路分别指定了源节点和目标节点。每个通信链路都有一个 `extension_group` 和 `extension` 属性，用于指定源节点，并定义了 TEN framework 支持的一组多模态数据协议（`audio_frame`、`video_frame`、`data`、`cmd`）。对于每个数据协议，它都有一个目标定义列表，并且每个目标定义都有一个 `name` 属性（它是属性数据的键）和一个 `dest` 属性（它定义了目标节点列表）。下面是如何连接两个节点的示例，

```json
{
  "extension_group": "default",
  "extension": "agora_rtc",
  "audio_frame": [
    {
      "name": "pcm_frame",
      "dest": [
        {
          "extension_group": "default",
          "extension": "deepgram_asr"
        }
      ]
    }
  ]
}
```

![Property JSON 连接](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_json_nodes.png?raw=true)

在上面的示例中，我们将 `agora_rtc` 扩展连接到 `deepgram_asr` 扩展。`agora_rtc` 扩展正在向 `deepgram_asr` 扩展发送 `pcm_frame` 数据。


# 理解 Web 服务器 (/cn/docs/ten_agent_examples/project_structure/server)

Web 服务器是一个简单的 Golang Web 服务器，用于处理 HTTP 请求。它负责处理传入的请求并启动/停止agent进程。

我们已构建 Playground/演示 UI 来与 Web 服务器进行交互。您还可以使用 curl 或任何其他 HTTP 客户端与 Web 服务器进行交互。

## 请求与响应示例

该服务器提供一个用于管理智能体进程的简单层。API 资源如下所述。

### API 资源

* [POST /start](#post-start)
* [POST /stop](#post-stop)
* [POST /ping](#post-ping)

### POST /start

此 API 将根据提供的图和覆盖属性启动一个智能体。该智能体启动后会加入指定的频道，并订阅您的浏览器或设备通过 RTC 加入时所用的用户 ID。

| 参数            | 描述                                                                                                                                  |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| request\_id   | 用于跟踪目的的任何 UUID                                                                                                                      |
| channel\_name | 频道名称，需要与您的浏览器/设备加入的频道相同，智能体需要与您的浏览器/设备在同一频道中进行通信                                                                                    |
| user\_uid     | 您的浏览器/设备的 RTC 加入时所用的 UID，智能体需要知道您的 RTC UID 才能订阅您的音频                                                                                 |
| bot\_uid      | 可选，机器人用于加入 RTC 的 UID                                                                                                                |
| graph\_name   | 启动智能体时要使用的图，将在 property.json 中查找                                                                                                    |
| properties    | 要在 property.json 中覆盖的其他属性，覆盖不会更改原始的 property.json，仅更改智能体启动时使用的属性                                                                    |
| timeout       | 确定智能体在未收到任何 ping 的情况下将保持活动状态的时间。如果超时设置为 `-1`，则智能体不会因不活动而终止。默认情况下，超时设置为 60 秒，但可以使用 `.env` 文件中的 `WORKER_QUIT_TIMEOUT_SECONDS` 变量进行调整。 |

示例:

```bash
curl 'http://localhost:8080/start' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "request_id": "c1912182-924c-4d15-a8bb-85063343077c",
    "channel_name": "test",
    "user_uid": 176573,
    "graph_name": "camera_va_openai_azure",
    "properties": {
      "openai_chatgpt": {
        "model": "gpt-4o"
      }
    }
  }'
```

### POST /stop

此 API 停止您启动的智能体。

| 参数            | 描述                |
| ------------- | ----------------- |
| request\_id   | 用于跟踪目的的任何 UUID    |
| channel\_name | 频道名称，即您用于启动智能体的频道 |

示例:

```bash
curl 'http://localhost:8080/stop' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "request_id": "c1912182-924c-4d15-a8bb-85063343077c",
    "channel_name": "test"
  }'
```

### POST /ping

此 API 向服务器发送 ping，以表明连接仍然有效。如果你在启动智能体时指定 `timeout:-1`，则不需要此操作，否则如果未在超时（以秒为单位）后收到 ping，则智能体会退出。

| 参数            | 描述                |
| ------------- | ----------------- |
| request\_id   | 用于跟踪目的的任何 UUID    |
| channel\_name | 频道名称，即您用于启动智能体的频道 |

示例:

```bash
curl 'http://localhost:8080/ping' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "request_id": "c1912182-924c-4d15-a8bb-85063343077c",
    "channel_name": "test"
  }'
```


# 搭建开发环境 (/cn/docs/ten_agent_examples/setup_development_env)

开始使用 TEN Agent 的最简单方法是使用提供的 Docker 容器。此容器包含运行agent所需的所有依赖项和配置。您还可以通过创建自己的扩展或修改现有扩展来自定义智能体。

如果您想开发智能体，使用合适的开发工具可以节省大量时间。本指南将包含以下教程：

* 使用 VSCode 设置本地开发环境以连接您的 Docker 容器。
* 使用 Github Codespace 设置云开发环境。


# 在 Codespace 中云端开发 (/cn/docs/ten_agent_examples/setup_development_env/setting_up_development_inside_codespace)

Github Codespace 是一个基于云的开发环境，允许您直接在云中开发代码。这是开发智能体的好方法，无需设置本地开发环境。您不需要强大的网络或强大的机器。本指南将向您展示如何使用 Github CodeSpace 为 TEN Agent 设置开发环境。

## 步骤 1：使用 TEN-Agent 存储库创建 Codespace

导航到 TEN-Agent github 存储库，然后单击“Code”按钮。然后，选择“Open with Codespaces”。Github 将在新的浏览器选项卡中为您创建一个 Codespace。首次创建 Codespace 时，可能需要几分钟才能设置完成。

![使用 Codespaces 打开](https://ten-framework-assets.s3.amazonaws.com/doc-assets/start_codespace.png?raw=true)

## 步骤 2：开始使用 Codespace

一旦 Codespace 准备就绪，您将在浏览器中看到 VSCode 编辑器。您可以开始在 Codespace 中使用 TEN-Agent 库。您可以像在本地开发环境中一样打开终端、创建新文件和运行命令。

这些步骤类似于[快速入门](./getting_started)指南中概述的过程。但是，无需使用 Codespace 运行 `docker compose up` 命令。您可以直接使用以下命令开始构建智能体：

```bash
# 从 .env.example 准备 .env 文件并在其中设置您的 API 密钥
cp ./.env.example .env
# 完成密钥配置后构建智能体
task use
```

成功构建后，您可以运行以下命令来启动服务器：

```bash
task run
```

## 步骤 3：启动前端 UI

由于我们没有在 Codespace 中使用 docker compose，因此您需要单独启动前端 UI。您可以通过在终端中运行以下命令来执行此操作：

```bash
cd playground
pnpm install
pnpm dev
```

这将启动 `localhost:3000` 上的前端 UI。

## 步骤 4：访问您的远程设置

这是一个仅适用于 Codespace 的特殊步骤。Codespace 将自动为您在 Codespace 中公开的每个端口生成一个 URL。您可以通过单击 Codespace 窗口左下角的“Ports”选项卡来找到 URL。

![端口](https://ten-framework-assets.s3.amazonaws.com/doc-assets/codespace_ports.png?raw=true)

所有端口的可见性默认设置为“private”，您可以通过右键单击并选择“port visibility”将其更改为“public”。将其公开后，您可以在浏览器中访问该 URL 以查看前端 UI。在这里，我们将右键单击 3000 端口的可见性列，并将其可见性更改为“public”。

![更改可见性](https://ten-framework-assets.s3.amazonaws.com/doc-assets/codespace_visibility.png?raw=true)

现在，您可以访问 Codespace 生成的 URL 中的前端 UI。找到端口 3000 的“Forwarded Address”列，单击该列，浏览器应为您启动一个新选项卡，其中包含您的前端 UI。

![转发地址](https://ten-framework-assets.s3.amazonaws.com/doc-assets/codespace_forwarded_addr.png?raw=true)

OK！您已成功使用 Github Codespace 为 TEN Agent 设置了开发环境。现在，您可以直接在云中开始开发您的智能体。对于调试和开发体验，它基本上与您的本地 VSCode 体验相同。


# VSCode 开发容器 (/cn/docs/ten_agent_examples/setup_development_env/setting_up_vscode_for_development_inside_container)

在使用 TEN 进行开发时，通常建议在容器内进行编译和开发。然而，如果在容器外使用 VSCode，可能会遇到符号无法解析的问题。这是因为一些环境依赖项安装在容器内部，而 VSCode 无法识别容器的环境，导致头文件无法解析。

为了解决这个问题，可以将 VSCode 挂载到容器内部，使其能够识别容器的环境并正确解析头文件。本指南将引导您使用 VSCode 的 Dev Containers 和 Docker 扩展来实现此目的。

## 步骤 1：安装 Docker 扩展

首先，在 VSCode 中安装 [Docker 扩展](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)。此扩展允许您直接在 VSCode 中管理 Docker 容器。

## 步骤 2：安装 Dev Containers 扩展

接下来，安装 [Dev Containers 扩展](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)。此扩展使 VSCode 能够连接到 Docker 容器进行开发。

## 步骤 3：使用 Docker Compose 启动开发环境

此步骤类似于[快速入门](./getting_started)指南中概述的过程。但是，请勿运行：

```shell title="终端"
docker compose up
```

使用 `docker compose up -d` 命令，在分离模式下启动容器：

```shell title="终端"
docker compose up -d
```

执行此命令后，容器应启动。打开 VSCode，切换到 Docker 扩展，您应该会看到正在运行的容器。

![Docker 容器](https://ten-framework-assets.s3.amazonaws.com/doc-assets/docker_containers.png)

## 步骤 4：连接到容器

在 VSCode 中的 Docker 扩展中，在列表中找到 `ten_agent_dev` 容器，然后单击“Attach Visual Studio Code”以连接到容器。然后，VSCode 将打开一个连接到容器的新窗口，您可以在其中继续进行开发。

在连接到容器的 Dev Container 环境中，不会应用您的本地扩展和设置，因为此环境位于容器内。因此，您需要在容器内安装扩展并配置设置。要在容器内安装扩展，请打开新启动的 VSCode 窗口，单击左侧边栏中的“Extensions”，搜索所需的扩展，然后按照提示在容器内安装它。

## 步骤 5：设置断点进行调试

在代码中设置断点是调试时常用的做法。要在代码中设置断点，请单击要设置断点的行号的左侧边距。将出现一个红色圆点，表示已设置断点。

![设置断点](https://ten-framework-assets.s3.amazonaws.com/doc-assets/setting_breakpoint.png)

如果无法设置断点，通常意味着您尚未在容器中安装语言扩展。您可以通过单击左侧边栏中的“Extensions”图标，搜索所需的扩展，然后按照提示安装它来安装语言扩展。

设置断点后，您可以通过单击左侧边栏中的“Run and Debug”图标，选择“debug python”配置，然后单击绿色播放按钮开始调试来开始调试。

![调试配置](https://ten-framework-assets.s3.amazonaws.com/doc-assets/debug_config.png?raw=true)

通过这种方式，VSCode 直接启动智能体应用程序，这意味着 Golang Web 服务器不参与运行。因此，您需要注意以下几点：

### 在此模式下使用哪个图？

Web 服务器将在启动智能体时帮助您操作 `property.json`，以帮助您选择要使用的图。但是，在此模式下，您需要手动修改 `property.json` 以选择要使用的图。Ten 默认会选择**第一个将 `auto_start` 属性设置为 true 的图**进行启动。

### RTC 属性

RTC 功能对于您的客户端与智能体服务器通信是强制性的。Web 服务器将帮助您生成 RTC 令牌和频道，并将这些信息更新到 `property.json` 中。但是，在此模式下，您需要手动修改 `property.json` 以自行更新 RTC 令牌和频道。最简单的方法是先使用 playground/demo 运行，然后会生成一个临时属性文件，您可以从中复制 RTC 令牌和频道。

您应在 ping 请求日志中找到临时属性文件路径，如下所示：

```shell
2024/11/23 08:39:19 INFO handlerPing start channelName=agora_74np6e requestId=218f8e36-f4d0-4c83-a6e8-d3b4dec2a187 service=HTTP_SERVER
2024/11/23 08:39:19 INFO handlerPing end worker="&{ChannelName:agora_74np6e HttpServerPort:10002 LogFile:/tmp/astra/app-agora_74np6e-20241123_083855_000.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-agora_74np6e-20241123_083855_000.json Pid:5330 QuitTimeoutSeconds:60 CreateTs:1732351135 UpdateTs:1732351159}" requestId=218f8e36-f4d0-4c83-a6e8-d3b4dec2a187 service=HTTP_SERVER
```

临时属性文件路径显示为 `PropertyJsonFile:/tmp/astra/property-agora_74np6e-20241123_083855_000.json`。


# TEN-Agent 中的中断机制 (/cn/docs/ten_agent_examples/tutorials/how_does_interrupt_work)

## 概述

[TEN-Agent](https://github.com/TEN-framework/TEN-Agent) 中的中断机制由两个主要部分组成：**中断检测**和**中断响应**。本文档详细介绍了这两个部分，并解释了中断命令如何在 AI 智能体的图中传播。

## 第 1 部分：中断检测

<div className="fd-steps">
  <div className="fd-step">
    ### 当前中断检测的实现

    当前的([interrupt\_detector\_python](https://github.com/TEN-framework/TEN-Agent/tree/main/agents/ten_packages/extension/interrupt_detector_python))扩展实现了一种基于文本的中断检测机制：

    ```python
    def on_data(self, ten: TenEnv, data: Data) -> None:
        text = data.get_property_string(TEXT_DATA_TEXT_FIELD)
        final = data.get_property_bool(TEXT_DATA_FINAL_FIELD)

        # 当文本为 final 或达到阈值长度时触发中断
        if final or len(text) >= 2:
            self.send_flush_cmd(ten)
    ```

    中断检测器在以下情况下触发：

    1. 当接收到最终文本时 (`is_final = true`)
    2. 当文本长度达到阈值时（≥ 2 个字符）
  </div>

  <div className="fd-step">
    ### 自定义中断检测

    要实现您自己的中断检测逻辑，您可以参考[interrupt\_detector\_python](https://github.com/TEN-framework/TEN-Agent/tree/main/agents/ten_packages/extension/interrupt_detector_python)的实现作为示例，并根据您的特定需求自定义中断条件。
  </div>
</div>

## 第 2 部分：中断响应

### AI 智能体图中的链式处理

在典型的 AI 智能体图中，中断命令 (`flush`) 遵循链式处理模式：

```text
   中断检测器
       ↓
    LLM/ChatGPT
       ↓
      TTS
       ↓
   agora_rtc
```

链中的每个扩展在接收到 `flush` 命令时都遵循两个关键步骤：

1. 清理其自身的资源和内部状态
2. 将 `flush` 命令转发到下游扩展

这确保：

* 扩展按正确的顺序清理
* 没有残留数据流经系统
* 每个扩展在下一个操作之前都返回到干净状态

## 结论

[TEN-Agent](https://github.com/TEN-framework/TEN-Agent) 的中断机制使用链式处理模式来确保 AI 智能体图中所有扩展的有序清理。当发生中断时，每个扩展首先清理自身的状态，然后将 `flush` 命令转发到下游扩展，从而确保后续操作的系统状态干净。


# How to Use a Local LLM with Ollama (/cn/docs/ten_agent_examples/tutorials/how_to_run_local_model_in_ollama)

Ten Agent enables integration with various LLMs that support the OpenAI request-response format. This guide outlines how to use Ollama models locally with Ten Agent by leveraging OpenAI-compatible endpoints.

## Step 1: Verify Hardware Requirements

Ensure your hardware meets the necessary requirements before running an AI model locally. Key components to check include:

* **CPU/GPU**: Verify the processing power required by the model.
* **Memory**: Confirm sufficient memory is available to load and run the model.

This ensures smooth operation of the model on your system.

## Step 2: Install Required Software and Dependencies

Prepare your environment by installing the necessary software and dependencies:

1. **Install Ollama**: Download and install Ollama on your local machine. Refer to [Ollama](https://ollama.com/).
2. **Download the Model**: Use the command `ollama pull {model name}` to download the desired LLM model. For example, `ollama pull deepseek-r1`
3. **Set Up Ten Agent**: Ensure Ten Agent Dev and Ten Agent Playground are properly configured. Refer to [Ten Agent: Getting Started](../getting_started)

## Step 3: Start the Ollama Server

After setting up Ollama and downloading the required models, start the Ollama server by running the following command in your terminal:

```bash
OLLAMA_HOST=0.0.0.0 ollama serve
```

This launches the Ollama server on your local machine. Since the Ten Agent runs in a Docker container, you’ll need your private IP address to access the Ollama server. Retrieve your private IP for the LLM endpoint.

## Step 4: Configure Ten Agent

Integrate Ollama with Ten Agent using its OpenAI API compatibility:

1. Open Ten Agent Playground and navigate to the **Voice Assistance Graph (voice\_assistant)**.
2. Select **OpenAI** as the LLM.
3. Set the Ollama API endpoint as the LLM `base_url`: `http://<your-local-machine-private-ip>:11434/v1/`.

Replace `<your-local-machine-private-ip>` with the private IP address of your local machine.

4. Provide a `prompt`, e.g., "You're a helpful assistant."
5. Specify the `model` name used in Ollama (e.g., `deepseek-r1`).

## Step 5: Test the Integration

Once configured, Ten Agent will interact with Ollama using the same flow as OpenAI’s API. Test queries to ensure proper functionality.

## Additional Resources

For more information on Ollama’s OpenAI compatibility, refer to their official blog post: [Ollama OpenAI Compatibility](https://ollama.com/blog/openai-compatibility).

## Summary

Integrating a local model with Ten Agent is straightforward, thanks to its support for the OpenAI API format. The same process can be applied to connect with cloud-hosted LLMs or any LLMs supporting OpenAI API compatibility.


# 如何在 Python 扩展中运行本地 AI 模型 (/cn/docs/ten_framework/ai_model/how_to_run_local_model_in_python_extensions)

在 TEN framework 中，扩展可以使用第三方 AI 服务或在本地运行 AI 模型，以提高性能并降低成本。本教程介绍了如何在 Python 扩展中运行本地 AI 模型，以及如何在扩展中与之交互。

## 步骤 1：检查硬件要求

在本地运行 AI 模型之前，请确保您的硬件满足必要的要求。要验证的关键组件包括：

* **CPU/GPU**：检查模型是否需要特定的处理能力。
* **内存**：确保有足够的内存来加载和运行模型。

验证您的系统是否可以支持模型的需求，以确保顺利运行。

## 步骤 2：安装必要的软件和依赖项

硬件准备就绪后，安装所需的软件和依赖项。请按照以下步骤操作：

1. **操作系统**：确保与您的模型兼容。大多数 AI 框架都支持 Windows、macOS 和 Linux，尽管可能需要特定版本。

2. **Python 版本**：确保与 TEN Python 运行平台和模型兼容。

3. **所需的库**：安装必要的库，例如：

   * TensorFlow
   * PyTorch
   * Numpy
   * vllm

   您可以将所需的依赖项列在 `requirements.txt` 文件中，以便轻松安装。

4. **下载模型**：获取您计划运行的 AI 模型的本地版本。

## 步骤 3：实现您的 Python 扩展

以下示例说明了如何在 Python 扩展中使用 `vllm` 推理引擎实现基本的文本生成功能。

首先，在扩展中初始化本地模型：

```python
from ten_runtime import (
    Extension,
    TenEnv,
    Cmd,
    CmdResult,
)
from vllm import LLM

class TextGenerationExtension(Extension):
    def on_init(self, ten_env: TenEnv) -> None:
        self.llm = LLM(model="<model_path>")
        ten_env.on_init_done()
```

接下来，实现 `on_cmd` 方法以根据提供的输入处理文本生成：

```python
    def on_cmd(self, ten_env: TenEnv, cmd: Cmd) -> None:
        prompt = cmd.get_property_string("prompt")

        outputs = self.llm.generate(prompt)
        generated_text = outputs[0].outputs[0].text

        cmd_result = CmdResult.create(StatusCode.OK)
        cmd_result.set_property_string("result", generated_text)
        ten_env.return_result(cmd_result, cmd)
```

在此代码中，`on_cmd` 方法检索 `prompt`，使用模型生成文本，并将生成的文本作为命令结果返回。

您可以通过处理相关的输入类型来调整此方法，以实现其他功能，例如图像识别或语音转文本。

## 步骤 4：卸载模型

在扩展清理期间卸载模型以释放资源非常重要：

```python
import gc
import torch

class TextGenerationExtension(Extension):
    ...

    def on_deinit(self, ten_env: TenEnv) -> None:
        del self.llm
        gc.collect()
        torch.cuda.empty_cache()
        torch.distributed.destroy_process_group()
        print("Successfully deleted the LLM pipeline and freed GPU memory!")
        ten_env.on_deinit_done()
```

这确保了有效的内存管理，尤其是在使用 GPU 资源时。

## 总结

在 TEN Python 扩展中运行本地模型类似于原生 Python 开发。通过在适当的扩展生命周期方法中加载和卸载模型，您可以轻松集成本地 AI 模型并有效地与之交互。


# API (/cn/docs/ten_framework/api/api)

在 TEN Framework 中，extension 的 API 定义包含在 `manifest.json` 文件中。TEN Framework 定义 API 时，不使用 JSON Schema，而是使用一种类似 JSON 的语法来定义 API Schema。

## 为什么不使用 JSON Schema

TEN Framework 选择不使用标准 JSON Schema 的原因如下：

* **类型系统差异**：TEN Framework 的类型系统包含了许多 JSON Schema 没有的类型，例如 `int8`、`int16`、`int32`、`int64`、`uint8`、`uint16`、`uint32`、`uint64`、`float32`、`float64`、`buf`、`ptr` 等。

* **减少冗余**：在 TEN Framework 的 API 定义中，许多地方不需要具有弹性的设定。例如，message 的 schema 一定是 object 类型，不能是其他类型。如果要求开发者明确写 `"type": "object"` 反而显得多余，且容易引起混淆。

基于以上原因，TEN Framework 采用了一种更自然的定义方式，使用类似 JSON 的语法来定义 API Schema。

## API 顶层定义

API Schema 的定义放在 `manifest.json` 文件中的 `api` 字段中：

```json
{
  "api": {
    "property": {
      "properties": {...},
      "required": [...]
    },
    "cmd_in": [{
      "name": "foo",
      "property": {
        "properties": {...},
        "required": [...]
      },
      "result": {
        "property": {
          "properties": {...},
          "required": [...]
        }
      }
    }]
  }
}
```

## API Schema 定义模式

### 表达 object 类型字段

使用 `properties` 字段来描述 object 类型字段内每个属性的类型：

```json
{
  "my_object": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "age": {
        "type": "int32"
      }
    }
  }
}
```

### 表达必需和可选字段

与 object 类型字段同级的 `required` 字段，用来描述必需和可选字段：

```json
{
  "some_concept": {
    "properties": {
      "foo": {},
      "bar": {}
    },
    "required": ["foo"]
  }
}
```

合理使用 `required` 字段确保关键数据的存在：

```json
{
  "name": "critical_cmd",
  "property": {
    "properties": {
      "essential_param": {
        "type": "string"
      },
      "optional_param": {
        "type": "int32"
      }
    },
    "required": ["essential_param"]
  }
}
```

### 表达 array 类型字段

使用 `items` 字段来描述 array 类型字段内每个元素的类型：

```json
{
  "my_array": {
    "type": "array",
    "items": {
      "type": "string"
    }
  }
}
```

### type 字段的使用原则

如果一个字段只能是特定类型（由 TEN Framework 本身约束），就不需要明确写 `type` 字段。如果字段可以是多种类型，则需要明确写 `type` 字段。

例如，下面的 `some_concept` 字段如果只能是 object 类型，就不需要明确写 `type` 字段。而其中的 `foo` 字段可以是多种类型，则需要明确写 `type` 字段：

```json
{
  "some_concept": {
    "properties": {
      "foo": {
        "type": "string"
      },
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "string"
          }
        },
        "required": ["baz"]
      }
    },
    "required": ["foo"]
  }
}
```

## 表达 enum 类型字段

使用 `enum` 字段来描述枚举类型的所有可能值：

```json
{
  "api": {
    "property": {
      "properties": {
        "status": {
          "type": "string",
          "enum": ["CREATING", "ACTIVE", "DELETING", "FAILED"]
        }
      }
    }
  }
}
```

## Extension Property Schema

定义 extension 属性的完整示例：

```json
{
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        },
        "config": {
          "type": "object",
          "properties": {
            "enabled": {
              "type": "bool"
            },
            "settings": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          },
          "required": ["enabled"]
        }
      },
      "required": ["api_key", "max_tokens", "temperature", "config"]
    }
  }
}
```

## Command-like Message Schema

命令支持输入 (`cmd_in`) 和输出 (`cmd_out`)：

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "process_text",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            },
            "options": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "max_length": {
                  "type": "int32"
                }
              }
            }
          },
          "required": ["text"]
        },
        "result": {
          "property": {
            "properties": {
              "processed_text": {
                "type": "string"
              },
              "confidence": {
                "type": "float32"
              }
            },
            "required": ["processed_text"]
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "notify_completion",
        "property": {
          "properties": {
            "task_id": {
              "type": "string"
            },
            "status": {
              "type": "string"
            }
          },
          "required": ["task_id", "status"]
        }
      }
    ]
  }
}
```

## Data-like Message Schema

数据类型消息的 schema 定义：

```json
{
  "api": {
    "data_in": [
      {
        "name": "input_data",
        "property": {
          "properties": {
            "content": {
              "type": "string"
            },
            "metadata": {
              "type": "object",
              "properties": {
                "timestamp": {
                  "type": "int64"
                }
              }
            }
          },
          "required": ["content"]
        }
      }
    ],
    "data_out": [
      {
        "name": "output_data",
        "property": {
          "properties": {
            "result": {
              "type": "string"
            }
          }
        }
      }
    ],
    "audio_frame_in": [
      {
        "name": "audio_input"
      }
    ],
    "audio_frame_out": [
      {
        "name": "audio_output"
      }
    ],
    "video_frame_in": [
      {
        "name": "video_input"
      }
    ],
    "video_frame_out": [
      {
        "name": "video_output"
      }
    ]
  }
}
```

## 常见错误和解决方案

<div className="fd-steps">
  <div className="fd-step">
    ### 属性名不符合规范

    **错误示例**：属性名包含连字符或以数字开头

    ```json
    {
      "api-key": {  // 错误：包含连字符
        "type": "string"
      },
      "2nd_param": {  // 错误：以数字开头
        "type": "int32"
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "api_key": {
        "type": "string"
      },
      "second_param": {
        "type": "int32"
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### 数组类型缺少 items 定义

    **错误示例**：

    ```json
    {
      "my_array": {
        "type": "array"
        // 缺少 items 定义
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "my_array": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### 对象类型缺少 properties 定义

    **错误示例**：

    ```json
    {
      "my_object": {
        "type": "object"
        // 缺少 properties 定义
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "my_object": {
        "type": "object",
        "properties": {
          "field1": {
            "type": "string"
          }
        }
      }
    }
    ```
  </div>
</div>

## 总结

TEN Framework 的 Schema 系统提供了一个强大且结构化的方式来定义和验证数据结构，确保组件之间的一致性和类型安全。通过遵循设计原则和最佳实践，开发者可以创建健壮、可维护的 TEN 应用程序和扩展。

该系统支持丰富的类型定义、复杂的 API 描述以及灵活的消息转换功能，为构建大规模的实时多模态应用提供了坚实的基础。


# 日志 (/cn/docs/ten_framework/api/log)

TEN framework 允许使用不同语言开发的 extension 在同一进程中运行。这使得需要以统一的日志格式和信息在统一的日志中查看所有这些 extension 的日志，从而简化调试。

为了解决这个问题，`ten_env` 提供了一个日志 API。在 extension 的每个回调中，都可以访问 `ten_env` 的实例。使用此实例，您可以调用日志 API，将使用各种语言开发的 extension 的日志输出到统一的日志输出中。

## 日志级别

TEN framework 支持以下日志级别，按严重程度递增：

* **DEBUG** (1): 调试信息
* **INFO** (2): 一般信息
* **WARN** (3): 警告信息
* **ERROR** (4): 错误信息

## API 接口

### C++ 语言

```cpp
#include "ten_runtime/binding/cpp/ten.h"

class Extension {
public:
    void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr<ten::cmd_t> cmd) {
        // 使用便捷宏
        TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
        TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
        TEN_ENV_LOG_WARN(ten_env, "Warning message");
        TEN_ENV_LOG_ERROR(ten_env, "Error occurred");

        // 直接调用方法，支持 category 和 fields 参数
        ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
                   "Direct log call", nullptr, nullptr);
    }
};
```

C++ 提供了以下便捷宏：

* `TEN_ENV_LOG_DEBUG(ten_env, msg)`
* `TEN_ENV_LOG_INFO(ten_env, msg)`
* `TEN_ENV_LOG_WARN(ten_env, msg)`
* `TEN_ENV_LOG_ERROR(ten_env, msg)`

### Python 语言

```python
from ten_runtime import Extension, TenEnv

class MyExtension(Extension):
    def on_cmd(self, ten_env: TenEnv, cmd):
        # 使用不同级别的日志方法
        ten_env.log_debug("调试信息")
        ten_env.log_info("一般信息")
        ten_env.log_warn("警告信息")
        ten_env.log_error("错误信息")

        # 支持格式化字符串和额外参数
        test_value = "example"
        ten_env.log_info(f"处理命令，测试值: {test_value}")

        # 支持 category 和 fields 参数
        ten_env.log_info("带分类的日志", category="my_extension")
```

Python 提供了以下方法：

* `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`

### Go 语言

```go
package main

import "ten_runtime/ten"

type MyExtension struct {
    ten.DefaultExtension
}

func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    // 使用不同级别的日志方法
    tenEnv.LogDebug("调试信息")
    tenEnv.LogInfo("一般信息")
    tenEnv.LogWarn("警告信息")
    tenEnv.LogError("错误信息")

    // 支持格式化
    cmdName := cmd.GetName()
    tenEnv.LogInfo("处理命令: " + cmdName)

    // 使用完整的 Log 方法支持更多参数
    category := "my_extension"
    tenEnv.Log(ten.LogLevelInfo, "带分类的日志", &category, nil, nil)
}
```

Go 提供了以下方法：

* `LogDebug(msg string) error`
* `LogInfo(msg string) error`
* `LogWarn(msg string) error`
* `LogError(msg string) error`
* `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`

### Node.js/TypeScript 语言

```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";

class MyExtension extends Extension {
    async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
        // 使用不同级别的日志方法
        tenEnv.logDebug("调试信息");
        tenEnv.logInfo("一般信息");
        tenEnv.logWarn("警告信息");
        tenEnv.logError("错误信息");

        // 支持字符串拼接
        const cmdName = cmd.getName();
        tenEnv.logInfo("处理命令: " + cmdName);

        // 支持 category 和 fields 参数
        tenEnv.logInfo("带分类的日志", "my_extension");
    }
}
```

Node.js 提供了以下方法：

* `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
* `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
* `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
* `logError(message: string, category?: string, fields?: Value): TenError | undefined`

## 使用示例

### 基本使用

```python
# Python 示例
def on_start(self, ten_env: TenEnv):
    ten_env.log_info("extension 启动")
    ten_env.on_start_done()

def on_cmd(self, ten_env: TenEnv, cmd):
    cmd_name = cmd.get_name()
    ten_env.log_debug(f"收到命令: {cmd_name}")

    # 处理命令逻辑
    if cmd_name == "hello":
        ten_env.log_info("处理 hello 命令", category="command_handler")
        # ... 处理逻辑
    else:
        ten_env.log_warn(f"未知命令: {cmd_name}")
```

### 在线程中使用

```python
import threading
import time

def log_routine(self, ten_env: TenEnv):
    """在单独线程中记录日志"""
    while not self.stop_flag:
        self.log_count += 1
        ten_env.log_info(f"日志消息 {self.log_count}")
        time.sleep(0.05)

def on_start(self, ten_env: TenEnv):
    # 启动日志线程
    self.log_thread = threading.Thread(
        target=self.log_routine, args=(ten_env,)
    )
    self.log_thread.start()
    ten_env.on_start_done()
```

### 错误处理

```python
def on_cmd(self, ten_env: TenEnv, cmd):
    try:
        # 业务逻辑
        result = self.process_command(cmd)
        ten_env.log_info("命令处理成功")
    except ValueError as e:
        ten_env.log_error(f"参数错误: {str(e)}", category="error_handling")
    except Exception as e:
        ten_env.log_error(f"严重错误: {str(e)}", category="error_handling")
        raise
```

## 日志格式和输出

TEN framework 的日志系统会自动包含以下信息：

* **时间戳**: 日志记录的时间
* **日志级别**: DEBUG、INFO、WARN、ERROR
* **函数名**: 调用日志的函数名（如果可用）
* **文件名**: 调用日志的源文件名（如果可用）
* **行号**: 调用日志的代码行号（如果可用）
* **分类**: 可选的日志分类信息
* **消息内容**: 用户提供的日志消息

输出格式类似：

```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] 用户日志消息
```

## 高级日志配置

TEN framework 支持通过 `property.json` 文件配置高级日志功能。这允许您自定义日志的输出格式、目标和过滤规则。

### 配置结构

在 `property.json` 文件中，您可以在 `ten.log` 部分定义高级日志配置：

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "my_extension"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

### 配置组件说明

<div className="fd-steps">
  <div className="fd-step">
    #### 日志处理器 (Handlers)

    每个处理器包含三个主要组件：

    * **matchers**: 定义哪些日志消息会被此处理器处理
    * **formatter**: 定义日志的输出格式
    * **emitter**: 定义日志的输出目标
  </div>

  <div className="fd-step">
    #### 匹配器 (Matchers)

    匹配器用于过滤日志消息：

    ```json
    {
      "level": "info",      // 日志级别: "off", "debug", "info", "warn", "error"
      "category": "auth"    // 可选：日志分类（字符串）
    }
    ```

    支持的日志级别：

    * `"off"`: 关闭日志
    * `"debug"`: 调试级别
    * `"info"`: 信息级别
    * `"warn"`: 警告级别
    * `"error"`: 错误级别
  </div>

  <div className="fd-step">
    #### 格式器 (Formatter)

    控制日志的输出格式：

    ```json
    {
      "type": "plain",      // 格式类型: "plain" 或 "json"
      "colored": true       // 可选：是否启用颜色输出（仅对 plain 格式有效）
    }
    ```

    * **plain**: 人类可读的纯文本格式
    * **json**: JSON 结构化格式，便于日志分析工具处理
  </div>

  <div className="fd-step">
    #### 发射器 (Emitter)

    定义日志的输出目标：

    ##### 控制台输出

    ```json
    {
      "type": "console",
      "config": {
        "stream": "stdout",  // "stdout" 或 "stderr"
        "encryption": {      // 可选：加密配置
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### 文件输出

    ```json
    {
      "type": "file",
      "config": {
        "path": "/path/to/logfile.log",  // 日志文件路径
        "encryption": {                  // 可选：加密配置
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### OTLP 输出

    OTLP (OpenTelemetry Protocol) 输出允许您将日志发送到支持 OpenTelemetry 的后端系统，如 Jaeger、Prometheus、Grafana 等。

    ```json
    {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4317",  // OTLP 接收端点地址
        "protocol": "grpc",                   // 可选：传输协议，"grpc"（默认）或 "http"
        "headers": {                          // 可选：自定义 HTTP 头
          "Authorization": "Bearer token",
          "X-Custom-Header": "value"
        },
        "service_name": "my-service"          // 可选：服务名称，用于标识日志来源
      }
    }
    ```

    配置参数说明：

    * **endpoint**: OTLP 接收端点的 URL（必需）
    * **protocol**: 传输协议，支持 `"grpc"` 或 `"http"`，默认为 `"grpc"`
    * **headers**: 自定义 HTTP 请求头，用于认证或其他元数据传递（可选）
    * **service\_name**: 服务名称，用于在日志聚合系统中标识不同的服务（可选）
  </div>
</div>

### 配置示例

#### 示例 1: 多级别日志分离

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/errors.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 2: 按分类分离日志

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "database"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/database.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info",
              "category": "api"
            }
          ],
          "formatter": {
            "type": "plain"
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 3: 加密日志输出

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/secure/logs/app.log",
              "encryption": {
                "enabled": true,
                "algorithm": "AES-CTR",
                "params": {
                  "key": "your-32-character-encryption-key",
                  "nonce": "your-16-character-nonce"
                }
              }
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 4: OTLP 日志输出到 OpenTelemetry 收集器

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "http://localhost:4317",
              "protocol": "grpc",
              "service_name": "ten-framework-app"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "https://api.monitoring.example.com/v1/logs",
              "protocol": "http",
              "headers": {
                "Authorization": "Bearer your-api-token",
                "X-Environment": "production"
              },
              "service_name": "ten-framework-app"
            }
          }
        }
      ]
    }
  }
}
```

### 配置注意事项

1. **多个处理器**: 可以定义多个处理器来实现复杂的日志路由策略
2. **匹配优先级**: 按照配置文件中的顺序匹配处理器
3. **加密安全**: 加密密钥和 nonce 应该安全存储，不要硬编码在配置文件中
4. **文件权限**: 确保日志文件路径具有适当的写入权限
5. **性能考虑**: JSON 格式可能会比 plain 格式产生更大的日志文件
6. **OTLP 端点**:
   * 使用 gRPC 协议时，默认端口通常为 4317
   * 使用 HTTP 协议时，默认端口通常为 4318
   * 确保 OTLP 收集器端点可访问且配置正确
7. **认证**: 使用 OTLP 时，如果后端需要认证，请在 headers 中添加相应的认证信息

## 最佳实践

1. **选择合适的日志级别**：
   * 使用 `DEBUG` 记录详细的调试信息
   * 使用 `INFO` 记录正常的业务流程
   * 使用 `WARN` 记录潜在问题
   * 使用 `ERROR` 记录错误信息

2. **提供有意义的日志消息**：
   * 包含上下文信息（如命令名称、参数值等）
   * 使用描述性的消息而不是简单的标识符
   * 合理使用 category 参数对日志进行分类

3. **避免过度日志记录**：
   * 不要在高频循环中使用过多的日志
   * 在生产环境中适当调整日志级别
   * 使用高级日志配置的匹配器来过滤不必要的日志

4. **线程安全**：
   * TEN framework 的日志 API 是线程安全的
   * 可以在任何线程中安全调用日志方法

5. **高级配置建议**：
   * 为不同的模块使用不同的 category 便于日志分析
   * 在生产环境中考虑使用文件输出以便于日志持久化
   * 对敏感日志使用加密功能
   * 定期轮转日志文件以控制磁盘使用
   * 使用 JSON 格式便于与日志分析工具集成
   * 考虑使用 OTLP 输出将日志集成到现代可观测性平台（如 Grafana、Jaeger、Datadog 等）
   * OTLP 输出与 OpenTelemetry 生态系统无缝集成，便于实现分布式追踪和日志关联

整体效果如下图所示：

![多语言 extension 的统一日志输出](https://ten-framework-assets.s3.amazonaws.com/doc-assets/log.png)


# required (/cn/docs/ten_framework/api/required)

## `required` 字段的原则

1. **通信链路中，源的 `required` 字段必须是目标 `required` 字段的超集**

   这意味着如果目标 extension 要求某个字段为必需，那么源 extension 也必须将该字段标记为必需。这确保了消息在传递过程中不会缺少必要的字段。

2. **源和目标 `required` 字段中的相同字段名称必须具有兼容的类型**

   框架会验证字段类型的兼容性，确保数据可以正确传递和解析。类型兼容性检查包括基本类型匹配、数组项类型匹配以及对象属性类型匹配。

## `required` 字段的使用

### Extension 发送消息时

当 extension 调用 `send_<foo>(msg_X)` 或 `return_result(result_Y)` 时，框架会根据 extension 中的相应 schema 检查 `msg_X` 或 `result_Y`。如果 `msg_X` 或 `result_Y` 缺少 schema 中标记为 `required` 的任何字段，则 schema 检查将失败。

以下三种情况的处理方式相同：

1. **发送 command 且 schema 检查失败时**

   `send_<foo>` 将立即返回 `false`，错误讯息将包含 schema 检查失败的错误消息。

2. **`return_result` 未通过 schema 检查时**

   `return_result` 将返回 `false`，错误讯息将包含 schema 检查失败的错误消息，通常会显示缺少的必需字段，例如："the required properties are absent: 'foo'"。

3. **发送 data-like message（如数据、音频帧或视频帧）时**

   `send_<foo>` 将返回 `false`，错误讯息将包含 schema 检查失败的错误消息。

**注意：** 框架的 schema 系统仅在字段存在时生效。如果某个字段在 schema 中定义但在消息中不存在，且该字段未在 `required` 中列出，则不会进行类型检查。只有当字段同时存在于消息中和 schema 定义中时，才会进行类型验证。

### Extension 接收消息时

在 TEN runtime 将 `msg_X` 或 `result_Y` 传递给 extension 的 `on_<foo>()` 或结果处理程序之前，它会检查 `msg_X` 或 `result_Y` 的 schema 中定义的所有 `required` 字段是否都存在。如果缺少任何 `required` 字段，则 schema 检查将失败。

1. **传入消息是 command 时**

   TEN runtime 会将错误返回给上一个 extension，错误消息会指明缺少的必需字段。

2. **传入消息是 TEN 命令结果时**

   TEN runtime 会将结果的 `status_code` 更改为错误，添加缺少的 `required` 字段，并根据其类型将这些字段的值设置为默认值。

3. **传入消息是数据类 TEN 消息时**

   TEN runtime 将直接丢弃该数据类消息。

### 错误消息格式

当 `required` 字段验证失败时，框架会提供详细的错误信息：

* 单个缺失字段：`"the required properties are absent: 'field_name'"`
* 多个缺失字段：`"the required properties are absent: 'field1', 'field2'"`
* 嵌套对象中的缺失字段：`".nested_object: the required properties are absent: 'field_name'"`


# 类型系统 (/cn/docs/ten_framework/api/type_system)

## 类型

TEN framework 类型系统是 TEN framework 内用于定义值的数据类型的系统。开发者可以通过 TEN schema 声明 message 或 extension property 的类型。

TEN framework 类型系统包括基本类型和复合类型。基本类型包括以下内容：

| 类型      | 描述                     |
| ------- | ---------------------- |
| null    | 空值类型。                  |
| bool    | 布尔值，true 或 false。      |
| int8    | 8 位有符号整数。              |
| int16   | 16 位有符号整数。             |
| int32   | 32 位有符号整数。             |
| int64   | 64 位有符号整数。             |
| uint8   | 8 位无符号整数。              |
| uint16  | 16 位无符号整数。             |
| uint32  | 32 位无符号整数。             |
| uint64  | 64 位无符号整数。             |
| float32 | 单精度（32 位）IEEE 754 浮点数。 |
| float64 | 双精度（64 位）IEEE 754 浮点数。 |
| string  | Unicode (UTF-8) 字符序列。  |
| buf     | 8 位无符号字节序列。            |
| ptr     | 指向内存地址的指针。             |

复合类型包括以下内容：

| 类型     | 描述                   |
| ------ | -------------------- |
| array  | 相同类型的元素集合。           |
| object | 表示复杂的键/值对。键类型始终为字符串。 |

对于基本类型，可以使用 `get_property()` / `set_property()` 等方法访问或设置属性。例如：

```cpp
// 获取属性值
int32_t value = cmd.get_property_int32("property_name");

// 设置属性值
cmd.set_property("property_name", 100);
```

对于复合类型，通常需要使用相关的序列化方法。例如：

```go
type MyProp struct {
    Name string `json:"name"`
}

var prop MyProp
bytes, _ := json.Marshal(&prop)
cmd.SetPropertyFromJSONBytes("property_name", bytes)
```

## JSON 类型映射

当从 JSON 创建 TEN value 时，TEN framework 使用以下映射规则：

| JSON 类型 | TEN 类型映射规则                                   |
| ------- | -------------------------------------------- |
| 整数      | 如果值在 int32 范围内，优先映射为 int32；否则映射为 int64       |
| 浮点数     | 如果值在 float32 范围内，优先映射为 float32；否则映射为 float64 |
| 字符串     | 映射为 string                                   |
| 布尔值     | 映射为 bool                                     |
| null    | 映射为 null                                     |
| 数组      | 映射为 array                                    |
| 对象      | 映射为 object                                   |

需要注意的是，JSON 中的数字类型在没有明确 schema 定义的情况下，会根据数值大小自动选择合适的 TEN 类型。这确保了数据的精度和效率。

## 类型和 schema

如果指定了 TEN schema，则将根据相应的 TEN schema 确定属性类型。如果未指定 TEN schema，则将根据初始值赋值的类型确定属性类型。例如，如果初始赋值为 C++ 的 `int32_t`，则属性类型为 `int32`；如果使用 JSON 完成初始赋值，则将根据 JSON 处理规则确定类型。

当有 TEN schema 定义时，JSON 数据会被强制转换为 schema 指定的类型，如果转换失败则会报错。

## 转换规则

TEN framework 支持不同类型值之间的灵活自动转换。转换分为安全转换和不安全转换两种类型。

### 类型兼容性

TEN framework 使用类型兼容性规则来确定两个类型之间是否可以进行转换：

* **整数类型兼容**：所有整数类型（int8, int16, int32, int64, uint8, uint16, uint32, uint64）之间相互兼容
* **浮点数兼容整数**：浮点数类型可以接受整数类型的值（当整数值在浮点数表示范围内时）
* **其他类型**：bool, ptr, string, array, object, buf, null 类型只与自身兼容

### 安全转换（必须成功）

将较低精度类型转换为较高精度类型始终是安全的，并且保证成功，因为较高精度类型可以完全容纳较低精度类型的值而不会丢失数据。

在 TEN 类型系统中，安全转换规则如下：

1. 在有符号整数类型中，从较低精度到较高精度
2. 在无符号整数类型中，从较低精度到较高精度
3. 从较小范围的无符号整数到较大范围的有符号整数（当值在目标类型范围内时）
4. 在浮点数类型中，从 float32 到 float64

| 从       | 到（允许的安全转换）                                       |
| ------- | ------------------------------------------------ |
| int8    | int16 / int32 / int64                            |
| int16   | int32 / int64                                    |
| int32   | int64                                            |
| uint8   | uint16 / uint32 / uint64 / int16 / int32 / int64 |
| uint16  | uint32 / uint64 / int32 / int64                  |
| uint32  | uint64 / int64                                   |
| float32 | float64                                          |

例如：

```cpp
// 设置属性值。TEN runtime 中 `property_name` 的类型为 `int32`。
cmd.set_property("property_name", 100);

// 获取属性值。正确。
int32_t value = cmd.get_property_int32("property_name");

// 获取属性值。正确。TEN 类型系统会自动将类型转换为 `int64`。
int64_t value2 = cmd.get_property_int64("property_name");

// 获取属性值。不正确，会抛出一个错误，因为这是不安全转换。
int16_t error_type = cmd.get_property_int16("property_name");
```

### 不安全转换（可能失败）

将较高精度类型转换为较低精度类型是不安全的，因为较高精度类型的值可能超出较低精度类型的范围，从而导致数据丢失。此转换称为不安全转换。执行不安全转换时，TEN runtime 会进行严格的范围检查。如果发生溢出或精度丢失，TEN 类型系统将抛出一个错误。

在 TEN framework 类型系统中，不安全转换包括：

1. 将较高精度整数转换为较低精度整数
2. 将有符号整数转换为无符号整数（当值为负数时）
3. 将无符号整数转换为有符号整数（当值超出有符号整数范围时）
4. 将 `float64` 转换为 `float32`
5. 将浮点数转换为整数（当有小数部分或超出整数范围时）
6. 整数到浮点数的转换（当可能发生精度丢失时）

| 转换类型              | 成功条件                   |
| ----------------- | ---------------------- |
| 高精度 → 低精度         | 值必须在目标类型范围内            |
| 有符号 → 无符号         | 值必须非负且在目标类型范围内         |
| 无符号 → 有符号         | 值必须在目标类型范围内            |
| float64 → float32 | 值必须在 float32 范围内且不丢失精度 |
| 浮点数 → 整数          | 值必须在目标整数类型范围内且无小数部分    |
| 整数 → 浮点数          | 整数值必须能被浮点数精确表示，不发生精度丢失 |

#### 范围检查细节

TEN framework 实现了精确的范围检查算法：

* **整数范围检查**：使用标准的 MIN/MAX 常量进行比较
* **浮点数范围检查**：使用 `ldexp` 函数计算精确的边界值
* **精度检查**：对于整数到浮点数的转换，会进行往返转换验证以确保精度不丢失

### 转换触发时机

TEN runtime 在以下情况下执行类型转换：

1. **加载 `property.json` 时**
   * JSON 整数根据数值大小解析为相应的整数类型
   * JSON 浮点数根据数值大小解析为相应的浮点数类型
   * 如果属性有定义的 TEN schema，会根据 schema 进行强制类型转换（可能为不安全转换）

2. **调用 `set_property_from_json()` 等方法时**
   * 传递序列化的 JSON 字符串时，会根据 schema 执行类型转换

3. **属性访问时**
   * 使用 `get_property_xxx()` 方法获取属性时，会尝试将存储类型转换为请求类型
   * 这时可能触发安全或不安全转换

4. **消息传递时**
   * 当消息属性需要在不同类型之间转换时

### 错误处理

当类型转换失败时，TEN framework 会：

* 返回错误状态码和详细的错误信息
* 在错误对象中提供源类型、目标类型和失败原因
* 对于超出范围的转换，错误消息格式为："out of range of \[target\_type]"
* 对于不兼容的类型转换，错误消息格式为："unsupported conversion from `[source_type]` to `[target_type]`"

#### 常见错误示例

```cpp
// 范围溢出错误
int8_t small_value = cmd.get_property_int8("large_number");  // 如果 large_number = 200
// 错误：out of range of int8

// 类型不兼容错误
std::string str_value = cmd.get_property_string("integer_prop");
// 错误：unsupported conversion from `int32` to `string`

// 精度丢失错误
float float_value = cmd.get_property_float32("precise_double");  // 如果 precise_double 超出 float32 精度
// 错误：out of range of float32
```

## 最佳实践

1. **使用合适的类型**：根据数据的实际范围选择合适的整数和浮点数类型
2. **明确定义 schema**：在 manifest.json 中明确定义属性类型以避免类型转换歧义
3. **处理转换错误**：在代码中适当处理类型转换可能产生的错误
4. **注意 JSON 映射**：了解 JSON 数据到 TEN 类型的映射规则，避免意外的类型转换


# 开发流程 (/cn/docs/ten_framework/development/development_process)

基本上，通过使用 TEN framework 来完成所需的场景，有两种开发方法或视角：

1. 从小扩展到大场景
2. 从大场景到小扩展

**注意：** 无论您采取这两种开发方法中的哪一种，您都可以从头开始编写自己的扩展，或者从 TEN 云商店下载预构建的扩展。

在第一种方法中，**从小扩展到大场景**，开发过程通常遵循以下步骤：

1. 开发扩展。
2. 将扩展组装成图，并进一步创建agent，例如 [TEN-Agent](https://github.com/TEN-framework/TEN-Agent)。

在第二种方法中，**从大场景到小扩展**，开发过程通常遵循以下步骤：

1. 选择所需的agent模板，例如 [TEN-Agent](https://github.com/TEN-framework/TEN-Agent)。
2. 将所选agent模板中的某些扩展替换为您自己的扩展。

在第二种开发方法中，通过使用框架提供的扩展**独立测试**功能，您可以确保替换后的扩展行为与原扩展一致。在此过程中，开发人员只需了解与 TEN 扩展相关的概念，包括如何独立开发和测试 TEN 扩展。其他与 TEN 相关的概念（扩展开发和测试之外）则无需关注，因为这些概念已封装在所选的 TEN agent中。


# TEN 扩展开发完整指南 (/cn/docs/ten_framework/development/how_to_develop_with_ext)

TEN Framework 提供了丰富的扩展模板，帮助开发者快速创建扩展并完成从开发到测试的完整流程。本指南将通过实际操作演示，详细介绍如何使用 C++，Go，Python，Node.js 四种语言进行扩展开发的全流程。

## 开发前准备

### 环境要求

在开始扩展开发之前，请确保您的开发环境已正确配置。通过以下命令验证安装：

```bash
tman --version
```

正常情况下，您应该看到类似以下的版本信息输出：

```bash
TEN Framework version: <version>
```

> **重要提示**：请确保您使用的 `tman` 版本 >= 0.10.12，如果版本过低，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 安装最新版本。

### 开发流程概览

无论使用哪种编程语言，TEN 扩展开发都遵循以下标准化流程：

1. **项目创建** - 使用官方模板生成扩展项目骨架
2. **依赖安装** - 配置运行环境并安装必要的依赖包
3. **核心开发** - 实现扩展的业务逻辑和功能代码
4. **构建测试** - 编译项目（如需要）并执行单元测试
5. **调试优化** - 使用专业调试工具定位并解决问题

***

## C++ 扩展开发

C++ 扩展适用于对性能要求极高的应用场景，如实时音视频处理、高频数据计算、底层系统操作等。

### 创建项目

使用 TEN 官方提供的 C++ 扩展模板快速创建新项目：

```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```

项目创建成功后，您会得到以下完整的项目结构：

```bash
my_example_ext_cpp/
├── BUILD.gn              # GN 构建系统配置文件
├── manifest.json         # 扩展元数据和配置信息
├── property.json         # 扩展属性和参数配置
├── src/
│   └── main.cc           # 扩展核心实现代码
├── tests/
│   ├── basic.cc          # 基础功能测试用例
│   └── gtest_main.cc     # 测试框架程序入口
├── include/              # 头文件存放目录
├── tools/                # 辅助工具和脚本
└── .vscode/
    └── launch.json       # VSCode 调试配置文件
```

### 环境配置验证

进入项目目录并验证构建工具是否正常工作：

```bash
cd my_example_ext_cpp
tgn --version
```

> **期望输出**：

```bash
0.1.0
```

如果命令无法执行，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 检查 TEN Framework 开发环境是否正确安装。

### 依赖包安装

安装扩展运行时所需的全部依赖包：

```bash
tman install --standalone
```

安装成功后，您将看到类似以下的详细安装日志：

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done

🏆  Install successfully in 1 second
```

### 项目构建

TEN framework 提供了两种便捷的构建方式供您选择：

#### 方式一：手动分步构建

```bash
# 第一步：生成构建配置文件（启用独立测试模式）
tgn gen linux x64 debug -- ten_enable_standalone_test=true

# 第二步：执行项目构建
tgn build linux x64 debug
```

#### 方式二：使用 tman 快捷命令

```bash
tman run build
```

构建完成后，检查生成的可执行测试文件：

```bash
ls -la bin/
# 应该能看到：my_example_ext_cpp_test
```

### 运行测试

验证扩展功能是否正常工作：

#### 方式一：直接执行测试文件

```bash
./bin/my_example_ext_cpp_test
```

#### 方式二：使用 tman 统一命令

```bash
tman run test
```

测试执行成功时的输出示例：

```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN      ] Test.Basic
[       OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[  PASSED  ] 1 test.
```

> **成功标志**：当您看到 `[  PASSED  ] 1 test.` 时表示测试全部通过。

### 核心代码结构详解

#### 扩展主体实现（src/main.cc）

C++ 扩展的核心类需要继承自 `ten::extension_t` 基类，并实现完整的生命周期管理方法：

```cpp
class my_example_ext_cpp_t : public ten::extension_t {
 public:
  // 扩展初始化阶段 - 进行基础配置和资源预分配
  void on_init(ten::ten_env_t &ten_env) override;

  // 扩展启动阶段 - 正式开始处理业务逻辑
  void on_start(ten::ten_env_t &ten_env) override;

  // 命令处理器 - 处理来自其他扩展或应用的命令请求
  void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr<ten::cmd_t> cmd) override;

  // 数据处理器 - 处理通用数据流
  void on_data(ten::ten_env_t &ten_env, std::unique_ptr<ten::data_t> data) override;

  // 音频帧处理器 - 处理实时音频流数据
  void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::audio_frame_t> frame) override;

  // 视频帧处理器 - 处理实时视频流数据
  void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::video_frame_t> frame) override;

  // 扩展停止阶段 - 清理资源和优雅关闭
  void on_stop(ten::ten_env_t &ten_env) override;
};
```

#### 测试框架实现（tests/basic.cc）

使用 TEN 专用测试框架编写完整的单元测试：

```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    // 创建测试用的命令对象
    auto new_cmd = ten::cmd_t::create("foo");

    // 发送命令到扩展并验证响应结果
    ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
      // 在这里验证测试结果的正确性
    });
  }
};
```

### 调试环境配置

#### VSCode 集成调试

1. 安装 VSCode 的 **CodeLLDB** 扩展插件
2. 在源代码中设置断点
3. 选择 "standalone test (lldb, launch)" 调试配置
4. 按 F5 键启动调试会话

调试配置文件 `.vscode/launch.json` 的内容：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (lldb, launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

#### 命令行调试

使用经典的 GDB 调试器进行命令行调试：

```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt  # 查看完整的函数调用栈
```

***

## Go 扩展开发

Go 扩展在高性能和开发效率之间提供了很好的平衡，特别适合构建网络服务、并发处理、微服务架构等应用场景。

### 创建项目

使用 Go 扩展模板创建新项目：

```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```

项目创建成功后会显示：

```bash
🏆  Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```

完整的项目结构如下：

```bash
my_example_ext_go/
├── extension.go             # 扩展核心实现代码
├── go.mod                   # Go 模块依赖管理配置
├── manifest.json            # 扩展元数据信息
├── property.json            # 扩展属性配置
├── README.md                # 项目说明文档
├── tests/                   # 测试相关文件
│   ├── basic_tester.go      # 测试逻辑实现
│   ├── basic_tester_test.go # 测试用例定义
│   ├── main_test.go         # 测试程序入口
│   └── bin/start            # 测试启动脚本
└── .vscode/launch.json      # VSCode 调试配置
```

### 依赖包安装

安装项目运行所需的依赖包：

```bash
tman install --standalone
```

### 运行测试

验证扩展功能的正确性：

#### 方式一：使用启动脚本

```bash
./tests/bin/start
```

#### 方式二：使用 tman 命令

```bash
tman run test
```

测试成功执行的输出示例：

```bash
=== RUN   TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```

### 核心代码结构详解

#### 扩展实现（extension.go）

Go 扩展的核心实现结构：

```go
import (
    ten "ten_framework/ten_runtime"
)

type ExampleExtension struct {
    ten.DefaultExtension
}

// 扩展启动生命周期
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStart")
    tenEnv.OnStartDone()
}

// 命令处理逻辑
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    tenEnv.LogDebug("OnCmd")
    cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
    tenEnv.ReturnResult(cmdResult, nil)
}

// 扩展停止生命周期
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStop")
    tenEnv.OnStopDone()
}
```

#### 测试框架（tests/basic\_tester.go）

Go 扩展的测试框架实现：

```go
type BasicExtensionTester struct {
    ten.DefaultExtensionTester
}

func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
    // 创建用于测试的命令对象
    cmdTest, _ := ten.NewCmd("test")

    // 发送命令并处理响应结果
    tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
        if err != nil {
            panic(err)
        }
        statusCode, _ := cr.GetStatusCode()
        if statusCode != ten.StatusCodeOk {
            panic(statusCode)
        }
        tenEnvTester.StopTest(nil)
    })

    tenEnvTester.OnStartDone()
}
```

### 开发环境优化

为了提升开发体验和调试便利性，建议在扩展根目录创建 `go.work` 工作空间文件：

```go
go 1.18

use (
    .
    .ten/app/ten_packages/system/ten_runtime_go/interface
)
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Go 官方扩展，然后使用以下调试配置：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (go, launch)",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}/tests/",
      "env": {
        "CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
        "LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
      },
      "args": ["-test.v"]
    }
  ]
}
```

***

## Python 扩展开发

Python 扩展具有最高的开发效率，特别适合快速原型开发、AI/ML 应用集成、复杂业务逻辑实现等场景。

### 创建项目

使用 Python 异步扩展模板创建项目：

```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```

完整的项目结构：

```bash
my_example_ext_python/
├── extension.py         # 扩展核心实现代码
├── addon.py             # 扩展插件注册入口
├── __init__.py          # Python 包初始化文件
├── requirements.txt     # Python 依赖包清单
├── manifest.json        # 扩展元数据配置
├── property.json        # 扩展属性配置
├── tests/               # 测试相关文件
│   ├── test_basic.py    # 基础测试用例
│   ├── conftest.py      # pytest 配置文件
│   └── bin/start        # 测试启动脚本
└── .vscode/launch.json  # VSCode 调试配置
```

### 依赖包安装

安装项目所需的 Python 依赖包：

```bash
tman install --standalone
```

### 运行测试

验证 Python 扩展的功能：

#### 方式一：使用启动脚本

```bash
./tests/bin/start
```

#### 方式二：使用 tman 命令

```bash
tman run test
```

测试成功执行的输出示例：

```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py .                                                                                [100%]
============================================ 1 passed in 0.04s =======================================
```

### 核心代码结构详解

#### 扩展实现（extension.py）

Python 扩展推荐使用现代异步编程模式，以获得更好的性能和并发处理能力：

```python
from ten_runtime import (
    AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
    Cmd, StatusCode, CmdResult, Data
)

class ExampleExtension(AsyncExtension):
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_init")

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_start")
        # TODO: 在这里读取配置文件，初始化必要的资源

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_name = cmd.get_name()
        ten_env.log_debug(f"on_cmd name {cmd_name}")

        # TODO: 在这里实现具体的业务逻辑处理
        cmd_result = CmdResult.create(StatusCode.OK, cmd)
        await ten_env.return_result(cmd_result)

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_stop")
        # TODO: 在这里清理资源，进行优雅关闭
```

#### 插件注册入口（addon.py）

扩展插件的注册和创建逻辑：

```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension

@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
    def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
        ten_env.log_info("on_create_instance")
        ten_env.on_create_instance_done(ExampleExtension(name), context)
```

#### 测试实现（tests/test\_basic.py）

完整的异步测试框架实现：

```python
from ten_runtime import (
    AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
    TenError, TenErrorCode
)

class ExtensionTesterBasic(AsyncExtensionTester):
    async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
        # 创建用于测试的命令对象
        new_cmd = Cmd.create("hello_world")

        ten_env.log_debug("send hello_world")
        result, err = await ten_env.send_cmd(new_cmd)

        # 验证测试结果的正确性
        if (err is not None or result is None
            or result.get_status_code() != StatusCode.OK):
            ten_env.stop_test(TenError.create(
                TenErrorCode.ErrorCodeGeneric,
                "Failed to send hello_world"
            ))
        else:
            ten_env.stop_test()

def test_basic():
    tester = ExtensionTesterBasic()
    tester.set_test_mode_single("my_example_ext_python")
    err = tester.run()
    if err is not None:
        assert False, err.error_message()
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Python 扩展和 debugpy 调试器，使用以下配置进行调试：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (debugpy, launch)",
      "type": "debugpy",
      "request": "launch",
      "python": "/usr/bin/python3",
      "module": "pytest",
      "args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
      "env": {
        "TEN_ENABLE_PYTHON_DEBUG": "true",
        "PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
      },
      "console": "integratedTerminal"
    }
  ]
}
```

***

## Node.js 扩展开发

Node.js 扩展提供了现代 JavaScript/TypeScript 开发体验，特别适合 Web 应用集成、快速原型开发、前端技术栈扩展等场景。得益于 Node.js 的异步特性和丰富的生态系统，开发者可以轻松构建高效的实时应用。

### 创建项目

使用 TEN 官方提供的 Node.js 扩展模板快速创建新项目：

```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```

项目创建成功后，您会得到以下完整的项目结构：

```bash
my_example_ext_nodejs/
├── manifest.json         # 扩展元数据和配置信息
├── property.json         # 扩展属性和参数配置
├── package.json          # Node.js 依赖包管理配置
├── tsconfig.json         # TypeScript 编译器配置
├── src/
│   └── index.ts          # 扩展核心实现代码
├── tests/                # 测试相关文件
│   ├── src/
│   │   ├── index.ts      # 测试器实现
│   │   ├── index.spec.ts # 测试用例定义
│   │   └── main.spec.ts  # 测试框架配置
│   ├── bin/start         # 测试启动脚本
│   ├── package.json      # 测试依赖配置
│   └── tsconfig.json     # 测试 TypeScript 配置
├── tools/                # 辅助工具和脚本
└── .vscode/
    └── launch.json       # VSCode 调试配置文件
```

### 环境配置验证

进入项目目录并验证开发环境：

```bash
cd my_example_ext_nodejs
node --version
npm --version
```

> **期望输出**：

```bash
v18.0.0 或更高版本
8.0.0 或更高版本
```

### 依赖包安装

安装扩展运行时所需的全部依赖包：

```bash
tman install --standalone
```

安装成功后，您将看到类似以下的详细安装日志：

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       2/2       Done

🏆  Install successfully in 1 second
```

> **重要提示**：`tman install --standalone` 会在项目目录下创建 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录，后续的构建和测试操作都需要在这个目录下进行。

### 项目构建

Node.js 扩展使用 TypeScript 进行开发，需要先安装独立模式依赖，然后编译为 JavaScript：

#### 方式一：手动构建

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 安装独立模式的 Node.js 依赖
npm run standalone-install

# 编译 TypeScript 代码
npm run build
```

#### 方式二：使用 tman 快捷命令

```bash
tman run build
```

构建完成后，检查生成的编译结果：

```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# 应该能看到：index.js 和相关映射文件
```

### 运行测试

验证扩展功能是否正常工作：

#### 方式一：使用启动脚本

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 运行测试
./tests/bin/start
```

#### 方式二：使用 tman 统一命令

```bash
tman run test
```

测试执行成功时的输出示例：

```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case2 (1010ms)

  2 passing (2s)
```

> **成功标志**：当您看到所有测试用例显示 `✓` 并且最后显示 `passing` 时表示测试全部通过。

### 核心代码结构详解

#### 扩展主体实现（src/index.ts）

Node.js 扩展的核心类需要继承自 `Extension` 基类，并实现完整的生命周期管理方法：

```typescript
import {
  Addon,
  RegisterAddonAsExtension,
  Extension,
  TenEnv,
  Cmd,
  CmdResult,
  StatusCode,
} from "ten-runtime-nodejs";

class DefaultExtension extends Extension {
  constructor(name: string) {
    super(name);
  }

  // 扩展配置阶段 - 进行配置参数的读取和验证
  async onConfigure(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onConfigure");
  }

  // 扩展初始化阶段 - 进行基础配置和资源预分配
  async onInit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onInit");
  }

  // 扩展启动阶段 - 正式开始处理业务逻辑
  async onStart(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStart");
  }

  // 命令处理器 - 处理来自其他扩展或应用的命令请求
  async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
    console.log("DefaultExtension onCmd", cmd.getName());

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("detail", "This is a demo");
    tenEnv.returnResult(cmdResult);
  }

  // 扩展停止阶段 - 清理资源和优雅关闭
  async onStop(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStop");
  }

  // 扩展销毁阶段 - 最终清理和资源释放
  async onDeinit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onDeinit");
  }
}
```

#### 插件注册入口

扩展插件的注册和创建逻辑：

```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
  async onCreateInstance(
    _tenEnv: TenEnv,
    instanceName: string,
  ): Promise<Extension> {
    return new DefaultExtension(instanceName);
  }
}
```

#### 测试框架实现（tests/src/index.ts）

使用 TEN 专用测试框架编写完整的单元测试：

```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";

export class MyExtensionTester extends ExtensionTester {
  async onStart(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStart");

    // 模拟异步操作和测试逻辑
    new Promise((resolve) => {
      setTimeout(() => {
        resolve(true);
      }, 1000);
    }).then(() => {
      // 停止测试并返回结果
      tenEnvTester.stopTest();
    });
  }

  async onStop(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStop");
  }

  async onDeinit(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onDeinit");
  }
}
```

#### 测试用例定义（tests/src/index.spec.ts）

使用 Mocha 测试框架编写具体的测试用例：

```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";

const test_addon_name = "default_extension_nodejs";

describe("MyExtensionTester", () => {
  it("case1", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });

  it("case2", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });
});
```

### TypeScript 配置

Node.js 扩展使用现代 TypeScript 配置，支持最新的语言特性：

```json
{
  "compilerOptions": {
    "allowJs": false,
    "composite": true,
    "module": "NodeNext",
    "target": "es6",
    "moduleResolution": "NodeNext",
    "outDir": "build",
    "removeComments": false,
    "sourceMap": true,
    "noImplicitAny": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "strictNullChecks": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "rootDir": "src",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
  },
  "exclude": [
    "node_modules"
  ],
  "include": [
    "src/**/*"
  ]
}
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Node.js 官方扩展，然后使用以下调试配置：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (nodejs) (mocha, launch)",
      "type": "node",
      "request": "launch",
      "program": "node_modules/mocha/bin/_mocha",
      "stopOnEntry": true,
      "args": [
        "--no-timeouts",
        "--package",
        "package.json",
      ],
      "cwd": "${workspaceFolder}/tests",
      "env": {
        "NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
      },
      "runtimeArgs": [
        "--expose-gc",
        "--loader",
        "ts-node/esm",
        "--no-warnings",
      ]
    }
  ]
}
```

#### 命令行调试

使用 Node.js 内置调试器进行命令行调试：

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 启动调试模式
node --inspect-brk ./tests/bin/start

# 或者使用 Chrome DevTools
node --inspect ./tests/bin/start
```

### 完整开发流程总结

为了帮助开发者快速上手，这里提供一个完整的 Node.js 扩展开发流程总结：

```bash
# 1. 创建扩展项目
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example

# 2. 进入项目目录
cd my_example_ext_nodejs

# 3. 安装依赖
tman install --standalone

# 4. 进入扩展安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 5. 安装独立模式依赖
npm run standalone-install

# 6. 构建项目
npm run build

# 7. 运行测试
./tests/bin/start
```

> **工作目录说明**：
>
> * 扩展源代码位于项目根目录的 `src/` 文件夹
> * 实际的构建、测试和运行操作都在 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录下进行
> * 这种设计确保了扩展的独立性和依赖管理的正确性

### 开发最佳实践

#### 异步编程

Node.js 扩展充分利用异步编程模式，提供更好的性能：

```typescript
// 推荐：使用 async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    const result = await processCommand(cmd);
    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("result", result);
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

#### 错误处理

实现完善的错误处理机制：

```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    // 业务逻辑处理
    const result = await this.handleBusinessLogic(cmd);

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("data", JSON.stringify(result));
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    console.error("Command processing error:", error);

    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

***

## 开发总结

通过遵循本指南提供的完整开发流程，您可以高效地进行 TEN 扩展的开发、测试和调试工作。无论选择 C++，Go，Python 还是 Node.js，TEN Framework 都为您提供了完整的工具链和最佳实践，帮助您充分发挥 TEN Framework 的强大功能，构建出高性能、高可靠性的扩展应用。

每种语言都有其独特的优势和适用场景：

* **C++**：适用于对性能要求极高的场景，如实时音视频处理、高频计算
* **Go**：在高性能和开发效率之间提供平衡，适合网络服务、并发处理
* **Python**：具有最高的开发效率，特别适合AI/ML应用、快速原型开发
* **Node.js**：提供现代 Web 开发体验，适用于前端技术栈扩展、实时应用

请根据您的具体需求和团队技术栈选择最合适的开发方案。在开发过程中，建议充分利用 TEN Framework 提供的调试工具和测试框架，以确保扩展的质量和稳定性。


# TEN Framework 快速入门指南 (/cn/docs/ten_framework/getting-started/quick-start)

## TEN Framework 快速入门指南

> 🎯 **目标**：5分钟内搭建开发环境并运行第一个 TEN 应用

## 系统要求

**支持的操作系统**：

* Linux (x64)
* Linux (arm64)
* macOS Intel (x64)
* macOS Apple Silicon (arm64)
* Windows (x64)

**必需的软件环境**：

* Python 3.10
* Go 1.20+
* Node.js / npm（用于安装和管理 JavaScript 依赖）

> 💡 **Windows 用户注意**：推荐使用 PowerShell 执行本指南中的命令。部分命令在 CMD 中可能不兼容。

## 第一步：检查环境

在开始之前，请确保你的系统已安装以下软件：

### Python 3.10

```bash
python3 --version
# 应显示: Python 3.10.x
```

> 💡 **重要**：TEN Framework 目前仅支持 Python 3.10。推荐使用 `pyenv` 或 `venv` 创建 Python 虚拟环境：
>
> **Linux / macOS：**
>
> ```bash
> # 使用 pyenv 安装和管理 Python 3.10（推荐）
> pyenv install 3.10.18
> pyenv local 3.10.18
>
> # 或使用 venv 创建虚拟环境
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows：**
>
> 从 [https://www.python.org/downloads/release/python-31011/](https://www.python.org/downloads/release/python-31011/) 底部表格中下载 Windows installer，启动后安装 Python 3.10。
>
> 注意，点击Install Now前务必勾选 "Add Python to PATH"
>
> ```bash
> # Windows平台需要配置python3命令：
>
> # 首先，确认python的安装路径：
> where.exe python
> # 输出示例: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # 然后，以管理员身份打开 PowerShell，在python.exe同目录下创建symlink：
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # 请将上面的路径替换为 where.exe python 的实际输出路径
>
> # 验证：
> python3 --version
> ```
>
> ```powershell
> # 推荐安装后使用 venv 创建虚拟环境，在该环境中工作
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # 激活环境
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # 若有权限错误，关闭终端/IDE，右键选择“以管理员身份运行”重新打开
> # 或者改变执行策略来允许ps1脚本执行
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```

### Go 1.20+

```bash
go version
# 应显示: go version go1.20 或更高版本

$env:CGO_ENABLED = "1"
# Windows下需要显式启用CGO
```

### Node.js / npm

```bash
node --version
npm --version
# 确保 node 和 npm 命令可用
```

### GCC (MinGW)

💡仅在windows上需要安装

```bash
# 首先确定winget存在。
winget --version

# 若不存在则需要从 <https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US> 安装。
#（系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11）

#安装MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT

# 或查找后自行选择合适版本安装
winget search "mingw"

#检查安装
gcc --version
```

> 💡 **提示**：如果缺少上述环境，请先安装对应版本后再继续。

## 第二步：安装 TEN Manager (tman)

TEN Manager (tman) 是 TEN Framework 的命令行工具，用于创建项目、管理依赖和运行应用。

方式一：通过包管理器安装（推荐）

**Linux (Ubuntu/Debian):**

```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```

**macOS:**

```bash
brew install TEN-framework/ten-framework/tman
```

**Windows:**

```powershell
winget install TEN-framework.tman
```

方式二：通过安装脚本

**Linux / macOS：**

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```

**Windows：**

```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```

方式三：如果你已经克隆了仓库，使用仓库内的安装脚本

```bash
cd ten-framework

# Linux/MacOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```

> 💡 **提示**：如果系统中已经安装了 tman，安装脚本会询问是否重新安装/升级，输入 `y` 继续安装，输入 `n` 取消。
>
> **非交互式安装**（适用于自动化脚本或 CI 环境，Windows不可用）：
>
> ```bash
> # 远程安装
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # 本地安装
> yes y | bash tools/tman/install_tman.sh
> ```

**验证安装**：

```bash
tman --version
```

> 💡 **提示**：如果出现类似
>
> * `tman: command not found`（Linux/macOS）
> * `无法将"tman"项识别为 cmdlet、函数、脚本文件或可运行程序的名称`（Windows）
>
>   的提示，请确保 tman 所在目录在你的 PATH 中：
>
> **Linux / macOS：**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc  # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc   # macOS
> source ~/.bashrc  # 或 source ~/.zshrc
> ```
>
> **Windows：**
>
> ```powershell
> # 检查 PATH 中是否包含 tman
> $env:Path -split ";" | Select-String "tman"
> # 如果不包含，请
> # 1. 通过 "系统属性 → 环境变量" 手动添加，然后重启终端以应用
> # 2. 运行以下命令来临时添加路径
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```

## 第三步：创建并运行示例应用

<div className="fd-steps">
  <div className="fd-step">
    ### 创建应用

    ```bash
    # 创建一个新的 transcriber_demo 应用
    tman install app transcriber_demo
    cd transcriber_demo
    ```
  </div>

  <div className="fd-step">
    ### 安装依赖

    ```bash
    # 安装 TEN 包依赖
    tman install

    # 安装 Python 和 npm 包的依赖
    tman run install_deps
    ```

    > ⏱️ **预计时间**：1-2 分钟
  </div>

  <div className="fd-step">
    ### 构建应用

    ```bash
    tman run build
    ```

    > ⏱️ **预计时间**：30 秒
  </div>

  <div className="fd-step">
    ### 配置环境变量

    在运行应用前，需要配置 ASR（语音识别）服务的密钥。当前示例使用 Azure ASR extension，你需要在 `transcriber_demo/.env` 文件中填写相关配置：

    **Linux / macOS：**

    ```bash
    # 创建 .env 文件
    cat > .env << EOF
    # Azure Speech Service 配置
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region      # 例如：eastus
    AZURE_STT_LANGUAGE=en-US                # 根据你的音频语种或实时录音语种设置，如：zh-CN, ja-JP, ko-KR 等
    EOF
    ```

    **Windows（PowerShell）：**

    ```powershell
    # 创建 .env 文件
    @"
    # Azure Speech Service 配置
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region
    AZURE_STT_LANGUAGE=en-US
    "@ | Out-File -Encoding utf8 .env
    ```

    > 💡 **提示**：Windows 用户也可以直接用文本编辑器创建 `.env` 文件并填写上述配置。
    >
    > 💡 **提示**：如果你想使用其他 ASR extension（如 OpenAI Whisper、Google Speech 等），可以从云商店下载并替换，同样将相应的 API key 等环境变量配置在 `.env` 文件中。
  </div>

  <div className="fd-step">
    ### 运行应用

    ```bash
    tman run start
    ```

    如果一切正常，你应该看到类似以下的输出：

    ```text
    [web_audio_control_go] Web server started on port 8001
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```
  </div>

  <div className="fd-step">
    ### 体验 Demo

    打开浏览器访问：

    ```text
    http://localhost:8001
    ```

    你应该能看到 Transcriber Demo 的 Web 界面，可以尝试：

    * 点击麦克风按钮进行实时语音转录
    * 上传音频文件进行转录
    * 查看实时转录以及字幕结果
  </div>
</div>

## 恭喜！🎉

你已经成功运行了第一个 TEN 应用！

### 了解应用架构

这个 `transcriber_demo` 应用展示了 TEN Framework 的多语言扩展能力，它由以下组件构成：

* **Go** - WebSocket 服务器扩展 (`web_audio_control_go`)
* **Python** - ASR 语音识别扩展 (`azure_asr_python`)
* **TypeScript** - VTT 字幕生成和音频录制扩展 (`vtt_nodejs`)

🎯 **你已经可以运行这些多语言插件了！**

### 下一步

现在你可以：

1. **从云商店探索和下载更多插件，设计和编排你的应用**

   ```bash
   tman designer  # 启动 TMAN Designer，在云商店中探索插件、下载插件并设计编排你的应用
   ```

2. **选择一个语言，开发自己的插件**
   * 支持 Go、Python、TypeScript/JavaScript、C++ 等多种语言
   * 查看 [TEN 扩展开发完整指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) 了解详情

## 进阶：开发和构建 C++ 插件

如果你想开发和使用 C++ 扩展，推荐安装 TEN 构建工具链（tgn）。以下是完整的步骤：

<div className="fd-steps">
  <div className="fd-step">
    ### 安装 tgn 构建工具

    tgn 是 TEN Framework 的 C/C++ 构建系统，基于 Google 的 GN。

    方式一：一键安装（推荐）

    **Linux / macOS：**

    ```bash
    curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
    ```

    **Windows：**

    ```powershell
    irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
    ```

    方式二：从克隆的仓库安装

    **Linux / macOS：**

    ```bash
    # 如果你已经克隆了 TEN Framework 仓库
    cd ten-framework
    bash tools/tgn/install_tgn.sh
    ```

    **Windows：**

    ```powershell
    # 如果你已经克隆了 TEN Framework 仓库
    & ".\tools\tgn\install_tgn.ps1"
    ```

    安装完成后，确认 tgn 已添加到 PATH：

    **Linux / macOS：**

    ```bash
    # 临时添加到当前会话
    export PATH="/usr/local/ten_gn:$PATH"

    # 或永久添加到 shell 配置（推荐）
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc  # Linux
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc   # macOS
    source ~/.bashrc  # 或 source ~/.zshrc
    ```

    **Windows：**

    ```powershell
    # 安装脚本会自动添加到用户 PATH，如需手动添加：
    $env:Path += ";$env:LOCALAPPDATA\ten_gn"
    ```

    验证安装：

    ```bash
    tgn --help
    ```
  </div>

  <div className="fd-step">
    ### 安装 C++ 扩展

    以 WebRTC VAD（语音活动检测）扩展为例，从云商店安装 C++ 扩展：

    ```bash
    cd transcriber_demo
    tman install extension webrtc_vad_cpp
    ```

    > 💡 **提示**：`webrtc_vad_cpp` 是一个用 C++ 实现的语音活动检测扩展，可以在实时语音识别场景中筛选出语音部分。
  </div>

  <div className="fd-step">
    ### 编译 C++ 扩展

    安装 C++ 扩展后，需要重新构建应用以编译 C++ 代码为动态库：

    ```bash
    tman run build
    ```

    > ⏱️ **预计时间**：首次编译 C++ 扩展可能需要 1-3 分钟，具体取决于你的机器性能。
  </div>

  <div className="fd-step">
    ### 运行带有 VAD 功能的应用

    ```bash
    tman run start_with_vad
    ```

    如果一切正常，你应该看到：

    ```text
    [web_audio_control_go] Web server started on port 8001
    [vad] WebRTC VAD initialized with mode 2
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```

    现在打开浏览器访问 `http://localhost:8001`，进入麦克风实时转写页面，你会看到经过vad后的silence状态变化，当silence状态为true时，表示当前音频中没有语音。
  </div>
</div>

### C++ 开发环境要求

开发和编译 C++ 扩展需要安装 C++ 编译器（gcc 或 clang）：

**Linux:**

```bash
# Ubuntu/Debian
sudo apt-get install gcc g++

# 或使用 clang
sudo apt-get install clang
```

**macOS:**

```bash
# 安装 Xcode Command Line Tools (包含 clang)
xcode-select --install
```

**Windows:**

```powershell
# 方式一：安装 Visual Studio Build Tools（推荐）
# 从 https://visualstudio.microsoft.com/visual-cpp-build-tools/ 下载安装

# 方式二：使用 winget 安装
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

# 💡注意：
#  - 安装时选择 "使用 C++ 的桌面开发" 工作负载
#  - 且务必勾选“适用于Windows的C++ Clang工具”
```

验证编译器安装：

```bash
# 检查 gcc（Linux/macOS/MSYS2）
gcc --version
g++ --version

# 或检查 clang（Linux/macOS）
clang --version
```

**Windows（使用 MSVC）：**

```powershell
# 在 "Developer PowerShell for VS" 中执行
clang-cl --version
```

### 常见问题（C++ 扩展）

1. tgn 命令找不到

   确保已经执行安装脚本并将 tgn 添加到 PATH：

   ```bash
   export PATH="/usr/local/ten_gn:$PATH"
   ```

2. 编译失败：找不到编译器

   请参考上面的"C++ 开发环境要求"部分安装编译器。

### 了解更多

* [ten\_gn 构建系统](https://github.com/TEN-framework/ten_gn) - TEN 的 C/C++ 构建工具
* [C++ 扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) - 完整的 C++ 扩展开发文档

## 常见问题

<div className="fd-steps">
  <div className="fd-step">
    ### macOS 上 Python 库加载失败

    **问题**：运行应用时提示找不到 `libpython3.10.dylib`

    **解决方案**：

    ```bash
    export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
    ```

    建议将这行添加到 `~/.zshrc` 或 `~/.bash_profile` 中。
  </div>

  <div className="fd-step">
    ### tman 下载失败或速度很慢

    **问题**：网络连接 GitHub 受限

    **解决方案**：

    * 手动下载：访问 [Releases 页面](https://github.com/TEN-framework/ten-framework/releases) 下载对应平台的 `tman` 二进制文件
  </div>

  <div className="fd-step">
    ### 端口 8001 已被占用

    **问题**：启动时提示端口冲突

    **解决方案**：

    * 查找占用端口的进程：
      * Linux/macOS：`lsof -i :8001`
      * Windows：`netstat -ano | findstr :8001`
    * 杀掉该进程：
      * Linux/macOS：`kill -9 <PID>`
      * Windows：`taskkill /PID <PID> /F`
    * 或修改应用配置文件（`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`）中的端口号
  </div>

  <div className="fd-step">
    ### Go build 失败

    **问题**：构建时提示 Go module 相关错误

    **解决方案**：

    ```bash
    # 设置代理
    # Linux/macOS
    export GOPROXY=https://goproxy.cn,direct
    # Windows PowerShell
    $env:GOPROXY = "https://goproxy.cn,direct"

    # 清理 Go module 缓存
    go clean -modcache

    # 重新安装依赖
    cd transcriber_demo
    tman run build
    ```
  </div>

  <div className="fd-step">
    ### Python 依赖安装失败

    **问题**：pip 安装超时或失败

    **解决方案**：使用国内镜像源

    ```bash
    pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
    ```
  </div>

  <div className="fd-step">
    ### Azure语音服务报错

    **问题**：Azure语音服务相关错误，比如认证失败

    **解决方案**：检查.env文件中的配置是否正确，确保 AZURE\_STT\_KEY 和 AZURE\_STT\_REGION 填写无误
  </div>

  <div className="fd-step">
    ### Windows下权限问题

    **问题**：Windows下在访问文件时报PermissionError

    **解决方案**：右键点击PowerShell，选择“以管理员身份运行”
  </div>
</div>

## 获取帮助

* **GitHub Issues**：[https://github.com/TEN-framework/ten-framework/issues](https://github.com/TEN-framework/ten-framework/issues)
* **文档**：[https://theten.ai/cn/docs](https://theten.ai/cn/docs)
* **贡献指南**：[contributing.md](../code-of-conduct/contributing.md)


# 基于WebSocket的语音助手快速入门 (/cn/docs/ten_framework/getting-started/websocket-voice-assistant-quick-start)

## 基于 WebSocket 的语音助手快速入门

在不到 10 分钟内构建并运行一个基于 WebSocket 的实时语音助手。

## 简介

本指南将帮助您快速运行一个完整的语音助手 Demo，支持实时语音对话和工具调用。该示例集成了：

| 组件   | 服务商        | 用途       |
| ---- | ---------- | -------- |
| ASR  | Deepgram   | 语音转文字    |
| LLM  | OpenAI     | 对话生成     |
| TTS  | ElevenLabs | 文字转语音    |
| Tool | WeatherAPI | 天气查询（可选） |

## 环境要求

### 前置依赖

请先参考 [TEN Framework 快速入门指南](./quick-start.cn.md) 完成基础环境安装，确保以下命令通过检查：

```bash
tman check env
```

预期输出（至少 Python 和 Go 为 Ready）：

```text
✅ Operating System: Supported
✅ Python:   Ready
✅ Go:       Ready
```

### 额外工具

安装以下工具用于构建和运行：

#### Task（任务运行器）

**Linux / macOS：**

```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version  # 验证安装
```

**Windows：**

```powershell
# 1. 使用winget安装（推荐）
# 从以下地址安装winget
# https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# 系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11
winget --version # 验证安装
winget install Task.Task
task --version  # 验证安装

# 2. 使用scoop安装
irm get.scoop.sh | iex
scoop --version # 验证安装
scoop install task
task --version # 验证安装
```

#### Bun（JavaScript 包管理器）

**Linux / macOS：**

```bash
curl -fsSL https://bun.sh/install | bash
bun --version  # 验证安装
```

**Windows：**

```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version  # 验证安装
```

#### uv（Python 包管理器）

**Linux / macOS：**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version  # 验证安装
```

**Windows：**

```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version  # 验证安装
```

> 💡 安装后需要重新加载终端配置：
>
> * Linux/macOS：`source ~/.zshrc`
> * Windows：重新打开 PowerShell 终端

## 快速开始

<div className="fd-steps">
  <div className="fd-step">
    ### 克隆仓库

    ```bash
    git clone https://github.com/TEN-framework/ten-framework.git
    cd ten-framework/ai_agents/agents/examples/websocket-example
    ```
  </div>

  <div className="fd-step">
    ### 安装依赖

    ```bash
    task install
    ```

    > ⏱️ 预计耗时：2-3 分钟

    该命令会自动完成：

    * 安装 TEN 扩展包（45+ 个）
    * 编译 Go 应用和 API 服务器
    * 安装 Python 和前端依赖
  </div>

  <div className="fd-step">
    ### 配置 API Keys

    在 `ai_agents/.env` 文件中配置服务商密钥：

    **Linux / macOS：**

    ```bash
    cd ../../../  # 回到 ai_agents 目录
    vim .env
    ```

    **Windows：**

    ```powershell
    cd ..\..\..   # 回到 ai_agents 目录
    notepad .env
    ```

    从.env.example复制所有内容后，填写以下配置：

    ```bash
    # Deepgram - 语音识别
    DEEPGRAM_API_KEY=your_deepgram_api_key

    # OpenAI - 大语言模型
    OPENAI_API_KEY=your_openai_api_key
    OPENAI_MODEL=your_openai_model

    # ElevenLabs - 语音合成
    ELEVENLABS_TTS_KEY=your_elevenlabs_api_key

    # WeatherAPI - 天气工具（可选）
    WEATHERAPI_API_KEY=your_weatherapi_key
    ```
  </div>

  <div className="fd-step">
    ### 启动服务

    ```bash
    cd agents/examples/websocket-example
    task run
    ```

    服务启动后，自动运行：

    * **前端 UI**：[http://localhost:3000](http://localhost:3000)
    * **API 服务器**：[http://localhost:8080](http://localhost:8080)
    * **TMAN Designer**：[http://localhost:49483](http://localhost:49483)
  </div>

  <div className="fd-step">
    ### 与助手对话

    打开浏览器访问 [http://localhost:3000](http://localhost:3000)，点击麦克风按钮开始语音对话。

    尝试以下对话：

    * "你好，你是谁？"
    * "今天北京的天气怎么样？"（需配置 WeatherAPI）
    * "给我讲个笑话"

    **🎉 恭喜！** 您已成功运行 TEN Agent 语音助手。
  </div>
</div>

## 进阶配置

### 更换扩展

TEN Framework 支持灵活替换 ASR、LLM、TTS 等扩展。

#### 方式 1：使用 TMAN Designer（推荐）

访问 [http://localhost:49483](http://localhost:49483)，通过可视化界面：

1. 点击扩展节点查看详情
2. 选择替代扩展
3. 配置参数
4. 保存并重启

#### 方式 2：编辑配置文件

编辑 `tenapp/property.json`，修改 `predefined_graphs` 中的扩展配置。

**示例：更换 ASR 为 Azure**

```json
{
  "type": "extension",
  "name": "stt",
  "addon": "azure_asr_python",
  "extension_group": "stt",
  "property": {
    "params": {
      "key": "${env:AZURE_STT_KEY}",
      "region": "${env:AZURE_STT_REGION}",
      "language": "en-US"
    }
  }
}
```

**示例：更换 TTS 为 Minimax**

```json
{
  "type": "extension",
  "name": "tts",
  "addon": "minimax_tts_websocket_python",
  "extension_group": "tts",
  "property": {
    "params": {
      "api_key": "${env:MINIMAX_TTS_API_KEY|}",
      "group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
      "model": "speech-02-turbo",
      "audio_setting": {
        "sample_rate": 16000
      },
      "voice_setting": {
        "voice_id": "female-shaonv"
      }
    }
  }
}
```

修改后重启服务（`Ctrl+C` 然后 `task run`）。

### 配置监控

使用 Grafana 监控应用的 Metrics 和 Logs。

<div className="fd-steps">
  <div className="fd-step">
    #### 启动监控栈

    **Linux / macOS：**

    ```bash
    cd tools/grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    ```

    **Windows：**

    ```powershell
    cd tools\grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    # 若docker未安装，请通过https://apps.microsoft.com/detail/xp8cbj40xlbwkx 安装，并重启电脑后，启动docker desktop
    ```

    启动的服务：

    * **Grafana**（端口 3001）- 可视化界面
    * **Prometheus**（端口 9091）- Metrics 存储
    * **Loki**（端口 3100）- Logs 存储
    * **OpenTelemetry Collector**（端口 4317/4318）- OTLP 数据接收
  </div>

  <div className="fd-step">
    #### 配置应用遥测

    编辑 `agents/examples/websocket-example/tenapp/property.json`：

    **配置 Logs 输出**

    在 `ten.log.handlers` 数组中添加：

    ```json
    {
      "matchers": [{ "level": "info" }],
      "formatter": { "type": "json", "colored": false },
      "emitter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
    ```
  </div>
</div>

#### 配置 Metrics 输出

在 `ten.services` 中添加：

```json
{
  "telemetry": {
    "enabled": true,
    "metrics": {
      "enabled": true,
      "exporter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
  }
}
```

<div className="fd-steps">
  <div className="fd-step">
    #### 重启并访问 Grafana

    ```bash
    cd ../../../agents/examples/websocket-example
    task run
    ```

    访问 [http://localhost:3001](http://localhost:3001)（用户名/密码：`admin/admin`）
  </div>
</div>

#### 查看 Metrics

进入 `Dashboards` → **"TEN Framework"** 查看：

* 扩展启动耗时
* Cmd 处理延迟
* 消息队列等待时间

#### 查看 Logs

进入 `Explore` → 选择 `Loki`，使用查询：

* `{service_name="websocket-example"}` - 所有日志
* `{service_name="websocket-example", ten_extension_name="stt"}` - ASR 日志
* `{service_name="websocket-example", ten_extension_name="llm"}` - LLM 日志
* `{service_name="websocket-example", ten_extension_name="tts"}` - TTS 日志

> 💡 使用 `ten_extension_name` 标签精确筛选扩展日志。

停止监控服务：

**Linux / macOS：**

```bash
cd tools/grafana-monitoring
docker compose -f docker-compose.push.yml down
```

**Windows：**

```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```

## 常见问题

<details>
<summary><strong>前端依赖安装失败</strong></summary>

如果 `bun install` 报错版本不存在，切换到 npm 官方源：

```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```

</details>

<details>
<summary><strong>macOS Python 库加载失败</strong></summary>

设置 Python 动态库路径：

```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# 添加到 ~/.zshrc 永久生效
```

</details>

<details>
<summary><strong>Windows Python 库加载失败</strong></summary>

确保 Python 3.10 安装路径在系统 PATH 中：

```powershell
# 检查 Python 路径
python -c "import sys; print(sys.prefix)"

# 将 Python DLL 目录添加到 PATH（根据实际路径调整）
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```

如果安装 Python 时未勾选 "Add Python to PATH"，可以重新运行安装程序选择 "Modify" 来添加。

</details>

<details>
<summary><strong>端口被占用</strong></summary>

查找并终止占用端口的进程：

**Linux / macOS：**

```bash
lsof -i :3000  # 或 :8080
kill -9 <PID>
```

**Windows：**

```powershell
netstat -ano | findstr :3000   # 或 :8080
taskkill /PID <PID> /F
```

</details>

<details>
<summary><strong>Go module相关错误</strong></summary>

```bash
# 设置代理
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"

# 清理 Go module 缓存
go clean -modcache

# 重新安装依赖
cd ai_agents/agents/examples/websocket-example
task install
```

</details>

## 下一步

跟随这些指南，将您的语音 AI 应用带入真实世界。

* **[扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext)** - 开发自定义扩展
* **[更多示例](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - 探索其他 Agent 示例
* **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - 报告问题或请求功能
* **[文档中心](https://theten.ai/cn/docs)** - 完整的 TEN Framework 文档


# 图（Graph） (/cn/docs/ten_framework/graph/graph)

在 TEN framework 中，存在两种类型的图：

1. 动态图
2. 预定义图（`predefined_graph`）

|       | 动态图                          | 预定义图                                      |
| ----- | ---------------------------- | ----------------------------------------- |
| 启动时机  | 当 TEN 应用收到 `start_graph` 命令时 | 当 TEN 应用启动时，或当 TEN 应用收到 `start_graph` 命令时 |
| 图的内容  | 在 `start_graph` 命令中指定        | 在 TEN 应用的属性中预先定义                          |
| 图的 ID | 随机生成的 UUID                   | 随机生成的 UUID                                |

![两种类型的图](https://ten-framework-assets.s3.amazonaws.com/doc-assets/two_types_of_graph.png)

预定义图具有 `auto_start` 属性，用于确定该图是否在 TEN 应用启动时自动启动。

此外，预定义图还有 `singleton` 属性，用于指示该图是否只能在 TEN 应用内生成 *一个* 对应的实例。

## 图 ID 和图名称

对于每个图实例，TEN 应用内部会使用一个唯一的 UUID4 字符串作为标识，称为**图 ID**。

对于预定义图，可以分配一个有意义且易于记忆的名称，称为**图名称**。当需要指定特定预定义图时，可以直接使用其图名称。如果预定义图具有 `singleton` 属性，则表示该预定义图在 TEN 应用内只能存在一个实例。在这种情况下，TEN runtime 会使用图名称来唯一标识从该预定义图生成的单一实例。

## 动态图

当 TEN 应用收到 `start_graph` 命令并创建动态图时，系统会分配一个随机 UUID 作为该图的 ID。如果其他客户端获取了此图 ID，它们也可以连接到这个图。

动态图 ID 示例：

`123e4567-e89b-12d3-a456-426614174000`

## 预定义图

预定义图与动态图的工作原理相似，主要区别在于内容定义方式。动态图的内容包含在 `start_graph` 命令中，而预定义图的内容则由 TEN 应用预先定义。客户端只需在 `start_graph` 命令中指定预定义图的名称即可启动该图。

预定义图的主要优势在于简化使用并保护敏感信息。使用预定义图，客户端无需了解图的详细结构，这既提高了可用性，也避免了暴露图中可能包含的敏感信息。

预定义图名称示例：

`http_server`

当 TEN 应用启动时，所有设置了 `auto_start` 属性的预定义图会自动启动。

## 图定义

无论是动态图还是预定义图，其定义结构都是相同的：

```json
{
  "nodes": [
    // 节点的定义
  ],
  "connections": [
    // 通信链路的定义
  ]
}
```

关键要点：

1. 如果只有一个 TEN 应用，可以省略 `app` 字段。如果有多个应用，则必须明确指定 `app` 字段。

2. `nodes` 字段定义图中的节点，如各种扩展（extension）。

3. 每个节点在 `nodes` 字段中只能出现一次。如果多次出现同一节点，TEN framework 会在验证过程中报错。

4. 在 `nodes` 字段中定义扩展的方式如下，其中 `property` 字段为可选：

   ```json
   {
     "type": "extension",
     "name": "simple_http_server_cpp",
     "addon": "simple_http_server_cpp",
     "extension_group": "default_extension_group",
     "app": "msgpack://127.0.0.1:8001/",
     "property": {
       "root_key": "player",
       "extra_keys": ["playerName"]
     }
   }
   ```

   `addon` 字段表示该扩展是由对应插件生成的实例。

5. `connections` 字段定义图中节点间的通信链路，其中 `extension` 值表示对应节点的名称。

完整示例：

```json
{
  "nodes": [
    {
      "type": "extension",
      "app": "msgpack://127.0.0.1:8001/",
      "name": "simple_http_server_cpp",
      "addon": "simple_http_server_cpp",
      "extension_group": "default_extension_group",
      "property": {
        "root_key": "player",
        "extra_keys": ["playerName"]
      }
    }
  ],
  "connections": [
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "simple_http_server_cpp",
      "cmd": [
        {
          "name": "start",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        },
        {
          "name": "stop",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        }
      ]
    },
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "gateway",
      "cmd": [
        {
          "name": "push_status_online",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "uap"
            }
          ]
        }
      ]
    }
  ]
}
```

## 预定义图的定义

预定义图是在 TEN 应用的配置文件（`property.json`）中的 `predefined_graphs` 字段下定义的，包含 `name`，`auto_start`，`singleton` 等属性：

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        // 完整的图定义
      }
    ]
  }
}
```

完整示例：

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        "nodes": [
          {
            "type": "extension",
            "name": "simple_http_server_cpp",
            "addon": "simple_http_server_cpp",
            "extension_group": "default_extension_group",
            "property": {
              "root_key": "player",
              "extra_keys": [
                "playerName"
              ]
            }
          }
        ],
        "connections": [
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "simple_http_server_cpp",
            "cmd": [
              {
                "name": "start",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              },
              {
                "name": "stop",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              }
            ]
          },
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "gateway",
            "cmd": [
              {
                "name": "push_status_online",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "uap"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## `start_graph` 命令的定义

动态图是通过向 TEN 应用发送 `start_graph` 命令创建的，将图定义放在命令的 `ten` 字段下：

```json
{
  "ten": {
    "type": "start_graph",
    // 完整的图定义
  }
}
```

完整示例：

```json
{
  "ten": {
    "type": "start_graph",
    "nodes": [
      {
        "type": "extension",
        "name": "simple_http_server_cpp",
        "addon": "simple_http_server_cpp",
        "extension_group": "default_extension_group",
        "property": {
          "root_key": "player",
          "extra_keys": ["playerName"]
        }
      }
    ],
    "connections": [
      {
        "app": "msgpack://127.0.0.1:8001/",
        "extension": "simple_http_server_cpp",
        "cmd": [
          {
            "name": "start",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          },
          {
            "name": "stop",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          }
        ]
      },
      {
        "extension": "gateway",
        "cmd": [
          {
            "name": "push_status_online",
            "dest": [
              {
                "extension": "uap"
              }
            ]
          }
        ]
      }
    ]
  }
}
```

### 使用 `import_uri` 引用图定义

在定义预定义图时，可以使用 `import_uri` 字段引用外部图定义文件：

```json
{
  "ten": {
    "uri": "http://localhost:8001",
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": false,
        "import_uri": "../graph.json"
      }
    ]
  }
}
```

`import_uri` 可以是相对路径，绝对路径或 URL：

* 相对路径：相对于 `property.json` 文件所在目录
* 绝对路径：相对于 TEN 应用所在环境的根目录
* URL：直接作为 URL 路径处理

## 图定义的规范

* **`nodes` 字段**：在图定义中必须提供 `nodes` 数组。相比之下，`connections` 数组虽然可选，但建议提供以定义节点间通信。

* **节点 `app` 字段**：`app` 字段不能设置为 `localhost`。在单应用图中，应省略 `app` URI；在多应用图中，`app` 字段值必须与各应用 `property.json` 中的 `ten.uri` 值匹配。

* **节点唯一性**：`nodes` 数组中的每个节点表示一个特定扩展实例，必须通过 `app` 和 `name` 组合唯一标识。不允许出现重复定义，如以下无效示例：

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_1",
        "extension_group": "test"
      },
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_2",
        "extension_group": "test"
      }
    ]
  }
  ```

* **连接定义的一致性**：`connections` 中引用的所有扩展实例必须在 `nodes` 中明确定义。以下示例无效，因为 `ext_2` 未在 `nodes` 中定义：

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "ext_1",
        "addon": "addon_1",
        "extension_group": "some_group"
      }
    ],
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **通信链路整合**：同一源扩展的所有消息应分组在单个部分中，不应分散在多个定义中。以下示例不正确：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      },
      {
        "extension": "ext_1",
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  正确做法是将同一源扩展的所有消息整合在一起：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ],
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **消息目标整合**：对于特定类型的每条消息，所有目标扩展应归类在单个条目下。以下示例不正确：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          },
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  正确做法是将同一消息的所有目标整合在一起：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              },
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  注意，相同名称的消息可以在不同类型（如 `cmd` 和 `data`）中共存，不会导致冲突。

## 子图（Subgraph）

TEN framework 的核心机制是基于图结构，这种结构由节点和连接构成。子图是一种强大的复用机制，可以将复杂的图结构拆分为可重用的模块，从而提升代码的组织性和可维护性。

### 子图设计理念

图的本质是定义扩展之间数据流动的方式。子图在实现上并没有改变这一基本原理，而是作为一种语法糖，最终会被展平到所属的大图中，并使用与普通图相同的机制启动。这种设计既简化了复杂系统的开发，又不增加运行时的复杂度。

#### 设计原则

子图设计遵循以下核心原则：

1. **独立性**：任何一张图都是一个完整的图，可以单独运行，也可以作为组件嵌入到其他图中。

2. **工具友好**：子图提供额外的信息，帮助开发工具更好地理解图结构，提升开发体验，同时不增加运行时的复杂度。

3. **展平机制**：子图最终会被展平为标准的图结构，以确保性能和兼容性。

4. **简洁性**：子图的设计旨在简化开发过程，而不是增加复杂性，避免增加 JSON 结构的复杂度。

子图的设计目的是让开发者可以将一个图当作黑盒使用，无需关注其内部的复杂性。因此，在引用子图的地方，不提供特殊的修补机制来调整子图内部的状态，以减少复杂性。如果需要修改子图的定义，应直接修改原始定义，而不是在引用处进行修补。这么做是为了避免复杂性扩散，增加维护成本。例如，当将子图 A 引入到大图 B 时，如果需要调整子图 A 内部的 extension 属性的 schema 定义，应直接修改该 extension 的 schema 定义，而不是在引用处提供修改机制，因为这可能会导致不良的连锁反应，只修改了 extension 的 schema 定义，而没有修补其他地方，从而导致其他地方出错。

换句话说，子图主要提供的是一种映射机制，用来将子图内的元素暴露给外部，而不是提供机制来修补子图内部的元素。

### 子图实现机制

接下来，我们将详细介绍子图的实现机制，包括子图的定义、引用方式和展平过程。

#### 子图定义示例

以下是 `subgraph.json` 示例，可作为独立图使用，也可作为子图被其他图引用：

```json
{
  "nodes": [
    {
      // 定义名为 ext_c 的扩展节点
      "type": "extension",
      "name": "ext_c",
      "addon": "extension_c"
    },
    {
      // 定义名为 ext_d 的扩展节点
      "type": "extension",
      "name": "ext_d",
      "addon": "extension_d"
    }
  ],
  "connections": [
    {
      // ext_c 将 B 命令传输到 ext_d
      "extension": "ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_d"
            }
          ]
        }
      ]
    }
  ],
  "exposed_messages": [
    // 表示图向外部暴露的消息接口, 主要供开发工具使用，提供智能提示
    {
      // ext_d 的 B 命令暴露给外部
      "extension": "ext_d",
      "type": "cmd_in",
      "name": "B"
    }
  ],
  "exposed_properties": [
    // 表示图向外部暴露的属性
    {
      "extension": "ext_c",
      "name": "a",
      "alias": "a1"
    }
  ]
}
```

#### 子图引用示例

引用子图的 `graph.json` 示例：

```json
{
  "nodes": [
    {
      // 定义名为 ext_a 的扩展节点
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // 定义名为 ext_b 的扩展节点
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 引用子图，命名为 subgraph_1
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "./ten_packages/extension/aaa/subgraph.json",
      "property": {
        // 可以写在这边的 property 字段, 必须要存在于子图的 exposed_properties 字段,
        // 不然会是一个错误情况
        "app_id": "${env:AGORA_APP_ID}",
        "token": "<agora_token>",
        "channel": "ten_agent_test"
      }
    },
    {
      // 引用子图，命名为 subgraph_2
      "type": "subgraph",
      "name": "subgraph_2",
      "import_uri": "./ten_packages/extension/bbb/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // 第一个目标是 ext_b
              "extension": "ext_b"
            },
            {
              // 第二个目标是子图中的 ext_d, 这算是进阶用法
              "extension": "subgraph_1:ext_d"
            },
            {
              // 第三个目标是子图, 但这需要子图有明确 expose 出 B 这个 cmd_in 接口, 不然
              // 会被当成是错误情况
              "subgraph": "subgraph_2"
            }
          ]
        }
      ]
    },
    {
      // 子图中的 ext_c 将 cmd H 传输给 ext_a, 这算是进阶用法
      "extension": "subgraph_1:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 子图 subgraph_2 将 cmd H 传输给 ext_a, 不过這需要子图 expose 出 H 这个
      // cmd_out 接口, 不然会被当成是错误情况
      "subgraph": "subgraph_2",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

#### 关键概念

子图机制引入三个关键概念：

1. **消息暴露**（exposed\_messages）：
   * 子图通过 `exposed_messages` 字段声明对外暴露的接口
   * 主要供开发工具使用，实现智能提示和检查
   * 隐藏子图内部细节，提升开发体验

2. **属性暴露**（exposed\_properties）：
   * 子图通过 `exposed_properties` 字段声明对外暴露的属性
   * 主要供开发工具使用，实现智能提示和检查
   * 隐藏子图内部细节，提升开发体验

3. **子图引用与命名**：
   * 通过 `"type": "subgraph"` 引用其他图文件
   * 每个子图有唯一标识名称，作为命名空间
   * 防止不同子图中的同名元素冲突

4. **跨图连接**：
   * 通过命名空间语法（如 `subgraph_1:ext_d`）引用子图内元素, 或者通过 `subgraph` 字段引用子图
   * 使子图内元素可与主图交互
   * 构建复杂的跨图消息流

总之，`exposed_messages` 以及 `exposed_properties` 字段描述了如何从图的边界向内流动消息，开发工具则利用此字段提供智能提示，便于开发者指定消息如何流向图的边界。

#### 展平机制

最终，引用子图的图会被展平为普通图结构，确保运行时统一性和高效性。不过这个展平的结果不会被用户看到，而是在 TEN runtime 或者 TEN manager 中被展平到内存中使用。注意在展平后的图定义中，就是原始的有哪些 extensions，以及在这些 extensions 之间的连线。

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 子图中的 ext_c 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "extension_c"
    },
    {
      // 子图中的 ext_d 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "extension_d",
      "property": {
        // 在引用子图时所定义的 property 字段, 会被展平到对应的 extension 的 property 字
        // 段
        "app_id": "${env:AGORA_APP_ID}"
      }
    },
    {
      // 子图中的 ext_e 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "extension_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b"
            },
            {
              "extension": "subgraph_1_ext_d"
            },
            {
              "extension": "subgraph_2_ext_e"
            }
          ]
        }
      ]
    },
    {
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 子图内部连接也被展平纳入
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "subgraph_1_ext_d"
            }
          ]
        }
      ]
    }
  ]
  // exposed_messages 以及 exposed_properties 字段在展平过程中被舍弃
}
```

子图展平机制遵循以下规则：

1. 展平前，冒号（`:`）符号表示元素位于子图中（如 `subgraph_1:ext_d`）。

2. 展平后，子图中元素名称添加子图名称作为前缀（如 `subgraph_1_ext_c`），确保全局唯一性。

3. 展平后的图定义不再包含冒号符号，以区分展平前后的状态。

4. 子图中的内部连接会保留并纳入展平后的图中，保证功能完整性。

### 高级功能与应用

随着项目复杂度增加，子图的高级功能可帮助更好地组织和管理系统。

#### 消息转换与子图

子图完全支持消息转换（msg\_conversion）机制，用于处理不同接口间的消息格式转换：

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "http://a.b.c.d/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1:ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "subgraph": "subgraph_2",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

开发工具可利用 `exposed_messages` 信息，提示兼容性并提供消息转换配置界面。配置完成后，转换规则会自动写入图定义，简化开发流程。

展平后，消息转换规则会正确保留，确保运行时行为与设计意图一致：

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "addon_c"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "addon_d"
    },
    {
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "addon_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1_ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_2_ext_e",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

通过这种方式，开发者可灵活处理不同组件间的消息格式差异，无需关心底层实现细节。

## "subgraph\_1:ext\_c" 的意义

在前面的示例中，`subgraph_1:ext_c` 中的 `subgraph_1` 不是表示子图本身，而是引用 `nodes` 字段中的一个元素。在子图应用中，`subgraph_1` 可视为一个命名空间。

通过这种机制，可以在图的连接中指定不同位置的扩展作为源或目标，实现跨多个图的连接。

核心概念是：图的连接本质上是扩展之间的连接。通过这种机制，可以在 `nodes` 字段中指定不同类型的元素作为连接的源或目标，实现跨多个子图甚至多个图的连接。

## 跨多个图的连接

除了子图外，还可以通过类似机制在一个图中指定与其他图的连接。

### 与同一 TEN 应用中的预定义图连接

示例：如何在一个图中连接到同一 TEN 应用中的预定义图：

```json
{
  "nodes": [
    {
      // 定义名为 ext_a 的扩展
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // 定义名为 ext_b 的扩展
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 引用其他图, 图 ID 是 default
      "type": "graph",
      "graph_id": "default"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // 第一个目标是 ext_b
              "extension": "ext_b"
            },
            {
              // 第二个目标是另一张图中的 ext_d
              "extension": "default:ext_d"
            }
          ]
        }
      ]
    },
    {
      // 另一张图中的 ext_c 将 cmd H 传输给 ext_a
      "extension": "default:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 另一张图将 cmd X 传输给 ext_a, 但这需要另一张图 expose 出 X 这个 cmd_out 接口,
      // 不然会被当成是错误情况. 这个错误情况如果可以被静态检查出来, 则会在静态检查的时候报
      // 错, 或者在运行时报错.
      "graph": "default",
      "cmd": [
        {
          "name": "X",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

### graph node 的种类

1. Subgraph

   ```json
   {
     "type": "subgraph",
     "name": "subgraph_1", // namespace 的意义
     "import_uri": "http://a.b.c.d/subgraph.json"
   }
   ```

2. Local graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": true
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": false
   }
   ```

3. Remote graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": true,
     "app": "msgpack:://127.0.0.1:8002/"
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": false,
     "app": "msgpack:://127.0.0.1:8002/"
   }
   ```


# 消息转换 (/cn/docs/ten_framework/graph/msg_conversion)

## 消息转换

TEN Framework 提供了强大的消息转换功能，允许您在消息传递过程中对消息内容进行转换和修改。通过配置转换规则，您可以实现消息字段的重命名、值的替换、以及复杂的数据结构转换。

### 基本配置

消息转换通过 `msg_conversion` 配置项进行设置：

```json
{
  "msg_conversion": {
    "type": "per_property",
    "keep_original": false,
    "rules": [
      {
        "path": "user.name",
        "conversion_mode": "fixed_value",
        "value": "default_user"
      },
      {
        "path": "data.items[0].id",
        "conversion_mode": "from_original",
        "original_path": "original_id"
      }
    ]
  }
}
```

### 配置参数说明

* **type**: 转换类型，目前支持 `per_property`（按属性转换）
* **keep\_original**: 是否保留原始消息内容，设为 `false` 时只保留转换后的字段
* **rules**: 转换规则数组，定义具体的转换逻辑

### 转换规则

每个转换规则包含以下字段：

* **path**: 目标字段路径，指定要设置的消息字段位置
* **conversion\_mode**: 转换模式
  * `fixed_value`: 设置固定值
  * `from_original`: 从原始消息的指定路径复制值
* **value**: 当转换模式为 `fixed_value` 时使用的固定值
* **original\_path**: 当转换模式为 `from_original` 时，指定原始消息中的源字段路径

### 路径表达式

消息转换中的路径表达式支持多种访问方式：

* **对象属性访问**: `user.profile.name` - 访问嵌套对象的属性
* **数组索引访问**: `items[0].name` - 访问数组指定索引的元素
* **混合路径访问**: `user.tags[1].value` - 结合对象和数组访问

### 使用场景

消息转换功能适用于以下场景：

* **字段重命名**: 将消息中的字段名称转换为目标系统要求的格式
* **数据格式转换**: 调整消息的数据结构以适配不同的处理模块
* **默认值设置**: 为缺失的字段提供默认值
* **数据清洗**: 过滤或转换不符合要求的数据内容


# Go (/cn/docs/ten_framework/language_binding/go)

让我们使用 TEN framework 构建一个 Go 应用程序。

## 创建 TEN 应用

首先，我们将通过集成几个预构建的 TEN 包来创建一个基本的 TEN Go 应用。请按照以下步骤操作：

```shell title="终端"
tman install app default_app_go
cd default_app_go

tman install protocol msgpack
tman install extension_group default_extension_group
```

### 安装默认的 TEN 扩展

接下来，安装一个用 Go 编写的默认 TEN 扩展：

```shell title="终端"
tman install extension default_extension_go
```

### 声明预构建的自动启动图

现在，我们将修改 TEN 应用的 `property.json` 文件，以包含图的声明。这将确保默认扩展在 TEN 应用启动时自动启动。

```json title="property.json"
"predefined_graphs": [
  {
    "name": "default",
    "auto_start": true,
    "nodes": [
      {
        "type": "extension_group",
        "name": "default_extension_group",
        "addon": "default_extension_group"
      },
      {
        "type": "extension",
        "name": "default_extension_go",
        "addon": "default_extension_go",
        "extension_group": "default_extension_group"
      }
    ]
  }
]
```

## 构建应用

与标准的 Go 项目不同，TEN Go 应用使用 CGo，因此您需要在构建之前设置某些环境变量。TEN runtime Go 绑定系统包中已经提供了一个构建脚本，因此您可以使用一个命令构建应用：

```shell title="终端"
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```

编译后的二进制文件 `main` 将在 `/bin` 文件夹中生成。

## 启动应用

由于需要设置一些环境变量，建议使用提供的脚本来启动应用：

```shell title="终端"
./bin/start
```

## 调试

如果您使用 Visual Studio Code (VSCode) 作为开发环境，则可以使用以下配置来调试 Go/C 代码。

### 调试 Go 代码

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Go app (launch)",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "cwd": "${workspaceFolder}",
      "output": "${workspaceFolder}/bin/tmp",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

### 调试 C 代码

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "C (launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/main",
      "cwd": "${workspaceFolder}",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

## CGO

### 生成的代码

当 Go 与 C 接口时，`cgo` 工具至关重要。当从 Go 调用 C 函数时，`cgo` 会将 Go 源文件转换为多个输出文件，包括 Go 和 C 源文件。然后，使用 gcc 或 clang 等编译器编译这些生成的 C 源文件。

使用以下命令生成这些 C/Go 源文件：

```bash title="终端"
mkdir out
go tool cgo -objdir out
```

示例：

```bash title="终端"
go tool cgo -objdir out cmd.go error.go
```

生成的文件包括：

```bash title="终端"
├── _cgo_export.c
├── _cgo_export.h
├── _cgo_flags
├── _cgo_gotypes.go
├── _cgo_main.c
├── _cgo_.o
├── cmd.cgo1.go
├── cmd.cgo2.c
├── error.cgo1.go
└── error.cgo2.c
```

* **`_cgo_export.h`** 是 cgo 上下文中 Go 和 C 之间互操作的关键。它包含可以从 C 访问的 Go 函数的必要声明。

  此文件是根据使用 `//export` 指令显式导出的 Go 函数自动生成的。它还包括与这些函数中使用的 Go 类型对应的 C 类型的定义，确保两种语言之间的兼容性。

  类型定义示例：

```c title=".h"
  typedef signed char GoInt8;
  typedef unsigned char GoUint8;
  typedef short GoInt16;
  typedef unsigned short GoUint16;
  typedef int GoInt32;
  typedef unsigned int GoUint32;
  typedef long long GoInt64;
  typedef unsigned long long GoUint64;
  typedef GoInt64 GoInt;
  typedef GoUint64 GoUint;
  typedef size_t GoUintptr;
  typedef float GoFloat32;
  typedef double GoFloat64;
```

* **`_cgo_gotypes.go`** 包含 C 中定义并在 Go 中使用的相应 Go 类型。

  例如，如果您在头文件 `common.h` 中定义了一个结构体 `ten_go_error_t` 并在 Go 中使用 `C.ten_go_error_t`，则在 `_cgo_gotypes.go` 中会有一个相应的 Go 类型：

```go title=".go"
  package ten

  type _Ctype_ten_go_status_t = _Ctype_struct_ten_go_status_t

  type _Ctype_struct_ten_go_status_t struct {
      err_no   _Ctype_int
      msg_size _Ctype_uint8_t
      err_msg  [256]_Ctype_char
      _        [3]byte
  }
```

* **`cmd.cgo1.go`** 是 `cgo` 生成的与原始 Go 源文件 `cmd.go` 对应的文件。它将对 C 函数和类型的直接调用替换为对 `cgo` 提供的生成的 Go 函数和类型的调用。

  示例：

```go title=".go"
  package ten

  func NewCmd(cmdName string) (Cmd, error) {
      // ...
      cStatus := func() _Ctype_struct_ten_go_status_t {
          var _cgo0 _cgo_unsafe.Pointer = unsafe.Pointer(unsafe.StringData(cmdName))
          var _cgo1 _Ctype_int = _Ctype_int(len(cmdName))
          _cgoBase2 := &bridge
          _cgo2 := _cgoBase2
          _cgoCheckPointer(_cgoBase2, 0 == 0)
          return _Cfunc_ten_go_cmd_create_custom_cmd(_cgo0, _cgo1, _cgo2)
      }()
      // ...
  }
```

* **`cmd.cgo2.c`** 是从 Go 调用的原始 C 函数的包装器。

  示例：

```c title=".c"
  CGO_NO_SANITIZE_THREAD void _cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd(void *v) {
      struct {
          const void* p0;
          int p1;
          char __pad12[4];
          ten_go_msg_t** p2;
          ten_go_error_t r;
      } __attribute__((__packed__, __gcc_struct__)) *_cgo_a = v;
      char *_cgo_stktop = _cgo_topofstack();
      __typeof__(_cgo_a->r) _cgo_r;
      _cgo_tsan_acquire();
      _cgo_r = ten_go_cmd_create_cmd(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2);
      _cgo_tsan_release();
      _cgo_a = (void*)((char*)_cgo_a + (_cgo_topofstack() - _cgo_stktop));
      _cgo_a->r = _cgo_r;
      _cgo_msan_write(&_cgo_a->r, sizeof(_cgo_a->r));
  }
```

因此，从 Go 调用 `C.ten_go_cmd_create_cmd()` 的调用顺序为：

```text
_Cfunc_ten_go_cmd_create_custom_cmd (自动生成的 Go)
           |
           V
   _cgo_runtime_cgocall (Go)
           |
           V
      entrysyscall     // 将 Go 堆栈切换到 C 堆栈
           |
           V
_cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd (自动生成的 C)
           |
           V
   ten_go_cmd_create_cmd (C)
           |
           V
      exitsyscall
```

### 不完整类型

如前所述，`cgo` 工具会根据通过 `import "C"` 导入的 C 头文件在 `_cgo_gotypes.go` 中生成相应的 Go 类型。如果结构体在 C 头文件中是不透明的，则 `cgo` 工具会生成一个不完整类型。

不透明 C 结构体的示例：

```c title=".h"
typedef struct ten_go_msg_t ten_go_msg_t;
```

`cgo` 工具将在 Go 中生成一个不完整类型：

```go title=".go"
type _Ctype_ten_go_msg_t = _Ctype_struct_ten_go_msg_t
type _Ctype_struct_ten_go_msg_t _cgopackage.Incomplete
```

#### 如果在 Go 中使用不完整类型会发生什么？

* **无法分配不完整类型。**

  由于 `sys.NotInHeap` 无法在 Go 堆上分配，因此 `new` 或 `make` 等操作将不起作用。尝试在 Go 中创建不透明结构体的新实例将导致编译器错误：

```go title=".go"
  msg := new(C.ten_go_msg_t) // 错误：无法在 Go 中分配；它是不完整的（或不可分配的）
```

* **无法将指向不完整类型的指针直接传递给 C。**

  如果您的 C 函数具有指向不透明结构体的指针作为参数，则根据 cgo 规则，将 Go 指针直接传递给此不完整类型的 C 函数将不起作用。Go 编译器将要求指针“固定”，以确保它符合 Go 的垃圾收集器 (GC) 约束。

### 使用 `C.uintptr_t` 而不是指向不透明结构体的指针的规则

* **优点：**

  * Go 中的 `C.uintptr_t` 和 `uintptr` 是足够大的整数，可以保存 C 指针，从而避免从 Go 传递到 C 时进行内存分配或转换。

* **限制：**

  * `uintptr` 是一个整数，而不是指针。它表示没有指针运算的地址的位模式。
  * `uintptr` 无法在 Go 中解引用。将其转换为 `unsafe.Pointer` 可能会导致 Go 的 GC 出现问题。
  * 由于 `uintptr` 是一个整数，因此无法将 `nil` 或 `NULL` 传递给 C。使用 `0` 而不是 `nil` 来表示空地址。


# Manifest (/cn/docs/ten_framework/metadata_system/manifest)

## 什么是 Manifest 文件

Manifest 文件是 TEN Framework 中用于存储软件包元数据的重要配置文件。它定义了扩展（Extension）或应用（App）的基本信息、依赖关系、API 接口、脚本配置等关键信息。

## Manifest 文件的作用

* **包标识**：定义包的类型、名称和版本信息
* **依赖管理**：声明包所需的依赖项及其版本要求
* **API 定义**：描述包对外提供的数据接口和命令接口
* **打包配置**：指定需要包含在发布版本中的文件
* **脚本管理**：定义构建、测试等自动化脚本

## Extension Manifest 示例

以下是一个 AI 聊天扩展的完整 Manifest 示例：

```json
{
  "type": "extension",
  "name": "openai_chatgpt_python",
  "version": "0.1.0",
  "tags": ["ai", "llm", "python"],
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "^0.8.0"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "*.py",
      "README.md"
    ]
  },
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        }
      }
    },
    "data_in": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "data_out": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_in": [
      {
        "name": "flush"
      },
      {
        "name": "tool_register",
        "property": {
          "properties": {
            "tool": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "description": {
                  "type": "string"
                },
                "parameters": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                }
              },
              "required": ["name", "description", "parameters"]
            }
          }
        },
        "result": {
          "property": {
            "properties": {
              "response": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "tool_call",
        "property": {
          "properties": {
            "name": {
              "type": "string"
            },
            "args": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    ]
  },
  "scripts": {
    "test": "python -m pytest tests/",
    "build": "python setup.py build"
  }
}
```

## App Manifest 示例

以下是一个语音助手应用的 Manifest 示例：

```json
{
  "type": "app",
  "name": "voice_assistant_app",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "^0.8.0"
    }
  ],
  "api": {
    "property": {
      "properties": {
        "app_id": {
          "type": "string"
        },
        "log_level": {
          "type": "uint8"
        },
        "server_config": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "int32"
            },
            "ssl_enabled": {
              "type": "bool"
            }
          },
          "required": ["host", "port"]
        }
      }
    }
  }
}
```

## 主要字段说明

### 基本信息

* **type**：包的类型，可以是 `extension` 或 `app`
* **name**：包的唯一标识名称
* **version**：遵循语义化版本规范的版本号
* **tags**：用于分类和搜索的标签数组

### 依赖配置

* **dependencies**：依赖项列表，每个依赖包含类型、名称和版本要求

### 打包配置

* **package.include**：指定打包时需要包含的文件和目录

### API 配置

* **api.property**：定义扩展的配置属性
* **api.data\_in/data\_out**：定义数据输入和输出接口
* **api.cmd\_in/cmd\_out**：定义命令输入和输出接口

### 脚本配置

* **scripts**：定义各种自动化脚本，如测试、构建等

## 最佳实践

1. **版本管理**：使用语义化版本控制，确保版本号的一致性
2. **依赖声明**：明确声明所有必需的依赖项及其版本范围
3. **API 设计**：清晰定义输入输出接口，便于其他组件集成
4. **文档完整**：确保 Manifest 中的配置与实际实现保持一致


# 元数据系统 (/cn/docs/ten_framework/metadata_system/metadata_system)

TEN framework 在各种类型的包中采用一致的元数据系统。

## 元数据类型

在 TEN 元数据系统中，有两种主要类型的元数据：

### 1. **Manifest（清单）**

* **位置：** 存储在相关 TEN 包的根目录中，文件名为 `manifest.json`。
* **内容：**

  * **包名称：** TEN 包的名称。
  * **包版本：** TEN 包的版本，遵循语义版本控制。
  * **API Schema：** 定义与包的属性和输入/输出消息相关的模式。

  > ⚠️ **注意：** `manifest.json` 中的 API schema *不是* JSON schema。

### 2. **Property（属性）**

* **位置：** 通常存储在 TEN 包根目录的 `property.json` 文件中。
* **目的：** `property.json` 文件存储初始属性值，这些属性值在运行时是可读写的。这意味着可以在 TEN runtime 运行时修改属性。

## 清单文件 (Manifest File)

`manifest.json` 文件充当 TEN 包的蓝图，定义其元数据、属性以及它处理的输入/输出消息。

### `manifest.json` 示例

```json
{
  "type": "app",
  "name": "default_app_cpp",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.0"
    }
  ],
  "api": {
    "property": {
      "exampleInt8": {
        "type": "int8"
      },
      "exampleString": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "detail": {
              "type": "string"
            },
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

### 清单中 TEN 模式的目的

`manifest.json` 中的 TEN 模式为 TEN runtime 提供了关于 extension 的外部 API 的元数据，包括属性和消息的类型信息。这有助于运行平台在操作期间正确处理 extension 的属性和消息。

### 何时使用 TEN 模式

1. **属性验证：** 当 TEN runtime 获取/设置 TEN 包或消息的属性时。
2. **数据转换：** 使用 `from_json` API 将 JSON 文档转换为 TEN 包或消息属性时。
3. **兼容性检查：** 当 TEN 管理器根据 TEN 模式检查消息是否可以从源 extension 输出到目标 extension 时。

## 属性管理

在 TEN framework 中，有两种类型的属性：

1. **消息属性：** 这些属性与框架中 extension 之间交换的消息相关联。消息属性定义消息中携带的特定数据或参数，例如命令参数、数据负载或与音频/视频帧相关的元数据。
2. **TEN 包属性：** 这些属性与 TEN 包本身（例如 extension）相关联。TEN 包属性定义特定于包的配置或设置。例如，extension 可能具有配置其行为的属性，例如运行时设置、初始化参数或其他配置数据。

这两种类型的属性都在 TEN framework 中进行管理，但服务于不同的目的：一种侧重于组件之间的通信（消息属性），另一种侧重于组件本身的配置和操作（TEN 包属性）。

![属性系统](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_system.png)

### 定义 TEN 包属性

`property.json` 文件定义 TEN 包的属性。以下是一个示例：

```json
{
  "prop_1_name": 0,
  "prop_2_name": "prop_2_value",
  "prop_3_name": ["hello", "prop_3_sub_value"],
  "prop_4_name": {
    "prop_4_1_name": 1,
    "prop_4_2_name": "prop_4_sub_value"
  }
}
```

> ⚠️ **注意：** `property.json` 文件中的每个属性名称必须是唯一的。

### 属性的 TEN 模式

您可以在 `manifest.json` 文件中为属性定义 TEN 模式，使 TEN runtime 能够更有效地处理这些属性。如果属性没有相应的 TEN 模式，则运行平台将使用默认的 JSON 处理方法（例如，将所有 JSON 数字视为 `float64`）。如果提供了 TEN 模式，则运行平台将使用它来验证和相应地处理属性。

| 属性 | TEN 模式 | 效果                                          |
| -- | ------ | ------------------------------------------- |
| 是  | 是      | TEN runtime 根据 TEN 模式验证属性值（例如，检查类型）。        |
| 是  | 否      | TEN runtime 使用默认处理，将所有 JSON 数字视为 `float64`。 |

### 从 Extension 访问属性

TEN runtime 为 extension 提供了访问各种属性的 API。

### 从 Extension 访问 App 的属性

> ⚠️ **注意：** 本节所描述的行为还未被实现，是未来的 roadmap。

TEN framework 提供了强大的属性访问机制，让 extension 能够读取和修改 app 的属性。通过 get/set property API，您可以访问所有属性，包括 app 的属性。当 API 参数以 `app:` 前缀开头时，系统会识别这是对 app 属性的操作请求。

各语言实现的 API 设计考量：

* **Go**：提供不带 callback 的同步版本即可，简洁高效。
* **Node.js**：实现为不带 callback 的 async function，符合 Node.js 的异步编程模型。
* **Python**：理想情况下应提供带 callback 和不带 callback 的版本。目前可先实现不带 callback 的同步版本，但在访问 `app:` 属性时会使用 event wait，可能影响性能。
* **Async Python**：实现为不带 callback 的 async function，与 Python 异步模型一致。
* **C++**：应提供带 callback 和不带 callback 的版本。目前可先实现不带 callback 的同步版本，但在访问 `app:` 属性时会使用 event wait，可能影响性能。

这种设计使 app 属性的增删改查通过统一的 get/set\_property API 实现，保持了接口的一致性，且不会影响 message 属性的相关 API。

关于 `ten:` 命名空间字段的处理，采用以下规则：

1. `get/set_property_to/from_json(nullptr)` 操作不处理 `ten:` 字段
2. 要访问 `ten:` 字段，必须明确指定如 `get/set_property("ten:xxx")` 的完整路径
3. 当使用 `get/set_property_to/from_json()` 时，只有明确指定 `"ten:xxx"` 形式的 key 时才会处理 `ten:` 字段

这种设计既保持了 API 的简洁性，又避免了为特定需求创建额外命令的复杂性。

### 在 `start_graph` 命令中指定属性值

与 TEN 包相关的属性值可以在 `start_graph` 命令中指定。TEN runtime 根据 TEN 模式（如果已定义）处理这些属性，并将它们存储在相应的 TEN 包实例中。

#### `start_graph` 命令中属性规范的示例

```json
{
  "nodes": [
    {
      "type": "extension_group",
      "name": "foo_extension_group",
      "addon": "foo_extension_group_addon"
    },
    {
      "type": "extension",
      "name": "bar_extension",
      "extension_group": "foo_extension_group",
      "property": {
        "prop_1_name": 0,
        "prop_2_name": "prop_2_value",
        "prop_3_name": ["hello", "prop_3_sub_value"],
        "prop_4_name": {
          "prop_4_1_name": 1,
          "prop_4_2_name": "prop_4_sub_value"
        }
      }
    }
  ]
}
```


# Property (/cn/docs/ten_framework/metadata_system/property)

## Property 文件概述

Property 文件是 TEN Framework 中用来存储扩展（extension）属性值的配置文件。它定义了扩展的运行时参数和配置选项，是扩展正常运行的重要组成部分。

## Property 文件格式

Property 文件采用 JSON 格式，包含扩展运行所需的各种配置参数，如 API 密钥、模型配置、运行参数等。

### 基础属性示例

以下是一个简单的 Property 文件示例：

```json
{
  "api_key": "your-api-key-here",
  "max_tokens": 1000,
  "temperature": 0.7,
  "model_config": {
    "name": "gpt-4",
    "version": "2024-01-01"
  },
  "allowed_languages": ["en", "zh", "es"]
}
```

### AI 模型配置示例

针对 AI 模型扩展的 Property 文件配置示例：

```json
{
  "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxx",
  "model": "gpt-4",
  "max_tokens": 4096,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "prompt": "You are a helpful AI assistant.",
  "greeting": "Hello! How can I help you today?",
  "max_memory_length": 10,
  "vendor": "openai"
}
```

## 图配置

Property 文件还可以包含图配置（Graph Configuration），定义扩展之间的连接关系和数据流。

### 预定义图配置示例

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "nodes": [
          {
            "type": "extension",
            "name": "my_extension",
            "addon": "my_extension_addon",
            "property": {
              "setting1": "value1",
              "setting2": 42
            }
          }
        ],
        "connections": [
          {
            "extension": "source_extension",
            "cmd": [
              {
                "name": "process_cmd",
                "dest": [
                  {
                    "extension": "target_extension"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## 使用说明

* Property 文件必须是有效的 JSON 格式
* 文件中的属性值会在扩展启动时被加载
* 可以通过修改 Property 文件来调整扩展的运行行为
* 图配置部分定义了扩展间的连接和数据流向


# 插件系统 (/cn/docs/ten_framework/software_packages/addon_system)

TEN framework 是一个软件模块系统，其中每个软件模块都由 **插件 (addon)** 的概念定义。本质上，TEN framework 中的每个软件模块都是一个插件。例如，用户可以定义以下类型的 TEN framework 插件：

1. TEN **扩展 (extension)** 插件
2. TEN **扩展组 (extension group)** 插件
3. TEN **协议 (protocol)** 插件

## 加载插件所需的信息

要将插件加载到 TEN runtime，需要三个信息：

1. **插件名称**

   插件名称在整个 TEN 应用中必须全局唯一。将多个同名插件加载到 TEN runtime 是未定义行为，可能会导致第二个插件实例无法正确加载。

2. **插件文件夹位置**

   在 TEN framework 中，加载插件时，用户必须告知 TEN runtime 插件目录的所在位置。此目录将作为插件的 `base_dir`。TEN runtime 将使用此基本目录加载与插件相关的其他文件，例如插件的 `manifest.json` 和 `property.json` 文件。

   基本上，在设计 TEN framework 的插件注册 API 时，其目标是自动检测插件所在的目录，因此用户无需显式指定 `base_dir`。但是，在某些 TEN framework 无法自动检测 `base_dir` 的特殊情况下，用户将需要为 TEN runtime 显式指定插件所在的文件夹。

3. **插件实例本身**

## 将插件注册到 TEN runtime

有两种方法可以将插件注册到 TEN runtime ：

1. 显式调用 API 以将插件的 `base_dir` 注册到 TEN runtime。

   **注意：** 目前，只有独立测试框架提供此类 API。

2. 将插件放置在 TEN 应用文件夹树中的指定位置。

通过将插件放置在指定位置，TEN 应用将在启动时自动加载这些插件。目前，此指定位置位于 TEN 应用的 `ten_packages/` 目录下。如果插件以此格式放置，它们将在 TEN 应用启动时自动加载。

```text
.
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    └── protocol/
        └── <protocol_a>/
```


# 应用 (/cn/docs/ten_framework/software_packages/app)

应用程序是 TEN 世界中的容器，包含组和扩展。
应用程序可以作为放置在 Docker 容器内的进程（主要在云中使用）。
应用程序也可以作为嵌入在现有应用程序内的线程（在客户端或服务器上）。
⇒ TEN 可以在现有项目上执行本地化修改（小幅增强），而无需根据 TEN 重写整个项目。

![将 TEN 作为进程或线程运行](https://ten-framework-assets.s3.amazonaws.com/doc-assets/run_ten_as_process_or_thread.png)

![应用程序运行的不同模式](https://ten-framework-assets.s3.amazonaws.com/doc-assets/different_mode_of_app_runnings.png)

## TEN 应用程序文件夹结构

在 TEN framework 中，开发人员可以使用各种编程语言创建 TEN 软件包。无论使用哪种语言，TEN 应用程序的整体文件夹结构都保持一致，确保不同实现之间的一致性。

![TEN 软件包文件夹结构](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_package_folder_structure.png)


# 扩展 (/cn/docs/ten_framework/software_packages/extension)

## 生命周期

扩展的生命周期分为以下阶段：

1. `on_configure`
2. `on_init`
3. `on_start`
4. `on_stop`
5. `on_deinit`

在所有这些阶段，扩展都可以发送消息，如果发送的消息是命令类型，则扩展还可以接收其发送的命令的结果。换句话说，不存在扩展**不能**发送消息或接收其自身发送的命令结果的阶段。因此，在所有这些阶段，扩展都可以向其他扩展发送消息，这有助于在所有阶段维持扩展之间的依赖关系。

每个生命周期阶段都对应一个回调函数，并且有一个相应的 `on_xxx_done()` 函数来标记该生命周期阶段的结束。

| 生命周期回调         | 生命周期结束函数            |
| -------------- | ------------------- |
| `on_configure` | `on_configure_done` |
| `on_init`      | `on_init_done`      |
| `on_start`     | `on_start_done`     |
| `on_stop`      | `on_stop_done`      |
| `on_deinit`    | `on_deinit_done`    |

### 阶段转换图

```mermaid
graph TD;
  on_configure((on_configure));
  on_init((on_init));
  on_start((on_start));
  on_stop((on_stop));
  on_deinit((on_deinit));

  on_configure-->on_configure_done
  on_configure_done-->on_init;
  on_configure-->on_stop;
  on_init-->on_init_done;
  on_init_done-->on_start;
  on_init-->on_stop;
  on_start-->on_stop;
  on_start-->on_start_done;
  on_stop-->on_stop_done;
  on_stop_done-->on_deinit;
  on_deinit-->on_deinit_done;
```

### on\_configure

用于设置扩展属性的初始值。这允许在其他生命周期阶段中使用 `ten_env` 的 `get_property` API 来检索在 `on_configure()` 阶段设置的属性。调用 `on_configure_done()` 函数以标记 `on_configure` 阶段的结束。

```c++
  void on_configure(ten::ten_env_t &ten_env) override {
    // 设置扩展属性的初始值。
    ten_env.init_property_from_json(
      R"({
           "ten": {
             "uri": "msgpack://127.0.0.1:8001/",
             "log_level": 2
           }
         })", nullptr);
    ten_env.on_configure_done();
  }
```

### on\_init

用于初始化扩展。由于在 `on_init_done()` 之前扩展尚未准备就绪，TEN runtime 会令发送到扩展的所有消息排队，直到扩展准备就绪。准备就绪后，会将排队的消息传输给扩展。重要的是，如果结果来自扩展自身发送的命令，则*不受*此限制；TEN runtime 会将命令的结果直接传递回扩展。这是因为发送命令是扩展的自我启动操作，因此它必须准备好接收并处理其自身命令的结果。

### on\_start

此阶段标志着运行时开始向扩展发送消息。`on_start_done` 函数没有特殊用途，但为了与其他生命周期阶段保持一致并降低学习复杂性，TEN framework 包含了它。通常，`on_start_done` 在 `on_start` 回调函数的末尾调用。TEN runtime 将在收到其 `on_start_done()` 后开始将来自其他扩展的消息传递到此扩展。

```c++
  void on_start(ten::ten_env_t &ten_env) override {
    // 一些操作。
    ten_env.on_start_done();
  }
```

### on\_stop

在许多情况下需要停止扩展，例如当包含扩展的应用程序或引擎终止时。当扩展即将停止时，TEN runtime 会使用此 `on_stop` 回调通知扩展它已进入 `on_stop` 生命周期阶段。然后，扩展可以执行任何必要的终止操作。

### on\_deinit

在扩展调用 `on_stop_done()` 后，它将进入 `on_deinit` 阶段。在此阶段，由于扩展内的资源可能不再完全可用，因此 TEN runtime 不会将来自其他扩展的任何消息传递到此扩展。

## 不同生命周期阶段的扩展之间的关系

基本上，没有固有的关系。每个扩展都独立运行，在其自身的生命周期阶段之间切换。扩展是彼此独立的，它们之间的任何依赖关系都必须由扩展本身显式实现。TEN runtime 不进行任何假设或保证。例如，如果扩展 A 需要等待扩展 B 完成其初始化才能完成自身的初始化，则扩展 A 可以在其 `on_init` 期间向扩展 B 发送命令。一旦扩展 B 完成初始化并收到命令，它可以回复结果，当扩展 A 收到结果时，它可以调用 `on_init_done`。

```mermaid
sequenceDiagram
  participant A as 扩展 A
  participant B as 扩展 B

  Note over A,B: on_init
  activate A
  activate B
  deactivate B
  Note over B: on_init_done

  Note over B: on_start
  activate B
  deactivate B
  Note over B: on_start_done

  A->>B: cmd
  B-->>A: result

  deactivate A
  Note over A: on_init_done
```

基本上，在所有其他生命周期阶段，如果想要实现不同扩展的顺序，可以使用相同的方法来实现。

## 与 TEN runtime 的接口

扩展主要通过三个接口与 TEN runtime 交互：

1. **生命周期回调**

   * 这些包括像 `on_init`、`on_deinit`、`on_start` 和 `on_stop` 这样的回调。

2. **接收消息的回调**

   * 这些包括诸如 `on_cmd`、`on_data`、`on_audio_frame` 和 `on_video_frame` 之类的回调，用于处理传入扩展的消息。

3. **发送消息的函数**

   * 这些包括诸如 `send_cmd`、`send_data`、`send_audio_frame` 和 `send_video_frame` 之类的函数，用于将消息发送出扩展。

## 生命周期回调函数

扩展生命周期的不同阶段及其与消息处理的联系如下：

* **on\_init \~ on\_init\_done**: 处理扩展自身的初始化。在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。
* **on\_start \~ on\_start\_done**: 在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。由于属性在 `on_configure` 期间初始化，因此您可以执行依赖于这些属性设置的操作。但是，由于这仍然是初始化阶段的一部分，扩展将不会收到其他扩展主动发起的消息，避免了各种检查的需要。允许主动发送消息。
* **After on\_start\_done \~ on\_stop\_done**: 在此阶段，扩展可以正常发送和接收所有类型的消息及其结果。
* **on\_deinit \~ on\_deinit\_done**: 处理扩展的反初始化。与 `on_init` 阶段类似，在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。

## 使用不同语言实现扩展

在 TEN framework 中，可以使用各种语言（例如 C++、Go 和 Python）来实现扩展。开发者可以使用相同的概念方法以不同的语言实现扩展。学习如何使用一种语言开发扩展后，相对容易使用其他语言进行开发。

## 扩展中的异步消息处理

<figure><img src="../../..https://ten-framework-assets.s3.amazonaws.com/doc-assets/asynchronous_message_processing.png" alt=""><figcaption><p>异步消息处理</p></figcaption></figure>

扩展以异步方式处理消息。当 TEN runtime 通过 `on_cmd`、`on_data`、`on_audio_frame` 或 `on_video_frame` 等回调函数将消息传递给扩展时，并不要求扩展立即在回调函数内处理该消息。相反，扩展可以将消息委托给其他线程、进程，甚至其他机器进行处理。这可以充分利用多核和分布式计算资源。

处理完成后，可以通过 `send_cmd`、`send_data`、`send_audio_frame` 或 `send_video_frame` 等回调函数将结果发送回 TEN runtime。整个过程是异步的，这意味着扩展无需在 `on_cmd`、`on_data`、`on_audio_frame` 或 `on_video_frame` 回调函数返回之前就发送处理后的结果。结果只有在真正准备就绪时才可以使用适当的发送函数进行传输。


# 概述 (/cn/docs/ten_framework/software_packages/overview)

一个基本的 TEN framework 软件包的目录结构如下所示。

```text
.
├── bin/
├── src/
├── manifest.json
├── manifest-lock.json
├── property.json
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    ├── protocol/
    │   └── <protocol_a>/
    └── system/
        ├── <system_package_1>/
        └── <system_package_2>/
```

根据编程语言的不同，可能还会存在其他必需的文件。例如，在 C++ 项目中，由于构建系统是 `ten_gn`，因此会存在一个 `BUILD.gn` 文件。在 Go 项目中，会存在诸如 `go.mod` 和 `go.sum` 之类的文件。在 Python 项目中，你可能会找到 `requirements.txt` 文件。

## TEN framework 软件包类型

| 类型                    | 描述              |
| --------------------- | --------------- |
| 应用 (App)              | 包含一个 TEN 应用。    |
| 扩展组 (Extension group) | 包含一个 TEN 扩展组。   |
| 扩展 (Extension)        | 包含一个 TEN 扩展。    |
| 协议 (Protocol)         | 包含一个 TEN 协议。    |
| 系统 (System)           | 包含一个 TEN 系统软件包。 |

## TEN framework 软件包种类

有两种类型的软件包：

* 发布包 (Release package)
* 开发包 (Development package)

这两种软件包之间的关系如下：

* 发布包可以通过构建过程从开发包生成。
* 如果我们将与源代码相关的内容添加到发布包中，它就会变成一个开发包。
* 如果我们从开发包中删除与源代码相关的内容，它就会变成一个发布包。

简而言之，开发包和发布包的目录结构非常相似，区别在于开发包包含与源代码相关的内容，而发布包则不包含。

### 开发包

开发包的主要目的是开发 TEN 软件包。例如，如果您想创建一个新的 TEN 扩展，您可以修改现有的开发包来实现此目的。这是因为开发包包含所有与源代码相关的内容，可以从中创建新的 TEN 软件包。

### 发布包

发布包的主要目的是创建和运行图。由于不需要构建任务，因此可以使用发布包来执行这些操作。但是，开发包也可以用于这些任务。本质上，开发包的应用范围比发布包更广泛。


# Python 异步扩展 (/cn/docs/ten_framework/software_packages/python_async_extension)

**Python 异步扩展**是一个使用 Python 的 `asyncio` 框架的模块化组件。如果您想将使用 `asyncio` 的现有 Python 代码包装到 TEN 扩展中，那么使用 Python 异步扩展是最简单、最方便的选择。

## 示例：默认 Python 异步扩展

以下是 Python 异步扩展的结构方式：

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    async def on_configure(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_json = cmd.to_json()
        ten_env.log_debug(f"DefaultAsyncExtension on_cmd: {cmd_json}")

        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

        # 向其他扩展发送新命令并等待结果。结果将返回给原始发送者。
        new_cmd = Cmd.create("hello")
        cmd_result = await ten_env.send_cmd(new_cmd)
        ten_env.return_result(cmd_result, cmd)
```

每种方法都使用 `await asyncio.sleep()` 模拟延迟。

## 常见问题

### 用于事件处理的异步循环

* 创建一个队列：使用 `asyncio.Queue`。
* 创建一个用于事件处理的异步任务。

这是示例代码：

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    queue = asyncio.Queue()
    loop:asyncio.AbstractEventLoop = None

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        self.loop = asyncio.get_event_loop()

        self.loop.create_task(self._consume())

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        self.queue.put(None)

    async def _consume(self) -> None:
        while True
            try:
                value = await self.queue.get()
                if value is None:
                    self.ten_env.log_info("async loop exit")
                    break

                # 用于处理从队列检索的值的代码。

            except Exception as e:
                self.ten_env.log_error(f"Failed to handle {e}")
```

## 结论

TEN 的 Python 异步扩展提供了一种以异步方式处理长时间运行任务的强大方法。通过集成 Python 的 `asyncio` 框架，扩展能够确保诸如网络调用或文件处理之类的操作是高效且非阻塞的。这使得 TEN 成为构建具有异步功能的可扩展模块化应用程序的绝佳选择。


# 标准接口 (/cn/docs/ten_framework/standard_extension/standard_interface)

在 TEN framework 中，我们鼓励开发者创建自己的扩展（Extension）并将其发布到 TEN 扩展市场，供其他开发者使用。为了让开发者能够轻松地选择和替换扩展，我们需要定义标准化的接口规范，这样开发者就可以根据这些标准接口快速找到符合需求的扩展。

简而言之，标准接口是一种语法糖，它让 TEN 工具（如 TEN Manager）能够更方便地协助开发者进行扩展的选择和替换等操作。

TEN 的接口定义类似于 TypeScript 的 `.d.ts` 文件，让 TypeScript 编译器能够进行类型检查，但在实际运行时，这些接口定义并不会被使用。因此，标准接口设计带来的功能和代码增强，主要体现在 TEN Manager 这类开发工具上，而不是运行时的 TEN Runtime 上。例如，TEN Manager 需要提供相关的 RESTful API 给 tman designer Frontend，让用户可以根据标准接口定义，通过 UI 界面进行扩展的选择和替换。

## 什么是标准接口

在 TEN framework 中，每个扩展的接口都包含以下几个核心组成部分：

1. **Properties（属性）** - 扩展的配置参数
2. **Command（命令）**
   * **Input command** - 输入命令
   * **Output command** - 输出命令
3. **Data（数据）**
   * **Input data** - 输入数据
   * **Output data** - 输出数据
4. **Audio frame（音频帧）**
   * **Input audio frame** - 输入音频帧
   * **Output audio frame** - 输出音频帧
5. **Video frame（视频帧）**
   * **Input video frame** - 输入视频帧
   * **Output video frame** - 输出视频帧

### 标准接口的意义和用途

标准接口就是将这些组成部分进行规范化定义，形成统一的接口标准。开发者可以参考这些接口标准来实现自己的扩展，同时扩展也可以声明自己支持特定的标准接口，这样开发工具就能够根据扩展声明的标准接口提供更好的开发支持，使用户能够根据标准接口来选择或替换所需的扩展。

在标准接口中，Command/Data/Audio frame/Video frame 用于规范扩展之间的交互，实现扩展的互换性和兼容性。而 Properties 则用于规范扩展的配置参数，实现与 tman designer 的配置兼容性。

需要注意的是，规范扩展之间的交互并不是接口本身的功能，这是 Command/Data/Audio frame/Video frame 的功能。接口只是一个用于聚合的语法糖，帮助开发者快速指定扩展的接口定义。

### 接口的检查与报错

在扩展的 Command/Data/Audio frame/Video frame 定义中，如果存在两个同名的消息，且它们的 schema 定义相同，这不算错误，但会被视为重复定义并被忽略。这个规则在标准接口中同样适用，目的是让不同的标准接口可以各自定义共性功能，形成自包含的标准接口定义。当一个扩展实现这些具有相同定义的标准接口时，不会因此产生错误。这个特性在标准接口中非常有用，因为标准接口的定义可以被引用，所以当一个扩展实现多个标准接口时，如果这些标准接口中（或其引用的其他标准接口中）有相同的 message 和相同的 schema，也不会因此产生错误。

由于 TEN framework 采用运行与开发分离的架构（类似于 JavaScript 与 TypeScript 的 .d.ts 文件的关系），标准接口的检查与报错主要发生在开发工具阶段，而不是运行时的 TEN Runtime 上。因此，TEN Runtime 在启动图（start graph）时，不会因为接口错误而报错。

在 tman designer 这个开发工具中，如果扩展的接口定义不完整，会显示某些功能暂时无法在该扩展上使用的提示。例如，在使用某个扩展替代另一个扩展的功能时，如果接口定义不完整，会提示该功能暂时无法在这个扩展上使用，需要等待接口定义完整后才能使用。

另外，由于接口定义可以是一个被引用的独立文件，因此在 TEN Manager 的 publish/install 操作中，接口定义不完整并不算错误。

### 接口的兼容规则

当要将实现接口 A 的扩展替换为实现接口 B 的扩展时，需要接口 A 兼容于接口 B 才能进行替换。接口之间的兼容规则有两种模式：

**严格模式：**

* 接口 A 的所有消息都必须出现在接口 B 中，且接口 A 中消息的 schema 必须兼容于接口 B 中对应消息的 schema。

**宽松模式：**

* 只有存在连线（即有 connection）的接口 A 中的消息必须出现在接口 B 中，且接口 A 中这些消息的 schema 必须兼容于接口 B 中同名消息的 schema。

严格模式和宽松模式供用户选择，适用于不同的场景。例如，如果确定扩展实现的接口 A 中的某个消息 X 一定不会被使用，那么因为消息 X 的不兼容或不存在而导致无法替换为想要的另一个扩展可能过于严格。

## 标准接口的实现方式

TEN 扩展支持的 API 定义在其 `manifest.json` 文件中。标准接口的实现就是将定义好的 API 规范写入独立的 JSON 文件，然后在 `manifest.json` 中引用这个文件。

```json
{
  "type": "extension",
  "name": "foo_extension",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.1"
    }
  ],
  "api": {
    "interface": [
      {
        "import_uri": "https://xxx/asr_interface.json"  // <= 引用标准接口定义文件
      },
      {
        "import_uri": "https://xxx/tts_interface.json"  // <= 引用标准接口定义文件
      }
    ],
    "property": {
      "bar": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "query_vector",
        "property": {
          "collection_name": {
            "type": "string"
          },
          "top_k": {
            "type": "int64"
          },
          "embedding": {
            "type": "array",
            "items": {
              "type": "float64"
            }
          }
        },
        "required": [
          "collection_name",
          "top_k",
          "embedding"
        ],
        "result": {
          "property": {
            "response": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "content": {
                    "type": "string"
                  },
                  "score": {
                    "type": "float64"
                  }
                }
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "publish",
        "property": {
          "message": {
            "type": "buf"
          }
        }
      },
      {
        "name": "set_presence_state"
      }
    ],
    "data_in": [
      {
        "name": "data"
      }
    ]
  }
}
```

### 标准接口定义文件示例

当 TEN 系统（包括 TEN Runtime、TEN Manager 等）解析 `manifest.json` 时，会根据 `import_uri` 下载对应的 JSON 文件，并将其内容合并到 `manifest.json` 中。被引用的标准接口文件内容示例如下，可以看到内容与 manifest.json 中的 `api` 字段相同，只是被独立成一个文件以便引用。并且这个独立的标准接口文件也可以再引用其他的标准接口文件。

```json
{
  "interface": [
    {
      "import_uri": "https://xxx/stt_interface.json"// <= 引用其他的标准接口定义文件
    }
  ],
  "property": {
    "bar": {
      "type": "string"
    }
  },
  "cmd_in": [
    {
      "name": "query_vector",
      "property": {
        "collection_name": {
          "type": "string"
        },
        "top_k": {
          "type": "int64"
        },
        "embedding": {
          "type": "array",
          "items": {
            "type": "float64"
          }
        }
      },
      "required": [
        "collection_name",
        "top_k",
        "embedding"
      ],
      "result": {
        "property": {
          "response": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string"
                },
                "score": {
                  "type": "float64"
                }
              }
            }
          }
        }
      }
    }
  ],
  "cmd_out": [
    {
      "name": "publish",
      "property": {
        "message": {
          "type": "buf"
        }
      }
    },
    {
      "name": "set_presence_state"
    }
  ],
  "data_in": [
    {
      "name": "data"
    }
  ]
}
```

### 接口合并规则

系统会将 `import_uri` 指向的 JSON 文件内容合并到 `manifest.json` 的 `api` 字段中，具体包括：

* `property`
* `cmd_in`、`cmd_out`
* `data_in`、`data_out`
* `audio_frame_in`、`audio_frame_out`
* `video_frame_in`、`video_frame_out`

**重要规则：**

1. **冲突检测**：如果合并过程中发现同名的同类字段（比如同名的 `cmd_in` 或同名的 `property`），如果 schema 相同则不算错误，否则系统会报错。
2. **递归合并**：合并过程是递归进行的，如果检测到循环引用，系统会报错。
3. **多接口支持**：一个扩展可以支持多个标准接口，在 `manifest.json` 中定义多个 `interface` 字段即可。
4. **重复引用检测**：如果不同的 `interface` 字段引用了相同的 `import_uri`，系统会报错。

## 标准接口的存储方式

标准接口文件支持以下几种存储方式：

<div className="fd-steps">
  <div className="fd-step">
    ### TEN 系统包（TEN System Package）

    **概述**：这种方式是将标准接口定义文件打包成 TEN 系统包，然后在 `manifest.json` 中引用。同时也可以在 manifest.json 的 dependencies 字段中依赖这个系统包，这样在安装扩展时就会一并安装所依赖的标准接口系统包，从而提供最佳的开发体验。通过这种方法，可以利用 TEN 包的版本机制来实现标准接口的版本管理，并且可以利用 TEN 包的依赖机制来实现标准接口的自动安装。

    使用 TEN 系统包，除了标准接口的 JSON 定义文件外，还可以包含相关的资源，例如模型文件、公共工具函数库、配套工具等。因此，当所有这些资源出现不兼容的变更时，都可以通过 TEN 系统包的版本号变更来更好地体现这种变化。例如，系统包内 JSON 文件的存放位置或者文件名称的变化，也算是该系统包的破坏性变更，因此应该升级该系统包的主版本号。

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "0.1.0"
        }
      ],
      "api": {
        "interface": [
          {
            "import_uri": "../../../ten_packages/system/foo_interface/interface.json"
          }
        ]
      }
    }
    ```

    **使用 TEN 系统包的优势：**

    1. **易于分发**：系统包可以上传到 TEN 扩展市场，用户可以直接下载使用
    2. **资源丰富**：包中不仅包含接口定义文件，还可以包含相关资源，如：
       * 模型文件
       * 公共工具函数库
       * 配套工具等
    3. **依赖管理**：系统包可以作为扩展的依赖项，当扩展被下载时，相关的系统包也会自动下载
    4. **版本管理**：不仅限于标准接口协议本身，该系统包内的任何资源都是该系统包版本管理的一部分

    需要注意的是，TEN 系统包的版本号并不仅仅代表标准接口的版本号，而是代表整个系统包的版本号。因此，如果系统包内的资源有破坏性变更，就应该升级系统包的主版本号。例如，系统包内 JSON 文件的存放位置或者文件名称的变化，即使没有改变 JSON 文件的内容，也算是该系统包的破坏性变更，因为对使用者来说，使用方式需要改变，所以应该升级该系统包的主版本号。

    因此，不能仅仅通过系统包的版本号来判断标准接口的兼容性。标准接口的兼容性还是需要通过接口的兼容规则来判断。

    #### 支持多版本共存

    > **注意**：这一节所描述的功能，基本上不会实现，只是描述如果之后要做的话，可能的实现方法以及需要考虑的点。

    **概述**：在某些场景下，一个 TEN 应用可能需要同时使用同一标准接口的不同版本。为了支持这种需求，TEN 允许在同一个应用内安装同一个系统包的多个版本。

    **系统包配置：**

    首先，在系统包的 `manifest.json` 中声明支持多版本共存：

    ```json
    {
      "type": "system",
      "name": "foo_interface",
      "version": "0.1.0",
      "install_policy": {
        "coinstallable": "true"
      }
    }
    ```

    **扩展依赖配置：**

    在依赖该系统包的扩展的 `manifest.json` 中，也需要声明使用多版本共存：

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "=0.1.0", // 注意：这里只能是 equal 的语法，不能用 >= 之类的语法
          "install_policy": {
            "coinstallable": "true"
          }
        }
      ],
    }
    ```

    并且在整个依赖树的构建过程中，如果一个系统包的 coinstallable 为 true，则需要所有依赖该系统包的地方都声明是 coinstallable，否则系统会报错。

    **文件系统路径：**

    启用多版本共存后，系统包在文件系统中的路径会变为：

    ```text
    ten_packages/system/<hash_value>_foo_extension_0_1_0/
    ```

    > **注意**：未来可能会引入映射机制，让用户在使用时不需要明确指定 hash\_value，而是通过某种映射机制自动找到对应的系统包。但这种机制如果要好用，还需要修改 TEN 所支持的语言中的模块加载机制，使得开发者可以不用在该语言中的模块加载机制中指定 hash\_value，而是通过某种映射机制自动找到对应的系统包。这种深入语言内部的改动也是这种单包多版本共存的机制基本上不会被实现的原因之一。
    >
    > **注意**：如果真的需要在一个 TEN 应用内使用多个版本的一个标准接口，可以使用下述的几种方式来实现。如果一个标准接口有配套的公用资源，例如模型文件、公共工具函数库、配套工具等，那么这些资源很多无法做到在一个 TEN 应用内的多版本共存，这也是这种单包多版本共存的机制基本上不会被实现的原因之一。
  </div>

  <div className="fd-step">
    ### 网络地址

    **概述**：将标准接口定义文件部署到网络服务器上，通过 HTTP/HTTPS 地址引用：

    ```json
    {
      "interface": [
        {
          "import_uri": "http://xxx/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    > **注意**：TEN Manager 在处理这种引用方式时，可以将文件下载到自己的缓存中，然后通过某种映射机制，让 TEN Manager 能够找到定义。

    **适用场景：**

    * **轻量级接口**：没有配套的公共资源的接口定义
    * **版本灵活性**：需要在一个 `manifest.json` 中引用同一标准接口的不同版本
  </div>

  <div className="fd-step">
    ### 本地路径

    **概述**：引用本地文件系统中的标准接口定义文件：

    **绝对路径：**

    ```json
    {
      "interface": [
        {
          "import_uri": "file://c:/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **相对路径：**

    ```json
    {
      "interface": [
        {
          "import_uri": "../asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **适用场景：**

    与网络地址方式类似：

    * **轻量级接口**：适合没有配套的公共资源的接口定义
    * **版本灵活性**：需要在一个 `manifest.json` 中引用同一标准接口的不同版本
  </div>
</div>

## 如何从云商店中筛选想要的扩展

标准接口与云商店的关系体现在将标准接口包装成 TEN 系统包。有以下两种方式可以搜索出符合特定条件的扩展：

1. **依赖搜索**：搜索依赖某个系统包的扩展。由于云商店中扩展的元数据包含了依赖信息，所以云商店可以据此筛选出依赖某个系统包的所有扩展。

2. **标签搜索**：通过 TEN package 的 tag 来快速搜索出符合某些条件的扩展，选择安装后，再进行替换操作。如果是 TEN 官方认证过的扩展，甚至可以打上 `ten:` 开头的标签来表明这是一个官方认证过的扩展，确实符合所声明的标准接口定义。


# 图检查 (/cn/docs/ten_framework/ten_manager/check_graph)

`tman` 提供了 `check graph` 命令，用于验证预定义图或 `start_graph` 命令的正确性。要查看使用细节，请使用以下命令：

```shell
$ tman check graph -h
检查预定义图或 start_graph 命令的图内容是否正确。有关更详细的用法，请运行“graph -h”

用法：tman check graph [选项] --app <APP>

选项：
      --app <APP>
          图中定义的应用程序的绝对路径。默认情况下，将从列表中的第一个应用程序读取预定义图。
      --predefined-graph-name <PREDEFINED_GRAPH_NAME>
          指定要检查的预定义图的名称，否则将检查所有预定义图。
      --graph <GRAPH>
          指定要检查的“start_graph”命令的 JSON 字符串。如果未指定，则将检查第一个应用程序中的预定义图。
  -h, --help
          打印帮助信息
```

`check graph` 命令旨在处理可能跨越多个应用程序的图，因此不需要从特定应用程序的根目录运行。相反，它可以从任何目录执行，并使用 `--app` 参数指定图中涉及的应用程序的文件夹。

## 使用示例

* **检查 `property.json` 中的所有预定义图：**

  ```shell
  tman check graph --app /home/TEN-Agent/agents
  ```

* **检查特定的预定义图：**

  ```shell
  tman check graph --predefined-graph-name va.openai.azure --app /home/TEN-Agent/agents
  ```

* **检查 `start_graph` 命令：**

  ```shell
  tman check graph --graph '{
    "type": "start_graph",
    "seq_id": "55",
    "nodes": [
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_2__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      },
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_1__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      }
    ]
  }' --app /home/TEN-Agent/agents
  ```

## 前提条件

* **预定义图定义要求：** 如果指定了预定义图的名称，则将从第一个 `--app` 参数指定的应用程序的 `property.json` 文件中提取其定义。
* **包安装要求：** 在运行 `check graph` 命令之前，必须使用 `tman install` 命令安装应用程序依赖的所有扩展。这是必需的，因为验证过程需要有关图中每个扩展的信息，例如其 `manifest.json` 文件中定义的 API。
* **唯一应用程序 URI 要求：** 在多应用程序图中，每个应用程序的 `property.json` 必须定义唯一的 `ten::uri`。此外，`uri` 值不能设置为 `"localhost"`。

## 验证规则

<div className="fd-steps">
  <div className="fd-step">
    ### 节点 (Nodes) 的存在性

    任何图定义中都必须存在 `nodes` 数组。如果缺少该数组，将会抛出错误。

    * **示例（未定义节点）：**

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": []
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        No extension node is defined in graph.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 节点的唯一性

    `nodes` 数组中的每个节点表示一个应用程序组中由指定插件创建的特定扩展实例。因此，每个扩展实例都应该由一个单独的节点唯一表示。必须通过 `app`、`extension_group` 和 `name` 的组合来唯一标识一个节点。不允许出现同一扩展实例的多个条目。

    * **示例（重复节点）：**

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": [
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_2__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          },
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_1__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        Duplicated extension was found in nodes[1], addon: basic_hello_world_1__test_extension, name: test_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在通信链路中使用的扩展应在 nodes 中定义

    `connections` 字段中引用的所有扩展实例（无论是作为源还是目标）都必须在 `nodes` 字段中显式定义。任何未在 `nodes` 数组中定义的实例都会导致验证错误。

    * **示例（未定义源扩展）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0] is not defined in nodes, extension_group: producer, extension: some_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

    * **示例（未定义目标扩展）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                },
                {
                  "type": "extension",
                  "name": "some_extension_1",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        }
                      ]
                    },
                    {
                      "name": "world",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        },
                        {
                          "extension": "consumer"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0].cmd[1] is not defined in nodes extension_group: some_group, extension: consumer.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 `nodes` 中声明的插件必须安装在应用中

    * **示例（`property.json` 中的 `ten::uri` 与 `nodes` 中的 `app` 字段不相等）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ],
          "uri": "http://localhost:8001"
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

      问题在于，应用程序中的所有包都将存储在以应用程序的 `uri` 为键的映射中，并且图中的每个节点都将通过 `app` 字段（默认为 `localhost`）检索。节点中的 `app`（即 `localhost`）与应用程序的 `uri`（即 `http://localhost:8001`）不匹配。

    * **示例（由于未执行 `tman install` 而导致 `ten_packages` 目录不存在）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ]
        }
      }
      ```

      并且 `ten_packages` 目录**不存在**。

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 通信链路中，从一个扩展发送的消息应在同一部分中定义

    * **示例：**

      假设所有包都已安装。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          },
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello_2",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        extension 'some_extension' is defined in connection[0] and connection[1], merge them into one section.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 通信链路中，从一个扩展发出的消息在每个类型中应具有唯一的名称

    * **示例：**

      假设所有包都已安装。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              },
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "some_extension"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        - connection[0]:
         - Merge the following cmd into one section:
            'hello' is defined in flow[0] and flow[1].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 通信链路中声明的消息应兼容

    根据扩展的 `manifest.json` 文件中的模式定义，将检查在 `connections` 中的每个消息流中声明的消息的模式是否兼容。消息兼容的规则如下。

    * 如果在源扩展和目标扩展中都找不到消息模式，则消息是兼容的。

    * 如果仅在其中一个扩展中找到消息模式，则该消息不兼容。

    * 只有满足以下条件，消息才是兼容的：

      * 属性类型兼容。
        * 如果属性是一个对象，则源模式和目标模式中的字段都必须具有兼容的类型。
        * 如果在目标模式中定义了 `required` 关键字，则源模式中必须存在一个 `required` 关键字，该关键字是目标中 `required` 的超集。

    * **示例：**

      假设所有包都已安装。

      `addon_a` 中的 `manifest.json` 的内容如下。

      ```json
      {
        "type": "extension",
        "name": "addon_a",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_out": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "string"
                }
              }
            }
          ]
        }
      }
      ```

      并且，`addon_b` 中的 `manifest.json` 的内容如下。

      ```json
      {
        "type": "extension",
        "name": "addon_b",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_in": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "int8"
                }
              }
            }
          ]
        }
      }
      ```

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        - connections[0]:
          - cmd[0]:  Schema incompatible to [extension_group: some_group, extension: another_ext], { .foo: type is incompatible, source is [string], but target is [int8] }

      All is done.
      ❌ Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 节点中的 `app` 必须明确

    每个节点中的 `app` 字段必须满足以下规则。

    * `app` 字段必须等于相应 TEN 应用程序的 `ten::uri`。

    * 要么所有节点都应该声明 `app`，要么都不应该声明。

    * `app` 字段不能为 `localhost`。

    * `app` 字段不能为空字符串。

    * **示例（某些节点指定了 `app` 字段）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        Either all nodes should have 'app' declared, or none should, but not a mix of both.
      ```

    * **示例（所有节点均未指定 `app`，但某些源扩展在通信链路中指定了 `app`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
         {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].the 'app' should not be declared, as not any node has declared it
      ```

    * **示例（所有节点均未指定 `app`，但某些目标扩展在通信链路中指定了 `app`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext",
                    "app": "http://localhost:8000"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].cmd[0].dest[0]: the 'app' should not be declared, as not any node has declared it
      ```

    * **示例 (节点中的 `app` 字段与应用程序的 `ten::uri` 不相等)：**

      与 [规则 4](#id-4.-the-addons-declared-in-the-nodes-must-be-installed-in-the-app) 相同。

    * **示例（单应用图中 `app` 字段为 `localhost`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, and the graph seems to be a single-app graph, just remove the 'app' field
      ```

    * **示例（多应用图中 `app` 字段为 `localhost`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, change the content of 'app' field to be consistent with 'ten::uri'
      ```
  </div>
</div>


# 概述 (/cn/docs/ten_framework/ten_manager/overview)

在 TEN 生态系统中，几乎所有事物都是组件。例如，一个 TEN 应用是一种组件，一个 TEN 扩展是另一种组件，而一个 TEN 协议又是另一种组件。为了帮助 TEN 开发者管理与这些组件相关的所有操作，我们提供了一个至关重要的工具，称为 TEN 管理器。

<Callout type="info">
  `tman`

   是 TEN 管理器 (TEN Manager) 的缩写。
</Callout>

TEN 管理器包含一个 TEN 软件包管理器，旨在简化 TEN 软件包的管理。此软件包管理器处理诸如确定哪些依赖包是必需的、获取这些依赖包以及安装它们等任务。简而言之，TEN 管理器简化了从头开始创建新的 TEN 软件包的过程，并使从 TEN 云商店安装 TEN 软件包更加容易。


# 调试 (/cn/docs/ten_framework/testing_and_debugging/debugging)

## 适用于 TEN framework 的开发人员

如果您使用 VS Code 开发 TEN framework，则 TEN framework 源代码树中有一个 `launch.json` 文件，其中提供了默认的调试目标。

## 适用于 TEN framework 的用户

如果您使用 VS Code 开发基于 TEN framework 的自己的应用程序，您可以将配置添加到您的 `.vscode/launch.json` 文件，以便您可以使用断点和变量检查来调试您的应用程序，无论您使用的是哪种编程语言（C++/Golang/Python）。

### 在 C++ 应用程序中调试

如果应用程序是用 C++ 编写的，这意味着扩展可以使用 C++ 或 Python 编写。

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

* **使用 lldb 调试 C++ 代码**

  ```json
  {
    "name": "app (C++) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    }
  }
  ```

* **使用 gdb 调试 C++ 代码**

  ```json
  {
    "name": "app (C++) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### 使用 debugpy 调试 Python 代码

您可以使用 `debugpy` 调试 Python 代码。 但是，由于程序不是直接通过 Python 解释器启动的，因此 Python 代码由嵌入的 Python 解释器执行。 因此，您只能将调试器附加到正在运行的进程。

首先，您需要在 Python 环境中安装 `debugpy`：

```shell
pip install debugpy
```

然后，您需要使用设置的环境变量 `TEN_ENABLE_PYTHON_DEBUG` 和 `TEN_PYTHON_DEBUG_PORT` 启动应用程序。 例如：

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

如果环境变量 `TEN_ENABLE_PYTHON_DEBUG` 设置为 `true`，则应用程序将阻塞，直到附加调试器。 如果设置了环境变量 `TEN_PYTHON_DEBUG_PORT`，则调试器服务器将侦听指定端口上的传入连接。

然后，您可以将调试器附加到正在运行的进程。 以下是一个配置示例：

**使用 debugpy 调试 Python 代码**

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### 同时调试 C++ 和 Python 代码

如果您想在 VSCode 中一键启动同时调试 C++ 和 Python 代码，您可以执行以下操作：

1. 定义一个任务，在附加调试器之前延迟几秒钟：

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. 将以下配置添加到 `launch.json` 文件：

   ```json
   "configurations": [
        {
            "name": "app (C++) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch", // "launch" or "attach"
            "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
            "cwd": "${workspaceFolder}/",
            "env": {
                "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
                "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (C++) (debugpy, attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "justMyCode": false,
            "preLaunchTask": "delay 3 seconds"
        }
   ],
   "compounds": [
        {
            "name": "app (C++) (lldb, launch) and app (C++) (debugpy, attach)",
            "configurations": [
                "app (C++) (lldb, launch with debugpy)",
                "app (C++) (debugpy, attach with delay)"
            ]
        }
   ]
   ```

然后，您可以使用复合配置 `app (C++) (lldb, launch) and app (C++) (debugpy, attach)` 启动调试 C++ 和 Python 代码。

### 在 Go 应用程序中调试

如果应用程序是用 Go 编写的，这意味着扩展可以使用 Go 或 C++ 或 Python 编写。

#### 使用 delve 调试 Go 代码

您可以使用以下配置在您的 Go 扩展中进行调试：

```json
{
  "name": "app (golang) (go, launch)",
  "type": "go",
  "request": "launch",
  "mode": "exec",
  "cwd": "${workspaceFolder}/",
  "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
  "env": {
    "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
    "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
    "TEN_APP_BASE_DIR": "${workspaceFolder}"
  }
}
```

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

* **使用 lldb 调试 C++ 代码**

  ```json
  {
    "name": "app (Go) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    },
    "initCommands": ["process handle SIGURG --stop false --pass true"]
  }
  ```

* **使用 gdb 调试 C++ 代码**

  ```json
  {
    "name": "app (Go) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### 使用 debugpy 调试 Python 代码

请参阅上面的部分，了解如何使用 debugpy 调试 Python 代码。 使用设置的环境变量 `TEN_ENABLE_PYTHON_DEBUG` 和 `TEN_PYTHON_DEBUG_PORT` 启动应用程序。

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

然后，您可以将调试器附加到正在运行的进程。 以下是一个配置示例：

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### 同时调试 C++、Go 和 Python 代码

如果您想在 VSCode 中一键启动同时调试 C++、Go 和 Python 代码，您可以执行以下操作：

1. 定义一个任务，在附加调试器之前延迟几秒钟：

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. 将以下配置添加到 `launch.json` 文件：

   ```json
   "configurations": [
        {
            "name": "app (golang) (debugpy, remote attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "delay 3 seconds",
            "justMyCode": false
        },
        {
            "name": "app (golang) (go, launch with debugpy)",
            "type": "go",
            "request": "launch",
            "mode": "exec",
            "cwd": "${workspaceFolder}/",
            "program": "${workspaceFolder}/bin/worker",
            "env": {
                "TEN_APP_BASE_DIR": "${workspaceFolder}",
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (golang) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/bin/worker",
            "cwd": "${workspaceFolder}/",
            "env": {
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            },
            "initCommands": [
                "process handle SIGURG --stop false --pass true"
            ]
        },
   ],
   "compounds": [
        {
            "name": "Mixed Go/Python",
            "configurations": [
                "app (golang) (go, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ],
        },
        {
            "name": "Mixed Go/Python/C++ (lldb)",
            "configurations": [
                "app (golang) (lldb, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ]
        }
   ]
   ```

如果您想同时调试 C++ 和 Python 代码，您可以使用复合配置 `Mixed Go/Python/C++ (lldb)`。 使用此配置，您可以同时对 C++、Go 和 Python 代码执行断点调试。 但是，变量检查仅支持 C++ 和 Python 代码，而不支持 Go 代码。

如果您想同时调试 Go 和 Python 代码，您可以使用复合配置 `Mixed Go/Python`。 使用此配置，您可以同时对 Go 和 Python 代码执行断点调试，并检查 Go 和 Python 代码的变量。

### 在 Python 应用程序中调试

如果应用程序是用 Python 编写的，这意味着扩展可以使用 Python 或 C++ 编写。

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

```json
{
  "name": "app (Python) (cpp, launch)",
  "type": "cppdbg", //
  "request": "launch", // "launch" or "attach"
  "program": "/usr/bin/python3", // Python 解释器路径
  "args": ["main.py"],
  "cwd": "${workspaceFolder}/",
  "environment": [
    {
      "name": "PYTHONPATH",
      "value": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface"
    },
    {
      "name": "TEN_APP_BASE_DIR",
      "value": "${workspaceFolder}/"
    }
  ],
  "MIMode": "gdb" // "gdb" 或 "lldb"
}
```

#### 使用 debugpy 调试 Python 代码

请参阅上面的部分，了解如何使用 debugpy 调试 Python 代码。 您可以使用以下配置来调试 Python 代码：

```json
{
  "name": "app (Python) (debugpy, launch)",
  "type": "debugpy",
  "python": "/usr/bin/python3",
  "request": "launch",
  "program": "${workspaceFolder}/main.py",
  "console": "integratedTerminal",
  "cwd": "${workspaceFolder}/",
  "env": {
    "PYTHONPATH": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface",
    "TEN_APP_BASE_DIR": "${workspaceFolder}/",
    "TEN_ENABLE_PYTHON_DEBUG": "true"
  }
}
```

#### 同时调试 C++ 和 Python 代码

如果您想同时启动 C++ 和 Python 代码的调试，您可以使用配置 `app (Python) (debugpy, launch)` 启动应用程序，并使用以下配置将调试器附加到正在运行的进程：

```json
{
  "name": "app (Python) (cpp, attach)",
  "type": "cppdbg",
  "request": "attach", // "launch" 或 "attach"
  "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
  "cwd": "${workspaceFolder}/",
  "MIMode": "gdb",
  "environment": [
    {
      "name": "LD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    },
    {
      "name": "DYLD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    }
  ]
}
```


# 面向 TEN framework 开发者 (/cn/docs/ten_framework/testing_and_debugging/for_developers_of_ten_framework)

TEN framework 包括三种类型的测试套件：

* 单元测试
* 冒烟测试
* 集成测试

TEN framework 使用 `gtest` 作为单元测试和冒烟测试的测试框架，使用 `pytest` 作为集成测试的测试框架。

## 单元测试

单元测试的源目录是 `tests/ten_runtime/unit`。如果您需要添加新的单元测试用例，请将它们放在 `tests/ten_runtime/unit directory` 中。

对于 C++ 项目，在成功编译后，您可以导航到输出目录（例如，`out/linux/x64/tests/standalone/`，具体取决于目标操作系统和 CPU）并执行以下命令：

```shell
./ten_runtime_unit_test
```

## 冒烟测试

冒烟测试的源目录是 `tests/ten_runtime/smoke`。如果您需要添加新的冒烟测试用例，请将它们放在 `tests/ten_runtime/smoke` 目录中。

导航到输出目录（例如，`out/linux/x64/tests/standalone`，具体取决于目标操作系统和 CPU）并执行以下命令：

```shell
./ten_runtime_smoke_test
```

## 循环测试

您可以使用以下命令执行多轮测试。

仅运行单元测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;
```

仅运行冒烟测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

同时运行单元测试和冒烟测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

## 集成测试

集成测试是 TEN framework 的黑盒测试。集成测试的源目录是 `tests/ten_runtime/integration`。这些测试用于验证实际的执行场景。

集成测试的文件结构如下：

```text
tests/ten_runtime/integration/
  ├── test_1/
  │    ├── test_case.py
  │    └── ...
  ├── test_2/
  │    ├── test_case.py
  │    └── ...
  └── ...
```

要执行集成测试，请使用以下命令：

```shell
cd path/to/out/
pytest tests/ten_runtime/integration/
```

## 测试技巧

### Linux

在 Linux 上，您可以使用以下技术来减慢 TEN 的执行速度，这有助于触发与时序相关的错误。

```shell
sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...
```

* `taskset 0x1`: 仅使用 1 个 CPU 核心来运行程序。
* `taskset 0x3`: 仅使用 2 个 CPU 核心来运行程序。


# 面向 TEN framework 用户 (/cn/docs/ten_framework/testing_and_debugging/for_users_of_ten_framework)

## 扩展的独立测试

TEN framework 提供了一种独立的扩展测试机制。该机制允许开发者在不依赖其他 TEN 概念（例如其他 TEN 扩展、TEN 图或 TEN 应用）的情况下测试扩展。当开发者需要在不运行整个 TEN 应用的情况下测试扩展的行为时，这种独立的扩展测试机制尤其有用。

TEN 扩展的独立测试框架遵循两个关键原则：

1. **与所用语言的任何原生测试框架兼容。**

   例如，如果使用 C++ 开发 TEN 扩展，则 Google gtest/gmock 测试框架可以与 TEN 扩展独立测试框架一起使用，以实现 C++ TEN 扩展的独立测试。

2. **用户无需修改被测扩展中的任何代码。**

   可以使用此独立测试框架对运行时使用的完全相同的扩展代码进行全面测试。

3. **以不同语言开发的扩展的独立测试流程遵循相同的设计原则和使用方法。**

   这确保一旦您学习了一种语言的独立测试概念和方法，就可以快速掌握其他语言的 TEN 扩展的独立测试概念和方法。

对于用户来说，TEN 扩展独立测试框架引入了两个主要概念：

1. **`extension_tester`**
2. **`ten_env_tester`**

`extension_tester` 的作用类似于一个测试驱动程序，负责设置和执行整个测试过程。另一方面，`ten_env_tester` 的行为类似于典型的 TEN 扩展的 `ten_env` 实例，使用户能够从 `extension_tester` 的回调中调用独立测试框架中的功能，例如向被测扩展发送消息和从被测扩展接收消息。

从 `extension_tester` 和 `ten_env_tester` 的 API 设计中，您可以看到它们与 **TEN 扩展** 和 `ten_env` 非常相似，这是有意为之的。主要目的是让熟悉扩展开发的用户能够快速适应独立测试框架，然后为扩展本身开发独立的测试用例。

然而，出于测试目的，不可避免地会有专门用于测试的 API 和功能。为了防止这些在实际运行时不需要的、专用于测试的功能和 API 污染运行平台 API 集，独立测试框架被设计为不直接使用或扩展 **extension** 和 `ten_env` 的 API 集。相反，它引入了独立于测试的类型和 API 集。这种分离确保了实际运行时使用的类型和 API 集与测试期间使用的类型和 API 集互不干扰，避免了任何负面影响。

## 独立测试框架内部机制

在内部，TEN 扩展独立测试框架会隐式启动一个测试应用，加载要测试的扩展插件（我们称之为扩展 A），并初始化一个包含要测试的扩展 A 和另一个专门用于测试其他扩展的扩展（我们称之为扩展 B）的图。扩展 A 的所有输入和输出消息都被重定向到扩展 B，允许用户在测试过程中自定义输入和输出，并完成实际的测试工作。

```mermaid
sequenceDiagram
  participant tester as Extension Tester
  participant testing as Extension Testing
  participant tested as Extension Tested

  tester->>testing: send messages
  testing->>tested: send messages

  tested-->>testing: send results
  testing-->>tester: send results
```

基本上，您可以将此隐藏在测试框架中的测试扩展视为被测试扩展和测试器之间的代理。它充当 TEN 图中的代理扩展，使用 TEN 环境的语言促进被测扩展和测试器之间的消息交换。

## 基本测试流程

基本测试流程和逻辑如下：

1. 创建一个扩展测试器来管理整个独立测试过程。
2. 向独立测试框架提供包含待测扩展的文件夹信息。
3. 设置测试模式，例如用于测试单个扩展的模式。
4. 开始测试。

## C++

以下是使用 Google gtest 进行 TEN 扩展独立测试的示例：

```c++
class extension_tester_basic : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    auto new_cmd = ten::cmd_t::create("hello_world");
    ten_env.send_cmd(std::move(new_cmd),
                     [](ten::ten_env_tester_t &ten_env,
                        std::unique_ptr<ten::cmd_result_t> result) {
                       if (result->get_status_code() == TEN_STATUS_CODE_OK) {
                         ten_env.stop_test();
                       }
                     });
  }
};

TEST(Test, Basic) {
  // 1. Create an extension tester to manage the entire standalone testing process.
  auto *tester = new extension_tester_basic();

  // 2. Set a testing mode, such as a mode for testing a single extension.
  tester->set_test_mode_single("default_extension_cpp");

  // 3. Start the testing.
  tester->run();

  delete tester;
}
```

## Golang

TODO: 待补充。

## Python

```python
class ExtensionTesterBasic(ExtensionTester):
    def check_hello(self, ten_env: TenEnvTester, result: CmdResult):
        statusCode = result.get_status_code()
        print("receive hello_world, status:" + str(statusCode))

        if statusCode == StatusCode.OK:
            ten_env.stop_test()

    def on_start(self, ten_env: TenEnvTester) -> None:
        new_cmd = Cmd.create("hello_world")

        print("send hello_world")
        ten_env.send_cmd(
            new_cmd,
            lambda ten_env, result: self.check_hello(ten_env, result),
        )

        print("tester on_start_done")
        ten_env.on_start_done()


def test_basic():
    # 1. Create an extension tester to manage the entire standalone testing process.
    tester = ExtensionTesterBasic()

    # 2. Set a testing mode, such as a mode for testing a single extension.
    tester.set_test_mode_single("default_extension_python")

    # 3. Start the testing.
    tester.run()
```


# 如何使用日志进行调试 (/cn/docs/ten_framework/testing_and_debugging/how_to_debug_with_logs)

我们将介绍如何查看日志，现在它们已移植到 stdout，因此查看它们非常容易。

例如，如果 TEN Agent 在 localhost:3000 上运行，您可能会在日志中看到类似于以下的输出：

```bash
...
[GIN] 2024/09/29 - 09:29:42 | 200 |     634.458µs |    192.168.65.1 | POST     "/ping"
[agora_5bnxv1] 09-29 09:29:42.895  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ How about you?], text_recv_ts: 1727602181374572, read_bytes: 37200, sent_frames: 117, first_frame_latency: 306ms, finish_latency: 428ms, synthesis_first_byte_latency: 300ms, synthesis_finish_latency: 332ms
[agora_5bnxv1] 09-29 09:29:43.350  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Anything exciting happening?], text_recv_ts: 1727602181383864, read_bytes: 63600, sent_frames: 199, first_frame_latency: 300ms, finish_latency: 452ms, synthesis_first_byte_latency: 295ms, synthesis_finish_latency: 332ms
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_5bnxv1 requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
2024/09/29 09:29:45 INFO handlerPing end worker="&{ChannelName:agora_5bnxv1 HttpServerPort:10024 LogFile:/tmp/astra/app-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.json Pid:7455 QuitTimeoutSeconds:60 CreateTs:1727602125 UpdateTs:1727602185}" requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |    1.038208ms |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
2024/09/29 09:29:45 ERROR handlerPing channel not existed channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |     442.375µs |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:48 INFO handlerPing start channelName=agora_5bnxv1 requestId=6fb07d85-
...
```

一旦您与 TEN Agent 交互，您应该会看到如下条目：

```bash title="Bash"
[agora_5bnxv1] 09-29 09:35:25.049  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ and the World Cup in Russia.], text_recv_ts: 1727602522313774, read_bytes: 64000, sent_frames: 200, first_frame_latency: 418ms, finish_latency: 568ms, synthesis_first_byte_latency: 413ms, synthesis_finish_latency: 458ms
[agora_5bnxv1] 09-29 09:35:25.570  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ In tech,], text_recv_ts: 1727602522347918, read_bytes: 29200, sent_frames: 92, first_frame_latency: 427ms, finish_latency: 515ms, synthesis_first_byte_latency: 419ms, synthesis_finish_latency: 460ms
[agora_5bnxv1] 09-29 09:35:26.502  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ we saw some big advancements like the rise of AI and electric cars becoming more mainstream.], text_recv_ts: 1727602522461866, read_bytes: 178800, sent_frames: 559, first_frame_latency: 510ms, finish_latency: 928ms, synthesis_first_byte_latency: 503ms, synthesis_finish_latency: 800ms
[agora_5bnxv1] 09-29 09:35:27.029  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Personally,], text_recv_ts: 1727602522468652, read_bytes: 34400, sent_frames: 108, first_frame_latency: 442ms, finish_latency: 524ms, synthesis_first_byte_latency: 436ms, synthesis_finish_latency: 463ms
```

当您看到这样的日志时，这意味着系统工作正常并且正在记录您说的每句话。


# Doc Window (/cn/docs/ten_framework/tman_designer/doc_window)

TEN 设计师内建文档窗口，可以查看 TEN Framework 的观念跟术语. 例如在 App 下拉选单中选择 `About App` 即可打开文档窗口并显示 App 的说明.

![TMAN Designer Doc Window](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_doc_window.png)


# 概述 (/cn/docs/ten_framework/tman_designer/overview)

TEN Framework 具有高度灵活性，提供了丰富的功能，包括 TEN 云商店、软件包管理、多语言支持和多平台支持等。然而，这些功能可能需要较长的学习时间和较高的技术门槛，才能熟练操作 TEN Framework 完成 AI Agent 的开发和定制化。为便于开发者使用，TEN Framework 提供了一个名为 TEN Manager 的命令行工具，使开发者能够简便地修改和定制基于 TEN Framework 构建的 AI Agent。

由于命令行工具不如图形用户界面友好，因此 TEN Manager 除了命令行界面外，还内置了基于 Web UI 的可视化开发界面 — TMAN Designer，提供丰富的功能帮助开发者开发和调试 TEN Apps 及 TEN Extension。这意味着开发者只需获取 TEN Manager 这一个工具，就能同时拥有命令行和图形用户界面的完整 TEN Framework 开发环境。

您可以通过以下链接获取 TMAN Designer：

```text
https://github.com/TEN-framework/ten-framework/releases
```

TMAN Designer 支持以下操作系统及架构，请根据您的平台获取对应版本：

* Windows
  * x86\_64
* Linux
  * x86\_64
* MacOS
  * x86\_64
  * arm64

下载并解压后，使用以下命令启动 TMAN Designer：

```shell
tman designer
```

您可以在任何 TEN App 的根目录下执行此命令，也可以在任意位置执行。如果在 TEN App 的根目录下执行，TMAN Designer 会默认将该 TEN App 载入。您也可以在 TMAN Designer 中手动载入任何 TEN App。因此，在特定 TEN App 根目录下启动 TMAN Designer 并非必要操作，但可以简化初始手动载入 TEN App 的步骤。

启动后将看到如下信息，表示 TMAN Designer 默认在端口 49483 上启动：

```text
🏆  Starting server at http://0.0.0.0:49483
```

您可以使用以下 URL 与 TMAN Designer 交互：

```text
http://127.0.0.1:49483/
```

> 注意：如果您的 TMAN Designer 运行在远程机器上，请使用远程机器的 IP 地址访问 TMAN Designer。

## 概览

使用浏览器打开 TMAN Designer 页面后，会看到如下界面：

![TMAN Designer 概观](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_overview.png)

在 TMAN Designer 右上方可以看到几个功能按钮：

* 最右侧按钮显示当前 TMAN Designer 的版本号，并自动检测是否有新版本发布。如有更新，会显示一个向上的箭头，点击后将打开新版本的下载页面。
* 左侧第一个按钮是 [TEN Agent](https://agent.theten.ai/) 按钮，点击后将打开 TEN Agent 页面。TEN Agent 是使用 TEN Framework 构建的完整 AI Agent 实现，TMAN Designer 的主要目标之一是让开发者能够轻松地通过 TMAN Designer 开发、调试和定制 TEN Agent。
* 左侧第二个按钮是 [TEN Framework](https://github.com/TEN-framework/ten-framework) 的 GitHub 页面，点击后将打开 TEN Framework 的 GitHub 仓库。如需了解更多关于 TEN Framework 的信息、源代码或提交问题，可以访问此页面。
* 最左侧有两个按钮，分别用于切换明暗模式和选择语言。

在 TMAN Designer 的左上方有几个下拉菜单，开发者可以通过这些菜单使用 TMAN Designer 的各项功能。

## 处理 TEN App

### 载入已有的 TEN App

通过 App 菜单的 Load App 按钮，可以载入已有的 TEN App。点击按钮后，会看到如下界面：

![载入已有的 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_load_app.png)

通过此文件浏览对话框，开发者可以指定已有 TEN App 的根目录来载入应用。

### 管理已载入的 TEN App

通过 App 菜单的 Manage Loaded App(s) 按钮，可以管理已载入的 TEN App。点击按钮后，会看到如下 Apps Manager 界面：

![管理已经载入的 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_manage_loaded_apps.png)

开发者可以通过此对话框：

* 卸载已载入的 TEN App
* 重新载入指定的 TEN App
* 重新载入所有已载入的 TEN App
* 安装指定 App 的所有依赖
* 运行指定 App 的功能

### 安装指定 App 的所有依赖

开发者可以在 TEN App 中安装各种 TEN 软件包，一个 TEN App 所依赖的所有 TEN 软件包都会记录在 `manifest.json` 文件中。以下是一个 `manifest.json` 文件的示例：

```json
{
  "type": "app",
  "name": "aaa",
  "version": "0.8.18",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.8.18"
    },
    {
      "type": "extension",
      "name": "ffmpeg_muxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_demuxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_client",
      "version": "^0.1.0"
    },
    {
      "type": "protocol",
      "name": "msgpack",
      "version": "^0.8.18"
    }
  ],
  "scripts": {
    "start": "python3 tools/run_script.py start",
    "build": "python3 tools/run_script.py build"
  }
}
```

从上述配置可以看出，该 TEN App 依赖以下几个 TEN 软件包：

* `ten_runtime`：TEN Framework 的系统软件包
* `ffmpeg_demuxer`：用于解码音频和视频的 TEN Extension 软件包
* `ffmpeg_muxer`：用于编码音频和视频的 TEN Extension 软件包
* `ffmpeg_client`：用于使用另外两个 ffmpeg extension 软件包处理音频和视频的 TEN Extension 软件包
* `msgpack`：用于使用 MessagePack 协议进行消息序列化和反序列化的 TEN Protocol 软件包

实际上，在完成 TEN App 所有依赖的安装后，系统中的 TEN 软件包文件结构如下：

```text
├── ten_packages
│   ├── extension
│   │   ├── ffmpeg_client
│   │   ├── ffmpeg_demuxer
│   │   └── ffmpeg_muxer
│   ├── protocol
│   │   └── msgpack
│   └── system
│       ├── ffmpeg
│       └── ten_runtime
```

值得注意的是，尽管没有在 `manifest.json` 中显式声明，系统中还多了一个 `ffmpeg` 系统软件包。这是因为 `ffmpeg_demuxer` 和 `ffmpeg_muxer` 依赖了 `ffmpeg` 这个系统软件包，TEN Manager 会自动识别并安装这些传递依赖。

通过 TMAN Designer 的 Apps Manager 界面，开发者可以对指定的 App 执行 Install All 操作，轻松完成上述所有依赖的安装过程。点击按钮后，会看到如下界面，显示安装进度：

![安装指定 App 的所有依赖](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_install_all.png)

### 从云商店安装 TEN Extension

通过 Graph 菜单的 Open Extension Store 按钮，可以打开 TEN 云商店，并从云商店安装 TEN Extension 到当前的 TEN App。点击按钮后，会看到如下界面：

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_store.png)

在 TEN 云商店中，开发者可以搜索所需的 TEN Extension，并点击 Install 按钮安装到当前的 TEN App。安装完成后，可以在当前 TEN App 中查看已安装的 TEN Extension。

### 运行 TEN App

在 Apps Manager 中，可以对指定的 App 执行预设的运行操作。在 Apps Manager 中点击 Run 按钮后，会看到如下界面：

![运行 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_run.png)

在对话框中，开发者可以从下拉菜单中选择要执行的预设操作。通常包括以下几种操作：

* 运行 App：启动 App 并在 TMAN Designer 的消息窗口中显示 App 的输出信息。
* 编译 App：对 App 进行编译。由于 TEN Framework 支持多种编程语言，有些语言需要编译，有些则无需编译。例如，基于 C++ 开发的 TEN App 需要编译，而基于 Python 开发的 TEN App 则无需编译。

以下是一个基于 C++ 开发的 TEN App 的编译界面：

![编译 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_build.png)

编译完成后，可以继续执行 TEN App 的启动操作，系统会弹出一个窗口，实时显示 TEN App 的运行日志：

![启动 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_start.png)

## 处理 TEN Graph

### 载入已有的 TEN Graph

通过 Graph 菜单的 Open Existing Graph 按钮，可以载入已有的 TEN Graph。载入成功后，会看到类似如下界面：

![TEN Graph](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_graph_overview.png)

以这个 Graph 为例，它包含 3 个 TEN Extensions，分别是：

* `ffmpeg_demuxer`
* `ffmpeg_muxer`
* `ffmpeg_client`

每个 TEN Extension 都有其对应的输入和输出。以 `ffmpeg_demuxer` 为例，它具有以下输入与输出：

* 2 个 cmd 输入
* 1 个 cmd 输出
* 1 个 audio\_frame 输出
* 1 个 video\_frame 输出

点击输入输出的数字按钮，可以打开如下对话框，用于管理输入输出的连接：

![TEN Message Input/Output Dialog](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_msg_info_dialog.png)

### 自动排布 Graph

当开发者手动调整 TEN Graph 中的 TEN Extension 位置后，可以点击 Graph 菜单中的 Auto Layout 按钮，自动排布 TEN Graph 中的 TEN Extension。

### 打开 TEN Extension 的上下文菜单

在 Graph 中，对每个 TEN Extension 右键点击，可以打开 TEN Extension 的上下文菜单：

![TEN extension context menu](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_context_menu.png)

在上下文菜单中，可以打开 TEN Extension 的 `manifest.json` 文件、`property.json` 文件，也可以直接在 TEN Extension 的根目录下启动终端，方便开发者对该 TEN Extension 进行修改和定制：

![Open Manifest](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_context_menu_effect.png)


# 如何在 Python 扩展中运行本地 AI 模型 (/cn/docs/ten_framework/0.11.59/ai_model/how_to_run_local_model_in_python_extensions)

在 TEN framework 中，扩展可以使用第三方 AI 服务或在本地运行 AI 模型，以提高性能并降低成本。本教程介绍了如何在 Python 扩展中运行本地 AI 模型，以及如何在扩展中与之交互。

## 步骤 1：检查硬件要求

在本地运行 AI 模型之前，请确保您的硬件满足必要的要求。要验证的关键组件包括：

* **CPU/GPU**：检查模型是否需要特定的处理能力。
* **内存**：确保有足够的内存来加载和运行模型。

验证您的系统是否可以支持模型的需求，以确保顺利运行。

## 步骤 2：安装必要的软件和依赖项

硬件准备就绪后，安装所需的软件和依赖项。请按照以下步骤操作：

1. **操作系统**：确保与您的模型兼容。大多数 AI 框架都支持 Windows、macOS 和 Linux，尽管可能需要特定版本。

2. **Python 版本**：确保与 TEN Python 运行平台和模型兼容。

3. **所需的库**：安装必要的库，例如：

   * TensorFlow
   * PyTorch
   * Numpy
   * vllm

   您可以将所需的依赖项列在 `requirements.txt` 文件中，以便轻松安装。

4. **下载模型**：获取您计划运行的 AI 模型的本地版本。

## 步骤 3：实现您的 Python 扩展

以下示例说明了如何在 Python 扩展中使用 `vllm` 推理引擎实现基本的文本生成功能。

首先，在扩展中初始化本地模型：

```python
from ten_runtime import (
    Extension,
    TenEnv,
    Cmd,
    CmdResult,
)
from vllm import LLM

class TextGenerationExtension(Extension):
    def on_init(self, ten_env: TenEnv) -> None:
        self.llm = LLM(model="<model_path>")
        ten_env.on_init_done()
```

接下来，实现 `on_cmd` 方法以根据提供的输入处理文本生成：

```python
    def on_cmd(self, ten_env: TenEnv, cmd: Cmd) -> None:
        prompt = cmd.get_property_string("prompt")

        outputs = self.llm.generate(prompt)
        generated_text = outputs[0].outputs[0].text

        cmd_result = CmdResult.create(StatusCode.OK)
        cmd_result.set_property_string("result", generated_text)
        ten_env.return_result(cmd_result, cmd)
```

在此代码中，`on_cmd` 方法检索 `prompt`，使用模型生成文本，并将生成的文本作为命令结果返回。

您可以通过处理相关的输入类型来调整此方法，以实现其他功能，例如图像识别或语音转文本。

## 步骤 4：卸载模型

在扩展清理期间卸载模型以释放资源非常重要：

```python
import gc
import torch

class TextGenerationExtension(Extension):
    ...

    def on_deinit(self, ten_env: TenEnv) -> None:
        del self.llm
        gc.collect()
        torch.cuda.empty_cache()
        torch.distributed.destroy_process_group()
        print("Successfully deleted the LLM pipeline and freed GPU memory!")
        ten_env.on_deinit_done()
```

这确保了有效的内存管理，尤其是在使用 GPU 资源时。

## 总结

在 TEN Python 扩展中运行本地模型类似于原生 Python 开发。通过在适当的扩展生命周期方法中加载和卸载模型，您可以轻松集成本地 AI 模型并有效地与之交互。


# API (/cn/docs/ten_framework/0.11.59/api/api)

在 TEN Framework 中，extension 的 API 定义包含在 `manifest.json` 文件中。TEN Framework 定义 API 时，不使用 JSON Schema，而是使用一种类似 JSON 的语法来定义 API Schema。

## 为什么不使用 JSON Schema

TEN Framework 选择不使用标准 JSON Schema 的原因如下：

* **类型系统差异**：TEN Framework 的类型系统包含了许多 JSON Schema 没有的类型，例如 `int8`、`int16`、`int32`、`int64`、`uint8`、`uint16`、`uint32`、`uint64`、`float32`、`float64`、`buf`、`ptr` 等。

* **减少冗余**：在 TEN Framework 的 API 定义中，许多地方不需要具有弹性的设定。例如，message 的 schema 一定是 object 类型，不能是其他类型。如果要求开发者明确写 `"type": "object"` 反而显得多余，且容易引起混淆。

基于以上原因，TEN Framework 采用了一种更自然的定义方式，使用类似 JSON 的语法来定义 API Schema。

## API 顶层定义

API Schema 的定义放在 `manifest.json` 文件中的 `api` 字段中：

```json
{
  "api": {
    "property": {
      "properties": {...},
      "required": [...]
    },
    "cmd_in": [{
      "name": "foo",
      "property": {
        "properties": {...},
        "required": [...]
      },
      "result": {
        "property": {
          "properties": {...},
          "required": [...]
        }
      }
    }]
  }
}
```

## API Schema 定义模式

### 表达 object 类型字段

使用 `properties` 字段来描述 object 类型字段内每个属性的类型：

```json
{
  "my_object": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "age": {
        "type": "int32"
      }
    }
  }
}
```

### 表达必需和可选字段

与 object 类型字段同级的 `required` 字段，用来描述必需和可选字段：

```json
{
  "some_concept": {
    "properties": {
      "foo": {},
      "bar": {}
    },
    "required": ["foo"]
  }
}
```

合理使用 `required` 字段确保关键数据的存在：

```json
{
  "name": "critical_cmd",
  "property": {
    "properties": {
      "essential_param": {
        "type": "string"
      },
      "optional_param": {
        "type": "int32"
      }
    },
    "required": ["essential_param"]
  }
}
```

### 表达 array 类型字段

使用 `items` 字段来描述 array 类型字段内每个元素的类型：

```json
{
  "my_array": {
    "type": "array",
    "items": {
      "type": "string"
    }
  }
}
```

### type 字段的使用原则

如果一个字段只能是特定类型（由 TEN Framework 本身约束），就不需要明确写 `type` 字段。如果字段可以是多种类型，则需要明确写 `type` 字段。

例如，下面的 `some_concept` 字段如果只能是 object 类型，就不需要明确写 `type` 字段。而其中的 `foo` 字段可以是多种类型，则需要明确写 `type` 字段：

```json
{
  "some_concept": {
    "properties": {
      "foo": {
        "type": "string"
      },
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "string"
          }
        },
        "required": ["baz"]
      }
    },
    "required": ["foo"]
  }
}
```

## 表达 enum 类型字段

使用 `enum` 字段来描述枚举类型的所有可能值：

```json
{
  "api": {
    "property": {
      "properties": {
        "status": {
          "type": "string",
          "enum": ["CREATING", "ACTIVE", "DELETING", "FAILED"]
        }
      }
    }
  }
}
```

## Extension Property Schema

定义 extension 属性的完整示例：

```json
{
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        },
        "config": {
          "type": "object",
          "properties": {
            "enabled": {
              "type": "bool"
            },
            "settings": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          },
          "required": ["enabled"]
        }
      },
      "required": ["api_key", "max_tokens", "temperature", "config"]
    }
  }
}
```

## Command-like Message Schema

命令支持输入 (`cmd_in`) 和输出 (`cmd_out`)：

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "process_text",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            },
            "options": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "max_length": {
                  "type": "int32"
                }
              }
            }
          },
          "required": ["text"]
        },
        "result": {
          "property": {
            "properties": {
              "processed_text": {
                "type": "string"
              },
              "confidence": {
                "type": "float32"
              }
            },
            "required": ["processed_text"]
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "notify_completion",
        "property": {
          "properties": {
            "task_id": {
              "type": "string"
            },
            "status": {
              "type": "string"
            }
          },
          "required": ["task_id", "status"]
        }
      }
    ]
  }
}
```

## Data-like Message Schema

数据类型消息的 schema 定义：

```json
{
  "api": {
    "data_in": [
      {
        "name": "input_data",
        "property": {
          "properties": {
            "content": {
              "type": "string"
            },
            "metadata": {
              "type": "object",
              "properties": {
                "timestamp": {
                  "type": "int64"
                }
              }
            }
          },
          "required": ["content"]
        }
      }
    ],
    "data_out": [
      {
        "name": "output_data",
        "property": {
          "properties": {
            "result": {
              "type": "string"
            }
          }
        }
      }
    ],
    "audio_frame_in": [
      {
        "name": "audio_input"
      }
    ],
    "audio_frame_out": [
      {
        "name": "audio_output"
      }
    ],
    "video_frame_in": [
      {
        "name": "video_input"
      }
    ],
    "video_frame_out": [
      {
        "name": "video_output"
      }
    ]
  }
}
```

## 常见错误和解决方案

<div className="fd-steps">
  <div className="fd-step">
    ### 属性名不符合规范

    **错误示例**：属性名包含连字符或以数字开头

    ```json
    {
      "api-key": {  // 错误：包含连字符
        "type": "string"
      },
      "2nd_param": {  // 错误：以数字开头
        "type": "int32"
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "api_key": {
        "type": "string"
      },
      "second_param": {
        "type": "int32"
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### 数组类型缺少 items 定义

    **错误示例**：

    ```json
    {
      "my_array": {
        "type": "array"
        // 缺少 items 定义
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "my_array": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### 对象类型缺少 properties 定义

    **错误示例**：

    ```json
    {
      "my_object": {
        "type": "object"
        // 缺少 properties 定义
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "my_object": {
        "type": "object",
        "properties": {
          "field1": {
            "type": "string"
          }
        }
      }
    }
    ```
  </div>
</div>

## 总结

TEN Framework 的 Schema 系统提供了一个强大且结构化的方式来定义和验证数据结构，确保组件之间的一致性和类型安全。通过遵循设计原则和最佳实践，开发者可以创建健壮、可维护的 TEN 应用程序和扩展。

该系统支持丰富的类型定义、复杂的 API 描述以及灵活的消息转换功能，为构建大规模的实时多模态应用提供了坚实的基础。


# 日志 (/cn/docs/ten_framework/0.11.59/api/log)

TEN framework 允许使用不同语言开发的 extension 在同一进程中运行。这使得需要以统一的日志格式和信息在统一的日志中查看所有这些 extension 的日志，从而简化调试。

为了解决这个问题，`ten_env` 提供了一个日志 API。在 extension 的每个回调中，都可以访问 `ten_env` 的实例。使用此实例，您可以调用日志 API，将使用各种语言开发的 extension 的日志输出到统一的日志输出中。

## 日志级别

TEN framework 支持以下日志级别，按严重程度递增：

* **DEBUG** (1): 调试信息
* **INFO** (2): 一般信息
* **WARN** (3): 警告信息
* **ERROR** (4): 错误信息

## API 接口

### C++ 语言

```cpp
#include "ten_runtime/binding/cpp/ten.h"

class Extension {
public:
    void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr<ten::cmd_t> cmd) {
        // 使用便捷宏
        TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
        TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
        TEN_ENV_LOG_WARN(ten_env, "Warning message");
        TEN_ENV_LOG_ERROR(ten_env, "Error occurred");

        // 直接调用方法，支持 category 和 fields 参数
        ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
                   "Direct log call", nullptr, nullptr);
    }
};
```

C++ 提供了以下便捷宏：

* `TEN_ENV_LOG_DEBUG(ten_env, msg)`
* `TEN_ENV_LOG_INFO(ten_env, msg)`
* `TEN_ENV_LOG_WARN(ten_env, msg)`
* `TEN_ENV_LOG_ERROR(ten_env, msg)`

### Python 语言

```python
from ten_runtime import Extension, TenEnv

class MyExtension(Extension):
    def on_cmd(self, ten_env: TenEnv, cmd):
        # 使用不同级别的日志方法
        ten_env.log_debug("调试信息")
        ten_env.log_info("一般信息")
        ten_env.log_warn("警告信息")
        ten_env.log_error("错误信息")

        # 支持格式化字符串和额外参数
        test_value = "example"
        ten_env.log_info(f"处理命令，测试值: {test_value}")

        # 支持 category 和 fields 参数
        ten_env.log_info("带分类的日志", category="my_extension")
```

Python 提供了以下方法：

* `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`

### Go 语言

```go
package main

import "ten_runtime/ten"

type MyExtension struct {
    ten.DefaultExtension
}

func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    // 使用不同级别的日志方法
    tenEnv.LogDebug("调试信息")
    tenEnv.LogInfo("一般信息")
    tenEnv.LogWarn("警告信息")
    tenEnv.LogError("错误信息")

    // 支持格式化
    cmdName := cmd.GetName()
    tenEnv.LogInfo("处理命令: " + cmdName)

    // 使用完整的 Log 方法支持更多参数
    category := "my_extension"
    tenEnv.Log(ten.LogLevelInfo, "带分类的日志", &category, nil, nil)
}
```

Go 提供了以下方法：

* `LogDebug(msg string) error`
* `LogInfo(msg string) error`
* `LogWarn(msg string) error`
* `LogError(msg string) error`
* `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`

### Node.js/TypeScript 语言

```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";

class MyExtension extends Extension {
    async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
        // 使用不同级别的日志方法
        tenEnv.logDebug("调试信息");
        tenEnv.logInfo("一般信息");
        tenEnv.logWarn("警告信息");
        tenEnv.logError("错误信息");

        // 支持字符串拼接
        const cmdName = cmd.getName();
        tenEnv.logInfo("处理命令: " + cmdName);

        // 支持 category 和 fields 参数
        tenEnv.logInfo("带分类的日志", "my_extension");
    }
}
```

Node.js 提供了以下方法：

* `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
* `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
* `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
* `logError(message: string, category?: string, fields?: Value): TenError | undefined`

## 使用示例

### 基本使用

```python
# Python 示例
def on_start(self, ten_env: TenEnv):
    ten_env.log_info("extension 启动")
    ten_env.on_start_done()

def on_cmd(self, ten_env: TenEnv, cmd):
    cmd_name = cmd.get_name()
    ten_env.log_debug(f"收到命令: {cmd_name}")

    # 处理命令逻辑
    if cmd_name == "hello":
        ten_env.log_info("处理 hello 命令", category="command_handler")
        # ... 处理逻辑
    else:
        ten_env.log_warn(f"未知命令: {cmd_name}")
```

### 在线程中使用

```python
import threading
import time

def log_routine(self, ten_env: TenEnv):
    """在单独线程中记录日志"""
    while not self.stop_flag:
        self.log_count += 1
        ten_env.log_info(f"日志消息 {self.log_count}")
        time.sleep(0.05)

def on_start(self, ten_env: TenEnv):
    # 启动日志线程
    self.log_thread = threading.Thread(
        target=self.log_routine, args=(ten_env,)
    )
    self.log_thread.start()
    ten_env.on_start_done()
```

### 错误处理

```python
def on_cmd(self, ten_env: TenEnv, cmd):
    try:
        # 业务逻辑
        result = self.process_command(cmd)
        ten_env.log_info("命令处理成功")
    except ValueError as e:
        ten_env.log_error(f"参数错误: {str(e)}", category="error_handling")
    except Exception as e:
        ten_env.log_error(f"严重错误: {str(e)}", category="error_handling")
        raise
```

## 日志格式和输出

TEN framework 的日志系统会自动包含以下信息：

* **时间戳**: 日志记录的时间
* **日志级别**: DEBUG、INFO、WARN、ERROR
* **函数名**: 调用日志的函数名（如果可用）
* **文件名**: 调用日志的源文件名（如果可用）
* **行号**: 调用日志的代码行号（如果可用）
* **分类**: 可选的日志分类信息
* **消息内容**: 用户提供的日志消息

输出格式类似：

```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] 用户日志消息
```

## 高级日志配置

TEN framework 支持通过 `property.json` 文件配置高级日志功能。这允许您自定义日志的输出格式、目标和过滤规则。

### 配置结构

在 `property.json` 文件中，您可以在 `ten.log` 部分定义高级日志配置：

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "my_extension"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

### 配置组件说明

<div className="fd-steps">
  <div className="fd-step">
    #### 日志处理器 (Handlers)

    每个处理器包含三个主要组件：

    * **matchers**: 定义哪些日志消息会被此处理器处理
    * **formatter**: 定义日志的输出格式
    * **emitter**: 定义日志的输出目标
  </div>

  <div className="fd-step">
    #### 匹配器 (Matchers)

    匹配器用于过滤日志消息：

    ```json
    {
      "level": "info",      // 日志级别: "off", "debug", "info", "warn", "error"
      "category": "auth"    // 可选：日志分类（字符串）
    }
    ```

    支持的日志级别：

    * `"off"`: 关闭日志
    * `"debug"`: 调试级别
    * `"info"`: 信息级别
    * `"warn"`: 警告级别
    * `"error"`: 错误级别
  </div>

  <div className="fd-step">
    #### 格式器 (Formatter)

    控制日志的输出格式：

    ```json
    {
      "type": "plain",      // 格式类型: "plain" 或 "json"
      "colored": true       // 可选：是否启用颜色输出（仅对 plain 格式有效）
    }
    ```

    * **plain**: 人类可读的纯文本格式
    * **json**: JSON 结构化格式，便于日志分析工具处理
  </div>

  <div className="fd-step">
    #### 发射器 (Emitter)

    定义日志的输出目标：

    ##### 控制台输出

    ```json
    {
      "type": "console",
      "config": {
        "stream": "stdout",  // "stdout" 或 "stderr"
        "encryption": {      // 可选：加密配置
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### 文件输出

    ```json
    {
      "type": "file",
      "config": {
        "path": "/path/to/logfile.log",  // 日志文件路径
        "encryption": {                  // 可选：加密配置
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### OTLP 输出

    OTLP (OpenTelemetry Protocol) 输出允许您将日志发送到支持 OpenTelemetry 的后端系统，如 Jaeger、Prometheus、Grafana 等。

    ```json
    {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4317",  // OTLP 接收端点地址
        "protocol": "grpc",                   // 可选：传输协议，"grpc"（默认）或 "http"
        "headers": {                          // 可选：自定义 HTTP 头
          "Authorization": "Bearer token",
          "X-Custom-Header": "value"
        },
        "service_name": "my-service"          // 可选：服务名称，用于标识日志来源
      }
    }
    ```

    配置参数说明：

    * **endpoint**: OTLP 接收端点的 URL（必需）
    * **protocol**: 传输协议，支持 `"grpc"` 或 `"http"`，默认为 `"grpc"`
    * **headers**: 自定义 HTTP 请求头，用于认证或其他元数据传递（可选）
    * **service\_name**: 服务名称，用于在日志聚合系统中标识不同的服务（可选）
  </div>
</div>

### 配置示例

#### 示例 1: 多级别日志分离

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/errors.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 2: 按分类分离日志

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "database"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/database.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info",
              "category": "api"
            }
          ],
          "formatter": {
            "type": "plain"
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 3: 加密日志输出

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/secure/logs/app.log",
              "encryption": {
                "enabled": true,
                "algorithm": "AES-CTR",
                "params": {
                  "key": "your-32-character-encryption-key",
                  "nonce": "your-16-character-nonce"
                }
              }
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 4: OTLP 日志输出到 OpenTelemetry 收集器

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "http://localhost:4317",
              "protocol": "grpc",
              "service_name": "ten-framework-app"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "https://api.monitoring.example.com/v1/logs",
              "protocol": "http",
              "headers": {
                "Authorization": "Bearer your-api-token",
                "X-Environment": "production"
              },
              "service_name": "ten-framework-app"
            }
          }
        }
      ]
    }
  }
}
```

### 配置注意事项

1. **多个处理器**: 可以定义多个处理器来实现复杂的日志路由策略
2. **匹配优先级**: 按照配置文件中的顺序匹配处理器
3. **加密安全**: 加密密钥和 nonce 应该安全存储，不要硬编码在配置文件中
4. **文件权限**: 确保日志文件路径具有适当的写入权限
5. **性能考虑**: JSON 格式可能会比 plain 格式产生更大的日志文件
6. **OTLP 端点**:
   * 使用 gRPC 协议时，默认端口通常为 4317
   * 使用 HTTP 协议时，默认端口通常为 4318
   * 确保 OTLP 收集器端点可访问且配置正确
7. **认证**: 使用 OTLP 时，如果后端需要认证，请在 headers 中添加相应的认证信息

## 最佳实践

1. **选择合适的日志级别**：
   * 使用 `DEBUG` 记录详细的调试信息
   * 使用 `INFO` 记录正常的业务流程
   * 使用 `WARN` 记录潜在问题
   * 使用 `ERROR` 记录错误信息

2. **提供有意义的日志消息**：
   * 包含上下文信息（如命令名称、参数值等）
   * 使用描述性的消息而不是简单的标识符
   * 合理使用 category 参数对日志进行分类

3. **避免过度日志记录**：
   * 不要在高频循环中使用过多的日志
   * 在生产环境中适当调整日志级别
   * 使用高级日志配置的匹配器来过滤不必要的日志

4. **线程安全**：
   * TEN framework 的日志 API 是线程安全的
   * 可以在任何线程中安全调用日志方法

5. **高级配置建议**：
   * 为不同的模块使用不同的 category 便于日志分析
   * 在生产环境中考虑使用文件输出以便于日志持久化
   * 对敏感日志使用加密功能
   * 定期轮转日志文件以控制磁盘使用
   * 使用 JSON 格式便于与日志分析工具集成
   * 考虑使用 OTLP 输出将日志集成到现代可观测性平台（如 Grafana、Jaeger、Datadog 等）
   * OTLP 输出与 OpenTelemetry 生态系统无缝集成，便于实现分布式追踪和日志关联

整体效果如下图所示：

![多语言 extension 的统一日志输出](https://ten-framework-assets.s3.amazonaws.com/doc-assets/log.png)


# required (/cn/docs/ten_framework/0.11.59/api/required)

## `required` 字段的原则

1. **通信链路中，源的 `required` 字段必须是目标 `required` 字段的超集**

   这意味着如果目标 extension 要求某个字段为必需，那么源 extension 也必须将该字段标记为必需。这确保了消息在传递过程中不会缺少必要的字段。

2. **源和目标 `required` 字段中的相同字段名称必须具有兼容的类型**

   框架会验证字段类型的兼容性，确保数据可以正确传递和解析。类型兼容性检查包括基本类型匹配、数组项类型匹配以及对象属性类型匹配。

## `required` 字段的使用

### Extension 发送消息时

当 extension 调用 `send_<foo>(msg_X)` 或 `return_result(result_Y)` 时，框架会根据 extension 中的相应 schema 检查 `msg_X` 或 `result_Y`。如果 `msg_X` 或 `result_Y` 缺少 schema 中标记为 `required` 的任何字段，则 schema 检查将失败。

以下三种情况的处理方式相同：

1. **发送 command 且 schema 检查失败时**

   `send_<foo>` 将立即返回 `false`，错误讯息将包含 schema 检查失败的错误消息。

2. **`return_result` 未通过 schema 检查时**

   `return_result` 将返回 `false`，错误讯息将包含 schema 检查失败的错误消息，通常会显示缺少的必需字段，例如："the required properties are absent: 'foo'"。

3. **发送 data-like message（如数据、音频帧或视频帧）时**

   `send_<foo>` 将返回 `false`，错误讯息将包含 schema 检查失败的错误消息。

**注意：** 框架的 schema 系统仅在字段存在时生效。如果某个字段在 schema 中定义但在消息中不存在，且该字段未在 `required` 中列出，则不会进行类型检查。只有当字段同时存在于消息中和 schema 定义中时，才会进行类型验证。

### Extension 接收消息时

在 TEN runtime 将 `msg_X` 或 `result_Y` 传递给 extension 的 `on_<foo>()` 或结果处理程序之前，它会检查 `msg_X` 或 `result_Y` 的 schema 中定义的所有 `required` 字段是否都存在。如果缺少任何 `required` 字段，则 schema 检查将失败。

1. **传入消息是 command 时**

   TEN runtime 会将错误返回给上一个 extension，错误消息会指明缺少的必需字段。

2. **传入消息是 TEN 命令结果时**

   TEN runtime 会将结果的 `status_code` 更改为错误，添加缺少的 `required` 字段，并根据其类型将这些字段的值设置为默认值。

3. **传入消息是数据类 TEN 消息时**

   TEN runtime 将直接丢弃该数据类消息。

### 错误消息格式

当 `required` 字段验证失败时，框架会提供详细的错误信息：

* 单个缺失字段：`"the required properties are absent: 'field_name'"`
* 多个缺失字段：`"the required properties are absent: 'field1', 'field2'"`
* 嵌套对象中的缺失字段：`".nested_object: the required properties are absent: 'field_name'"`


# 类型系统 (/cn/docs/ten_framework/0.11.59/api/type_system)

## 类型

TEN framework 类型系统是 TEN framework 内用于定义值的数据类型的系统。开发者可以通过 TEN schema 声明 message 或 extension property 的类型。

TEN framework 类型系统包括基本类型和复合类型。基本类型包括以下内容：

| 类型      | 描述                     |
| ------- | ---------------------- |
| null    | 空值类型。                  |
| bool    | 布尔值，true 或 false。      |
| int8    | 8 位有符号整数。              |
| int16   | 16 位有符号整数。             |
| int32   | 32 位有符号整数。             |
| int64   | 64 位有符号整数。             |
| uint8   | 8 位无符号整数。              |
| uint16  | 16 位无符号整数。             |
| uint32  | 32 位无符号整数。             |
| uint64  | 64 位无符号整数。             |
| float32 | 单精度（32 位）IEEE 754 浮点数。 |
| float64 | 双精度（64 位）IEEE 754 浮点数。 |
| string  | Unicode (UTF-8) 字符序列。  |
| buf     | 8 位无符号字节序列。            |
| ptr     | 指向内存地址的指针。             |

复合类型包括以下内容：

| 类型     | 描述                   |
| ------ | -------------------- |
| array  | 相同类型的元素集合。           |
| object | 表示复杂的键/值对。键类型始终为字符串。 |

对于基本类型，可以使用 `get_property()` / `set_property()` 等方法访问或设置属性。例如：

```cpp
// 获取属性值
int32_t value = cmd.get_property_int32("property_name");

// 设置属性值
cmd.set_property("property_name", 100);
```

对于复合类型，通常需要使用相关的序列化方法。例如：

```go
type MyProp struct {
    Name string `json:"name"`
}

var prop MyProp
bytes, _ := json.Marshal(&prop)
cmd.SetPropertyFromJSONBytes("property_name", bytes)
```

## JSON 类型映射

当从 JSON 创建 TEN value 时，TEN framework 使用以下映射规则：

| JSON 类型 | TEN 类型映射规则                                   |
| ------- | -------------------------------------------- |
| 整数      | 如果值在 int32 范围内，优先映射为 int32；否则映射为 int64       |
| 浮点数     | 如果值在 float32 范围内，优先映射为 float32；否则映射为 float64 |
| 字符串     | 映射为 string                                   |
| 布尔值     | 映射为 bool                                     |
| null    | 映射为 null                                     |
| 数组      | 映射为 array                                    |
| 对象      | 映射为 object                                   |

需要注意的是，JSON 中的数字类型在没有明确 schema 定义的情况下，会根据数值大小自动选择合适的 TEN 类型。这确保了数据的精度和效率。

## 类型和 schema

如果指定了 TEN schema，则将根据相应的 TEN schema 确定属性类型。如果未指定 TEN schema，则将根据初始值赋值的类型确定属性类型。例如，如果初始赋值为 C++ 的 `int32_t`，则属性类型为 `int32`；如果使用 JSON 完成初始赋值，则将根据 JSON 处理规则确定类型。

当有 TEN schema 定义时，JSON 数据会被强制转换为 schema 指定的类型，如果转换失败则会报错。

## 转换规则

TEN framework 支持不同类型值之间的灵活自动转换。转换分为安全转换和不安全转换两种类型。

### 类型兼容性

TEN framework 使用类型兼容性规则来确定两个类型之间是否可以进行转换：

* **整数类型兼容**：所有整数类型（int8, int16, int32, int64, uint8, uint16, uint32, uint64）之间相互兼容
* **浮点数兼容整数**：浮点数类型可以接受整数类型的值（当整数值在浮点数表示范围内时）
* **其他类型**：bool, ptr, string, array, object, buf, null 类型只与自身兼容

### 安全转换（必须成功）

将较低精度类型转换为较高精度类型始终是安全的，并且保证成功，因为较高精度类型可以完全容纳较低精度类型的值而不会丢失数据。

在 TEN 类型系统中，安全转换规则如下：

1. 在有符号整数类型中，从较低精度到较高精度
2. 在无符号整数类型中，从较低精度到较高精度
3. 从较小范围的无符号整数到较大范围的有符号整数（当值在目标类型范围内时）
4. 在浮点数类型中，从 float32 到 float64

| 从       | 到（允许的安全转换）                                       |
| ------- | ------------------------------------------------ |
| int8    | int16 / int32 / int64                            |
| int16   | int32 / int64                                    |
| int32   | int64                                            |
| uint8   | uint16 / uint32 / uint64 / int16 / int32 / int64 |
| uint16  | uint32 / uint64 / int32 / int64                  |
| uint32  | uint64 / int64                                   |
| float32 | float64                                          |

例如：

```cpp
// 设置属性值。TEN runtime 中 `property_name` 的类型为 `int32`。
cmd.set_property("property_name", 100);

// 获取属性值。正确。
int32_t value = cmd.get_property_int32("property_name");

// 获取属性值。正确。TEN 类型系统会自动将类型转换为 `int64`。
int64_t value2 = cmd.get_property_int64("property_name");

// 获取属性值。不正确，会抛出一个错误，因为这是不安全转换。
int16_t error_type = cmd.get_property_int16("property_name");
```

### 不安全转换（可能失败）

将较高精度类型转换为较低精度类型是不安全的，因为较高精度类型的值可能超出较低精度类型的范围，从而导致数据丢失。此转换称为不安全转换。执行不安全转换时，TEN runtime 会进行严格的范围检查。如果发生溢出或精度丢失，TEN 类型系统将抛出一个错误。

在 TEN framework 类型系统中，不安全转换包括：

1. 将较高精度整数转换为较低精度整数
2. 将有符号整数转换为无符号整数（当值为负数时）
3. 将无符号整数转换为有符号整数（当值超出有符号整数范围时）
4. 将 `float64` 转换为 `float32`
5. 将浮点数转换为整数（当有小数部分或超出整数范围时）
6. 整数到浮点数的转换（当可能发生精度丢失时）

| 转换类型              | 成功条件                   |
| ----------------- | ---------------------- |
| 高精度 → 低精度         | 值必须在目标类型范围内            |
| 有符号 → 无符号         | 值必须非负且在目标类型范围内         |
| 无符号 → 有符号         | 值必须在目标类型范围内            |
| float64 → float32 | 值必须在 float32 范围内且不丢失精度 |
| 浮点数 → 整数          | 值必须在目标整数类型范围内且无小数部分    |
| 整数 → 浮点数          | 整数值必须能被浮点数精确表示，不发生精度丢失 |

#### 范围检查细节

TEN framework 实现了精确的范围检查算法：

* **整数范围检查**：使用标准的 MIN/MAX 常量进行比较
* **浮点数范围检查**：使用 `ldexp` 函数计算精确的边界值
* **精度检查**：对于整数到浮点数的转换，会进行往返转换验证以确保精度不丢失

### 转换触发时机

TEN runtime 在以下情况下执行类型转换：

1. **加载 `property.json` 时**
   * JSON 整数根据数值大小解析为相应的整数类型
   * JSON 浮点数根据数值大小解析为相应的浮点数类型
   * 如果属性有定义的 TEN schema，会根据 schema 进行强制类型转换（可能为不安全转换）

2. **调用 `set_property_from_json()` 等方法时**
   * 传递序列化的 JSON 字符串时，会根据 schema 执行类型转换

3. **属性访问时**
   * 使用 `get_property_xxx()` 方法获取属性时，会尝试将存储类型转换为请求类型
   * 这时可能触发安全或不安全转换

4. **消息传递时**
   * 当消息属性需要在不同类型之间转换时

### 错误处理

当类型转换失败时，TEN framework 会：

* 返回错误状态码和详细的错误信息
* 在错误对象中提供源类型、目标类型和失败原因
* 对于超出范围的转换，错误消息格式为："out of range of \[target\_type]"
* 对于不兼容的类型转换，错误消息格式为："unsupported conversion from `[source_type]` to `[target_type]`"

#### 常见错误示例

```cpp
// 范围溢出错误
int8_t small_value = cmd.get_property_int8("large_number");  // 如果 large_number = 200
// 错误：out of range of int8

// 类型不兼容错误
std::string str_value = cmd.get_property_string("integer_prop");
// 错误：unsupported conversion from `int32` to `string`

// 精度丢失错误
float float_value = cmd.get_property_float32("precise_double");  // 如果 precise_double 超出 float32 精度
// 错误：out of range of float32
```

## 最佳实践

1. **使用合适的类型**：根据数据的实际范围选择合适的整数和浮点数类型
2. **明确定义 schema**：在 manifest.json 中明确定义属性类型以避免类型转换歧义
3. **处理转换错误**：在代码中适当处理类型转换可能产生的错误
4. **注意 JSON 映射**：了解 JSON 数据到 TEN 类型的映射规则，避免意外的类型转换


# 开发流程 (/cn/docs/ten_framework/0.11.59/development/development_process)

基本上，通过使用 TEN framework 来完成所需的场景，有两种开发方法或视角：

1. 从小扩展到大场景
2. 从大场景到小扩展

**注意：** 无论您采取这两种开发方法中的哪一种，您都可以从头开始编写自己的扩展，或者从 TEN 云商店下载预构建的扩展。

在第一种方法中，**从小扩展到大场景**，开发过程通常遵循以下步骤：

1. 开发扩展。
2. 将扩展组装成图，并进一步创建agent，例如 [TEN-Agent](https://github.com/TEN-framework/TEN-Agent)。

在第二种方法中，**从大场景到小扩展**，开发过程通常遵循以下步骤：

1. 选择所需的agent模板，例如 [TEN-Agent](https://github.com/TEN-framework/TEN-Agent)。
2. 将所选agent模板中的某些扩展替换为您自己的扩展。

在第二种开发方法中，通过使用框架提供的扩展**独立测试**功能，您可以确保替换后的扩展行为与原扩展一致。在此过程中，开发人员只需了解与 TEN 扩展相关的概念，包括如何独立开发和测试 TEN 扩展。其他与 TEN 相关的概念（扩展开发和测试之外）则无需关注，因为这些概念已封装在所选的 TEN agent中。


# TEN 扩展开发完整指南 (/cn/docs/ten_framework/0.11.59/development/how_to_develop_with_ext)

TEN Framework 提供了丰富的扩展模板，帮助开发者快速创建扩展并完成从开发到测试的完整流程。本指南将通过实际操作演示，详细介绍如何使用 C++，Go，Python，Node.js 四种语言进行扩展开发的全流程。

## 开发前准备

### 环境要求

在开始扩展开发之前，请确保您的开发环境已正确配置。通过以下命令验证安装：

```bash
tman --version
```

正常情况下，您应该看到类似以下的版本信息输出：

```bash
TEN Framework version: <version>
```

> **重要提示**：请确保您使用的 `tman` 版本 >= 0.10.12，如果版本过低，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 安装最新版本。

### 开发流程概览

无论使用哪种编程语言，TEN 扩展开发都遵循以下标准化流程：

1. **项目创建** - 使用官方模板生成扩展项目骨架
2. **依赖安装** - 配置运行环境并安装必要的依赖包
3. **核心开发** - 实现扩展的业务逻辑和功能代码
4. **构建测试** - 编译项目（如需要）并执行单元测试
5. **调试优化** - 使用专业调试工具定位并解决问题

***

## C++ 扩展开发

C++ 扩展适用于对性能要求极高的应用场景，如实时音视频处理、高频数据计算、底层系统操作等。

### 创建项目

使用 TEN 官方提供的 C++ 扩展模板快速创建新项目：

```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```

项目创建成功后，您会得到以下完整的项目结构：

```bash
my_example_ext_cpp/
├── BUILD.gn              # GN 构建系统配置文件
├── manifest.json         # 扩展元数据和配置信息
├── property.json         # 扩展属性和参数配置
├── src/
│   └── main.cc           # 扩展核心实现代码
├── tests/
│   ├── basic.cc          # 基础功能测试用例
│   └── gtest_main.cc     # 测试框架程序入口
├── include/              # 头文件存放目录
├── tools/                # 辅助工具和脚本
└── .vscode/
    └── launch.json       # VSCode 调试配置文件
```

### 环境配置验证

进入项目目录并验证构建工具是否正常工作：

```bash
cd my_example_ext_cpp
tgn --version
```

> **期望输出**：

```bash
0.1.0
```

如果命令无法执行，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 检查 TEN Framework 开发环境是否正确安装。

### 依赖包安装

安装扩展运行时所需的全部依赖包：

```bash
tman install --standalone
```

安装成功后，您将看到类似以下的详细安装日志：

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done

🏆  Install successfully in 1 second
```

### 项目构建

TEN framework 提供了两种便捷的构建方式供您选择：

#### 方式一：手动分步构建

```bash
# 第一步：生成构建配置文件（启用独立测试模式）
tgn gen linux x64 debug -- ten_enable_standalone_test=true

# 第二步：执行项目构建
tgn build linux x64 debug
```

#### 方式二：使用 tman 快捷命令

```bash
tman run build
```

构建完成后，检查生成的可执行测试文件：

```bash
ls -la bin/
# 应该能看到：my_example_ext_cpp_test
```

### 运行测试

验证扩展功能是否正常工作：

#### 方式一：直接执行测试文件

```bash
./bin/my_example_ext_cpp_test
```

#### 方式二：使用 tman 统一命令

```bash
tman run test
```

测试执行成功时的输出示例：

```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN      ] Test.Basic
[       OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[  PASSED  ] 1 test.
```

> **成功标志**：当您看到 `[  PASSED  ] 1 test.` 时表示测试全部通过。

### 核心代码结构详解

#### 扩展主体实现（src/main.cc）

C++ 扩展的核心类需要继承自 `ten::extension_t` 基类，并实现完整的生命周期管理方法：

```cpp
class my_example_ext_cpp_t : public ten::extension_t {
 public:
  // 扩展初始化阶段 - 进行基础配置和资源预分配
  void on_init(ten::ten_env_t &ten_env) override;

  // 扩展启动阶段 - 正式开始处理业务逻辑
  void on_start(ten::ten_env_t &ten_env) override;

  // 命令处理器 - 处理来自其他扩展或应用的命令请求
  void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr<ten::cmd_t> cmd) override;

  // 数据处理器 - 处理通用数据流
  void on_data(ten::ten_env_t &ten_env, std::unique_ptr<ten::data_t> data) override;

  // 音频帧处理器 - 处理实时音频流数据
  void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::audio_frame_t> frame) override;

  // 视频帧处理器 - 处理实时视频流数据
  void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::video_frame_t> frame) override;

  // 扩展停止阶段 - 清理资源和优雅关闭
  void on_stop(ten::ten_env_t &ten_env) override;
};
```

#### 测试框架实现（tests/basic.cc）

使用 TEN 专用测试框架编写完整的单元测试：

```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    // 创建测试用的命令对象
    auto new_cmd = ten::cmd_t::create("foo");

    // 发送命令到扩展并验证响应结果
    ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
      // 在这里验证测试结果的正确性
    });
  }
};
```

### 调试环境配置

#### VSCode 集成调试

1. 安装 VSCode 的 **CodeLLDB** 扩展插件
2. 在源代码中设置断点
3. 选择 "standalone test (lldb, launch)" 调试配置
4. 按 F5 键启动调试会话

调试配置文件 `.vscode/launch.json` 的内容：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (lldb, launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

#### 命令行调试

使用经典的 GDB 调试器进行命令行调试：

```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt  # 查看完整的函数调用栈
```

***

## Go 扩展开发

Go 扩展在高性能和开发效率之间提供了很好的平衡，特别适合构建网络服务、并发处理、微服务架构等应用场景。

### 创建项目

使用 Go 扩展模板创建新项目：

```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```

项目创建成功后会显示：

```bash
🏆  Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```

完整的项目结构如下：

```bash
my_example_ext_go/
├── extension.go             # 扩展核心实现代码
├── go.mod                   # Go 模块依赖管理配置
├── manifest.json            # 扩展元数据信息
├── property.json            # 扩展属性配置
├── README.md                # 项目说明文档
├── tests/                   # 测试相关文件
│   ├── basic_tester.go      # 测试逻辑实现
│   ├── basic_tester_test.go # 测试用例定义
│   ├── main_test.go         # 测试程序入口
│   └── bin/start            # 测试启动脚本
└── .vscode/launch.json      # VSCode 调试配置
```

### 依赖包安装

安装项目运行所需的依赖包：

```bash
tman install --standalone
```

### 运行测试

验证扩展功能的正确性：

#### 方式一：使用启动脚本

```bash
./tests/bin/start
```

#### 方式二：使用 tman 命令

```bash
tman run test
```

测试成功执行的输出示例：

```bash
=== RUN   TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```

### 核心代码结构详解

#### 扩展实现（extension.go）

Go 扩展的核心实现结构：

```go
import (
    ten "ten_framework/ten_runtime"
)

type ExampleExtension struct {
    ten.DefaultExtension
}

// 扩展启动生命周期
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStart")
    tenEnv.OnStartDone()
}

// 命令处理逻辑
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    tenEnv.LogDebug("OnCmd")
    cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
    tenEnv.ReturnResult(cmdResult, nil)
}

// 扩展停止生命周期
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStop")
    tenEnv.OnStopDone()
}
```

#### 测试框架（tests/basic\_tester.go）

Go 扩展的测试框架实现：

```go
type BasicExtensionTester struct {
    ten.DefaultExtensionTester
}

func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
    // 创建用于测试的命令对象
    cmdTest, _ := ten.NewCmd("test")

    // 发送命令并处理响应结果
    tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
        if err != nil {
            panic(err)
        }
        statusCode, _ := cr.GetStatusCode()
        if statusCode != ten.StatusCodeOk {
            panic(statusCode)
        }
        tenEnvTester.StopTest(nil)
    })

    tenEnvTester.OnStartDone()
}
```

### 开发环境优化

为了提升开发体验和调试便利性，建议在扩展根目录创建 `go.work` 工作空间文件：

```go
go 1.18

use (
    .
    .ten/app/ten_packages/system/ten_runtime_go/interface
)
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Go 官方扩展，然后使用以下调试配置：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (go, launch)",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}/tests/",
      "env": {
        "CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
        "LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
      },
      "args": ["-test.v"]
    }
  ]
}
```

***

## Python 扩展开发

Python 扩展具有最高的开发效率，特别适合快速原型开发、AI/ML 应用集成、复杂业务逻辑实现等场景。

### 创建项目

使用 Python 异步扩展模板创建项目：

```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```

完整的项目结构：

```bash
my_example_ext_python/
├── extension.py         # 扩展核心实现代码
├── addon.py             # 扩展插件注册入口
├── __init__.py          # Python 包初始化文件
├── requirements.txt     # Python 依赖包清单
├── manifest.json        # 扩展元数据配置
├── property.json        # 扩展属性配置
├── tests/               # 测试相关文件
│   ├── test_basic.py    # 基础测试用例
│   ├── conftest.py      # pytest 配置文件
│   └── bin/start        # 测试启动脚本
└── .vscode/launch.json  # VSCode 调试配置
```

### 依赖包安装

安装项目所需的 Python 依赖包：

```bash
tman install --standalone
```

### 运行测试

验证 Python 扩展的功能：

#### 方式一：使用启动脚本

```bash
./tests/bin/start
```

#### 方式二：使用 tman 命令

```bash
tman run test
```

测试成功执行的输出示例：

```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py .                                                                                [100%]
============================================ 1 passed in 0.04s =======================================
```

### 核心代码结构详解

#### 扩展实现（extension.py）

Python 扩展推荐使用现代异步编程模式，以获得更好的性能和并发处理能力：

```python
from ten_runtime import (
    AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
    Cmd, StatusCode, CmdResult, Data
)

class ExampleExtension(AsyncExtension):
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_init")

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_start")
        # TODO: 在这里读取配置文件，初始化必要的资源

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_name = cmd.get_name()
        ten_env.log_debug(f"on_cmd name {cmd_name}")

        # TODO: 在这里实现具体的业务逻辑处理
        cmd_result = CmdResult.create(StatusCode.OK, cmd)
        await ten_env.return_result(cmd_result)

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_stop")
        # TODO: 在这里清理资源，进行优雅关闭
```

#### 插件注册入口（addon.py）

扩展插件的注册和创建逻辑：

```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension

@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
    def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
        ten_env.log_info("on_create_instance")
        ten_env.on_create_instance_done(ExampleExtension(name), context)
```

#### 测试实现（tests/test\_basic.py）

完整的异步测试框架实现：

```python
from ten_runtime import (
    AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
    TenError, TenErrorCode
)

class ExtensionTesterBasic(AsyncExtensionTester):
    async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
        # 创建用于测试的命令对象
        new_cmd = Cmd.create("hello_world")

        ten_env.log_debug("send hello_world")
        result, err = await ten_env.send_cmd(new_cmd)

        # 验证测试结果的正确性
        if (err is not None or result is None
            or result.get_status_code() != StatusCode.OK):
            ten_env.stop_test(TenError.create(
                TenErrorCode.ErrorCodeGeneric,
                "Failed to send hello_world"
            ))
        else:
            ten_env.stop_test()

def test_basic():
    tester = ExtensionTesterBasic()
    tester.set_test_mode_single("my_example_ext_python")
    err = tester.run()
    if err is not None:
        assert False, err.error_message()
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Python 扩展和 debugpy 调试器，使用以下配置进行调试：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (debugpy, launch)",
      "type": "debugpy",
      "request": "launch",
      "python": "/usr/bin/python3",
      "module": "pytest",
      "args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
      "env": {
        "TEN_ENABLE_PYTHON_DEBUG": "true",
        "PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
      },
      "console": "integratedTerminal"
    }
  ]
}
```

***

## Node.js 扩展开发

Node.js 扩展提供了现代 JavaScript/TypeScript 开发体验，特别适合 Web 应用集成、快速原型开发、前端技术栈扩展等场景。得益于 Node.js 的异步特性和丰富的生态系统，开发者可以轻松构建高效的实时应用。

### 创建项目

使用 TEN 官方提供的 Node.js 扩展模板快速创建新项目：

```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```

项目创建成功后，您会得到以下完整的项目结构：

```bash
my_example_ext_nodejs/
├── manifest.json         # 扩展元数据和配置信息
├── property.json         # 扩展属性和参数配置
├── package.json          # Node.js 依赖包管理配置
├── tsconfig.json         # TypeScript 编译器配置
├── src/
│   └── index.ts          # 扩展核心实现代码
├── tests/                # 测试相关文件
│   ├── src/
│   │   ├── index.ts      # 测试器实现
│   │   ├── index.spec.ts # 测试用例定义
│   │   └── main.spec.ts  # 测试框架配置
│   ├── bin/start         # 测试启动脚本
│   ├── package.json      # 测试依赖配置
│   └── tsconfig.json     # 测试 TypeScript 配置
├── tools/                # 辅助工具和脚本
└── .vscode/
    └── launch.json       # VSCode 调试配置文件
```

### 环境配置验证

进入项目目录并验证开发环境：

```bash
cd my_example_ext_nodejs
node --version
npm --version
```

> **期望输出**：

```bash
v18.0.0 或更高版本
8.0.0 或更高版本
```

### 依赖包安装

安装扩展运行时所需的全部依赖包：

```bash
tman install --standalone
```

安装成功后，您将看到类似以下的详细安装日志：

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       2/2       Done

🏆  Install successfully in 1 second
```

> **重要提示**：`tman install --standalone` 会在项目目录下创建 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录，后续的构建和测试操作都需要在这个目录下进行。

### 项目构建

Node.js 扩展使用 TypeScript 进行开发，需要先安装独立模式依赖，然后编译为 JavaScript：

#### 方式一：手动构建

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 安装独立模式的 Node.js 依赖
npm run standalone-install

# 编译 TypeScript 代码
npm run build
```

#### 方式二：使用 tman 快捷命令

```bash
tman run build
```

构建完成后，检查生成的编译结果：

```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# 应该能看到：index.js 和相关映射文件
```

### 运行测试

验证扩展功能是否正常工作：

#### 方式一：使用启动脚本

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 运行测试
./tests/bin/start
```

#### 方式二：使用 tman 统一命令

```bash
tman run test
```

测试执行成功时的输出示例：

```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case2 (1010ms)

  2 passing (2s)
```

> **成功标志**：当您看到所有测试用例显示 `✓` 并且最后显示 `passing` 时表示测试全部通过。

### 核心代码结构详解

#### 扩展主体实现（src/index.ts）

Node.js 扩展的核心类需要继承自 `Extension` 基类，并实现完整的生命周期管理方法：

```typescript
import {
  Addon,
  RegisterAddonAsExtension,
  Extension,
  TenEnv,
  Cmd,
  CmdResult,
  StatusCode,
} from "ten-runtime-nodejs";

class DefaultExtension extends Extension {
  constructor(name: string) {
    super(name);
  }

  // 扩展配置阶段 - 进行配置参数的读取和验证
  async onConfigure(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onConfigure");
  }

  // 扩展初始化阶段 - 进行基础配置和资源预分配
  async onInit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onInit");
  }

  // 扩展启动阶段 - 正式开始处理业务逻辑
  async onStart(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStart");
  }

  // 命令处理器 - 处理来自其他扩展或应用的命令请求
  async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
    console.log("DefaultExtension onCmd", cmd.getName());

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("detail", "This is a demo");
    tenEnv.returnResult(cmdResult);
  }

  // 扩展停止阶段 - 清理资源和优雅关闭
  async onStop(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStop");
  }

  // 扩展销毁阶段 - 最终清理和资源释放
  async onDeinit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onDeinit");
  }
}
```

#### 插件注册入口

扩展插件的注册和创建逻辑：

```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
  async onCreateInstance(
    _tenEnv: TenEnv,
    instanceName: string,
  ): Promise<Extension> {
    return new DefaultExtension(instanceName);
  }
}
```

#### 测试框架实现（tests/src/index.ts）

使用 TEN 专用测试框架编写完整的单元测试：

```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";

export class MyExtensionTester extends ExtensionTester {
  async onStart(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStart");

    // 模拟异步操作和测试逻辑
    new Promise((resolve) => {
      setTimeout(() => {
        resolve(true);
      }, 1000);
    }).then(() => {
      // 停止测试并返回结果
      tenEnvTester.stopTest();
    });
  }

  async onStop(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStop");
  }

  async onDeinit(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onDeinit");
  }
}
```

#### 测试用例定义（tests/src/index.spec.ts）

使用 Mocha 测试框架编写具体的测试用例：

```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";

const test_addon_name = "default_extension_nodejs";

describe("MyExtensionTester", () => {
  it("case1", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });

  it("case2", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });
});
```

### TypeScript 配置

Node.js 扩展使用现代 TypeScript 配置，支持最新的语言特性：

```json
{
  "compilerOptions": {
    "allowJs": false,
    "composite": true,
    "module": "NodeNext",
    "target": "es6",
    "moduleResolution": "NodeNext",
    "outDir": "build",
    "removeComments": false,
    "sourceMap": true,
    "noImplicitAny": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "strictNullChecks": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "rootDir": "src",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
  },
  "exclude": [
    "node_modules"
  ],
  "include": [
    "src/**/*"
  ]
}
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Node.js 官方扩展，然后使用以下调试配置：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (nodejs) (mocha, launch)",
      "type": "node",
      "request": "launch",
      "program": "node_modules/mocha/bin/_mocha",
      "stopOnEntry": true,
      "args": [
        "--no-timeouts",
        "--package",
        "package.json",
      ],
      "cwd": "${workspaceFolder}/tests",
      "env": {
        "NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
      },
      "runtimeArgs": [
        "--expose-gc",
        "--loader",
        "ts-node/esm",
        "--no-warnings",
      ]
    }
  ]
}
```

#### 命令行调试

使用 Node.js 内置调试器进行命令行调试：

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 启动调试模式
node --inspect-brk ./tests/bin/start

# 或者使用 Chrome DevTools
node --inspect ./tests/bin/start
```

### 完整开发流程总结

为了帮助开发者快速上手，这里提供一个完整的 Node.js 扩展开发流程总结：

```bash
# 1. 创建扩展项目
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example

# 2. 进入项目目录
cd my_example_ext_nodejs

# 3. 安装依赖
tman install --standalone

# 4. 进入扩展安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 5. 安装独立模式依赖
npm run standalone-install

# 6. 构建项目
npm run build

# 7. 运行测试
./tests/bin/start
```

> **工作目录说明**：
>
> * 扩展源代码位于项目根目录的 `src/` 文件夹
> * 实际的构建、测试和运行操作都在 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录下进行
> * 这种设计确保了扩展的独立性和依赖管理的正确性

### 开发最佳实践

#### 异步编程

Node.js 扩展充分利用异步编程模式，提供更好的性能：

```typescript
// 推荐：使用 async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    const result = await processCommand(cmd);
    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("result", result);
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

#### 错误处理

实现完善的错误处理机制：

```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    // 业务逻辑处理
    const result = await this.handleBusinessLogic(cmd);

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("data", JSON.stringify(result));
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    console.error("Command processing error:", error);

    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

***

## 开发总结

通过遵循本指南提供的完整开发流程，您可以高效地进行 TEN 扩展的开发、测试和调试工作。无论选择 C++，Go，Python 还是 Node.js，TEN Framework 都为您提供了完整的工具链和最佳实践，帮助您充分发挥 TEN Framework 的强大功能，构建出高性能、高可靠性的扩展应用。

每种语言都有其独特的优势和适用场景：

* **C++**：适用于对性能要求极高的场景，如实时音视频处理、高频计算
* **Go**：在高性能和开发效率之间提供平衡，适合网络服务、并发处理
* **Python**：具有最高的开发效率，特别适合AI/ML应用、快速原型开发
* **Node.js**：提供现代 Web 开发体验，适用于前端技术栈扩展、实时应用

请根据您的具体需求和团队技术栈选择最合适的开发方案。在开发过程中，建议充分利用 TEN Framework 提供的调试工具和测试框架，以确保扩展的质量和稳定性。


# TEN Framework 快速入门指南 (/cn/docs/ten_framework/0.11.59/getting-started/quick-start)

## TEN Framework 快速入门指南

> 🎯 **目标**：5分钟内搭建开发环境并运行第一个 TEN 应用

## 系统要求

**支持的操作系统**：

* Linux (x64)
* Linux (arm64)
* macOS Intel (x64)
* macOS Apple Silicon (arm64)
* Windows (x64)

**必需的软件环境**：

* Python 3.10
* Go 1.20+
* Node.js / npm（用于安装和管理 JavaScript 依赖）

> 💡 **Windows 用户注意**：推荐使用 PowerShell 执行本指南中的命令。部分命令在 CMD 中可能不兼容。

## 第一步：检查环境

在开始之前，请确保你的系统已安装以下软件：

### Python 3.10

```bash
python3 --version
# 应显示: Python 3.10.x
```

> 💡 **重要**：TEN Framework 目前仅支持 Python 3.10。推荐使用 `pyenv` 或 `venv` 创建 Python 虚拟环境：
>
> **Linux / macOS：**
>
> ```bash
> # 使用 pyenv 安装和管理 Python 3.10（推荐）
> pyenv install 3.10.18
> pyenv local 3.10.18
>
> # 或使用 venv 创建虚拟环境
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows：**
>
> 从 [https://www.python.org/downloads/release/python-31011/](https://www.python.org/downloads/release/python-31011/) 底部表格中下载 Windows installer，启动后安装 Python 3.10。
>
> 注意，点击Install Now前务必勾选 "Add Python to PATH"
>
> ```bash
> # Windows平台需要配置python3命令：
>
> # 首先，确认python的安装路径：
> where.exe python
> # 输出示例: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # 然后，以管理员身份打开 PowerShell，在python.exe同目录下创建symlink：
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # 请将上面的路径替换为 where.exe python 的实际输出路径
>
> # 验证：
> python3 --version
> ```
>
> ```powershell
> # 推荐安装后使用 venv 创建虚拟环境，在该环境中工作
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # 激活环境
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # 若有权限错误，关闭终端/IDE，右键选择“以管理员身份运行”重新打开
> # 或者改变执行策略来允许ps1脚本执行
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```

### Go 1.20+

```bash
go version
# 应显示: go version go1.20 或更高版本

$env:CGO_ENABLED = "1"
# Windows下需要显式启用CGO
```

### Node.js / npm

```bash
node --version
npm --version
# 确保 node 和 npm 命令可用
```

### GCC (MinGW)

💡仅在windows上需要安装

```bash
# 首先确定winget存在。
winget --version

# 若不存在则需要从 <https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US> 安装。
#（系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11）

#安装MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT

# 或查找后自行选择合适版本安装
winget search "mingw"

#检查安装
gcc --version
```

> 💡 **提示**：如果缺少上述环境，请先安装对应版本后再继续。

## 第二步：安装 TEN Manager (tman)

TEN Manager (tman) 是 TEN Framework 的命令行工具，用于创建项目、管理依赖和运行应用。

方式一：通过包管理器安装（推荐）

**Linux (Ubuntu/Debian):**

```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```

**macOS:**

```bash
brew install TEN-framework/ten-framework/tman
```

**Windows:**

```powershell
winget install TEN-framework.tman
```

方式二：通过安装脚本

**Linux / macOS：**

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```

**Windows：**

```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```

方式三：如果你已经克隆了仓库，使用仓库内的安装脚本

```bash
cd ten-framework

# Linux/MacOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```

> 💡 **提示**：如果系统中已经安装了 tman，安装脚本会询问是否重新安装/升级，输入 `y` 继续安装，输入 `n` 取消。
>
> **非交互式安装**（适用于自动化脚本或 CI 环境，Windows不可用）：
>
> ```bash
> # 远程安装
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # 本地安装
> yes y | bash tools/tman/install_tman.sh
> ```

**验证安装**：

```bash
tman --version
```

> 💡 **提示**：如果出现类似
>
> * `tman: command not found`（Linux/macOS）
> * `无法将"tman"项识别为 cmdlet、函数、脚本文件或可运行程序的名称`（Windows）
>
>   的提示，请确保 tman 所在目录在你的 PATH 中：
>
> **Linux / macOS：**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc  # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc   # macOS
> source ~/.bashrc  # 或 source ~/.zshrc
> ```
>
> **Windows：**
>
> ```powershell
> # 检查 PATH 中是否包含 tman
> $env:Path -split ";" | Select-String "tman"
> # 如果不包含，请
> # 1. 通过 "系统属性 → 环境变量" 手动添加，然后重启终端以应用
> # 2. 运行以下命令来临时添加路径
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```

## 第三步：创建并运行示例应用

<div className="fd-steps">
  <div className="fd-step">
    ### 创建应用

    ```bash
    # 创建一个新的 transcriber_demo 应用
    tman install app transcriber_demo
    cd transcriber_demo
    ```
  </div>

  <div className="fd-step">
    ### 安装依赖

    ```bash
    # 安装 TEN 包依赖
    tman install

    # 安装 Python 和 npm 包的依赖
    tman run install_deps
    ```

    > ⏱️ **预计时间**：1-2 分钟
  </div>

  <div className="fd-step">
    ### 构建应用

    ```bash
    tman run build
    ```

    > ⏱️ **预计时间**：30 秒
  </div>

  <div className="fd-step">
    ### 配置环境变量

    在运行应用前，需要配置 ASR（语音识别）服务的密钥。当前示例使用 Azure ASR extension，你需要在 `transcriber_demo/.env` 文件中填写相关配置：

    **Linux / macOS：**

    ```bash
    # 创建 .env 文件
    cat > .env << EOF
    # Azure Speech Service 配置
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region      # 例如：eastus
    AZURE_STT_LANGUAGE=en-US                # 根据你的音频语种或实时录音语种设置，如：zh-CN, ja-JP, ko-KR 等
    EOF
    ```

    **Windows（PowerShell）：**

    ```powershell
    # 创建 .env 文件
    @"
    # Azure Speech Service 配置
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region
    AZURE_STT_LANGUAGE=en-US
    "@ | Out-File -Encoding utf8 .env
    ```

    > 💡 **提示**：Windows 用户也可以直接用文本编辑器创建 `.env` 文件并填写上述配置。
    >
    > 💡 **提示**：如果你想使用其他 ASR extension（如 OpenAI Whisper、Google Speech 等），可以从云商店下载并替换，同样将相应的 API key 等环境变量配置在 `.env` 文件中。
  </div>

  <div className="fd-step">
    ### 运行应用

    ```bash
    tman run start
    ```

    如果一切正常，你应该看到类似以下的输出：

    ```text
    [web_audio_control_go] Web server started on port 8001
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```
  </div>

  <div className="fd-step">
    ### 体验 Demo

    打开浏览器访问：

    ```text
    http://localhost:8001
    ```

    你应该能看到 Transcriber Demo 的 Web 界面，可以尝试：

    * 点击麦克风按钮进行实时语音转录
    * 上传音频文件进行转录
    * 查看实时转录以及字幕结果
  </div>
</div>

## 恭喜！🎉

你已经成功运行了第一个 TEN 应用！

### 了解应用架构

这个 `transcriber_demo` 应用展示了 TEN Framework 的多语言扩展能力，它由以下组件构成：

* **Go** - WebSocket 服务器扩展 (`web_audio_control_go`)
* **Python** - ASR 语音识别扩展 (`azure_asr_python`)
* **TypeScript** - VTT 字幕生成和音频录制扩展 (`vtt_nodejs`)

🎯 **你已经可以运行这些多语言插件了！**

### 下一步

现在你可以：

1. **从云商店探索和下载更多插件，设计和编排你的应用**

   ```bash
   tman designer  # 启动 TMAN Designer，在云商店中探索插件、下载插件并设计编排你的应用
   ```

2. **选择一个语言，开发自己的插件**
   * 支持 Go、Python、TypeScript/JavaScript、C++ 等多种语言
   * 查看 [TEN 扩展开发完整指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) 了解详情

## 进阶：开发和构建 C++ 插件

如果你想开发和使用 C++ 扩展，推荐安装 TEN 构建工具链（tgn）。以下是完整的步骤：

<div className="fd-steps">
  <div className="fd-step">
    ### 安装 tgn 构建工具

    tgn 是 TEN Framework 的 C/C++ 构建系统，基于 Google 的 GN。

    方式一：一键安装（推荐）

    **Linux / macOS：**

    ```bash
    curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
    ```

    **Windows：**

    ```powershell
    irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
    ```

    方式二：从克隆的仓库安装

    **Linux / macOS：**

    ```bash
    # 如果你已经克隆了 TEN Framework 仓库
    cd ten-framework
    bash tools/tgn/install_tgn.sh
    ```

    **Windows：**

    ```powershell
    # 如果你已经克隆了 TEN Framework 仓库
    & ".\tools\tgn\install_tgn.ps1"
    ```

    安装完成后，确认 tgn 已添加到 PATH：

    **Linux / macOS：**

    ```bash
    # 临时添加到当前会话
    export PATH="/usr/local/ten_gn:$PATH"

    # 或永久添加到 shell 配置（推荐）
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc  # Linux
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc   # macOS
    source ~/.bashrc  # 或 source ~/.zshrc
    ```

    **Windows：**

    ```powershell
    # 安装脚本会自动添加到用户 PATH，如需手动添加：
    $env:Path += ";$env:LOCALAPPDATA\ten_gn"
    ```

    验证安装：

    ```bash
    tgn --help
    ```
  </div>

  <div className="fd-step">
    ### 安装 C++ 扩展

    以 WebRTC VAD（语音活动检测）扩展为例，从云商店安装 C++ 扩展：

    ```bash
    cd transcriber_demo
    tman install extension webrtc_vad_cpp
    ```

    > 💡 **提示**：`webrtc_vad_cpp` 是一个用 C++ 实现的语音活动检测扩展，可以在实时语音识别场景中筛选出语音部分。
  </div>

  <div className="fd-step">
    ### 编译 C++ 扩展

    安装 C++ 扩展后，需要重新构建应用以编译 C++ 代码为动态库：

    ```bash
    tman run build
    ```

    > ⏱️ **预计时间**：首次编译 C++ 扩展可能需要 1-3 分钟，具体取决于你的机器性能。
  </div>

  <div className="fd-step">
    ### 运行带有 VAD 功能的应用

    ```bash
    tman run start_with_vad
    ```

    如果一切正常，你应该看到：

    ```text
    [web_audio_control_go] Web server started on port 8001
    [vad] WebRTC VAD initialized with mode 2
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```

    现在打开浏览器访问 `http://localhost:8001`，进入麦克风实时转写页面，你会看到经过vad后的silence状态变化，当silence状态为true时，表示当前音频中没有语音。
  </div>
</div>

### C++ 开发环境要求

开发和编译 C++ 扩展需要安装 C++ 编译器（gcc 或 clang）：

**Linux:**

```bash
# Ubuntu/Debian
sudo apt-get install gcc g++

# 或使用 clang
sudo apt-get install clang
```

**macOS:**

```bash
# 安装 Xcode Command Line Tools (包含 clang)
xcode-select --install
```

**Windows:**

```powershell
# 方式一：安装 Visual Studio Build Tools（推荐）
# 从 https://visualstudio.microsoft.com/visual-cpp-build-tools/ 下载安装

# 方式二：使用 winget 安装
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

# 💡注意：
#  - 安装时选择 "使用 C++ 的桌面开发" 工作负载
#  - 且务必勾选“适用于Windows的C++ Clang工具”
```

验证编译器安装：

```bash
# 检查 gcc（Linux/macOS/MSYS2）
gcc --version
g++ --version

# 或检查 clang（Linux/macOS）
clang --version
```

**Windows（使用 MSVC）：**

```powershell
# 在 "Developer PowerShell for VS" 中执行
clang-cl --version
```

### 常见问题（C++ 扩展）

1. tgn 命令找不到

   确保已经执行安装脚本并将 tgn 添加到 PATH：

   ```bash
   export PATH="/usr/local/ten_gn:$PATH"
   ```

2. 编译失败：找不到编译器

   请参考上面的"C++ 开发环境要求"部分安装编译器。

### 了解更多

* [ten\_gn 构建系统](https://github.com/TEN-framework/ten_gn) - TEN 的 C/C++ 构建工具
* [C++ 扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) - 完整的 C++ 扩展开发文档

## 常见问题

<div className="fd-steps">
  <div className="fd-step">
    ### macOS 上 Python 库加载失败

    **问题**：运行应用时提示找不到 `libpython3.10.dylib`

    **解决方案**：

    ```bash
    export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
    ```

    建议将这行添加到 `~/.zshrc` 或 `~/.bash_profile` 中。
  </div>

  <div className="fd-step">
    ### tman 下载失败或速度很慢

    **问题**：网络连接 GitHub 受限

    **解决方案**：

    * 手动下载：访问 [Releases 页面](https://github.com/TEN-framework/ten-framework/releases) 下载对应平台的 `tman` 二进制文件
  </div>

  <div className="fd-step">
    ### 端口 8001 已被占用

    **问题**：启动时提示端口冲突

    **解决方案**：

    * 查找占用端口的进程：
      * Linux/macOS：`lsof -i :8001`
      * Windows：`netstat -ano | findstr :8001`
    * 杀掉该进程：
      * Linux/macOS：`kill -9 <PID>`
      * Windows：`taskkill /PID <PID> /F`
    * 或修改应用配置文件（`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`）中的端口号
  </div>

  <div className="fd-step">
    ### Go build 失败

    **问题**：构建时提示 Go module 相关错误

    **解决方案**：

    ```bash
    # 设置代理
    # Linux/macOS
    export GOPROXY=https://goproxy.cn,direct
    # Windows PowerShell
    $env:GOPROXY = "https://goproxy.cn,direct"

    # 清理 Go module 缓存
    go clean -modcache

    # 重新安装依赖
    cd transcriber_demo
    tman run build
    ```
  </div>

  <div className="fd-step">
    ### Python 依赖安装失败

    **问题**：pip 安装超时或失败

    **解决方案**：使用国内镜像源

    ```bash
    pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
    ```
  </div>

  <div className="fd-step">
    ### Azure语音服务报错

    **问题**：Azure语音服务相关错误，比如认证失败

    **解决方案**：检查.env文件中的配置是否正确，确保 AZURE\_STT\_KEY 和 AZURE\_STT\_REGION 填写无误
  </div>

  <div className="fd-step">
    ### Windows下权限问题

    **问题**：Windows下在访问文件时报PermissionError

    **解决方案**：右键点击PowerShell，选择“以管理员身份运行”
  </div>
</div>

## 获取帮助

* **GitHub Issues**：[https://github.com/TEN-framework/ten-framework/issues](https://github.com/TEN-framework/ten-framework/issues)
* **文档**：[https://theten.ai/cn/docs](https://theten.ai/cn/docs)
* **贡献指南**：[contributing.md](../code-of-conduct/contributing.md)


# 基于WebSocket的语音助手快速入门 (/cn/docs/ten_framework/0.11.59/getting-started/websocket-voice-assistant-quick-start)

## 基于 WebSocket 的语音助手快速入门

在不到 10 分钟内构建并运行一个基于 WebSocket 的实时语音助手。

## 简介

本指南将帮助您快速运行一个完整的语音助手 Demo，支持实时语音对话和工具调用。该示例集成了：

| 组件   | 服务商        | 用途       |
| ---- | ---------- | -------- |
| ASR  | Deepgram   | 语音转文字    |
| LLM  | OpenAI     | 对话生成     |
| TTS  | ElevenLabs | 文字转语音    |
| Tool | WeatherAPI | 天气查询（可选） |

## 环境要求

### 前置依赖

请先参考 [TEN Framework 快速入门指南](./quick-start.cn.md) 完成基础环境安装，确保以下命令通过检查：

```bash
tman check env
```

预期输出（至少 Python 和 Go 为 Ready）：

```text
✅ Operating System: Supported
✅ Python:   Ready
✅ Go:       Ready
```

### 额外工具

安装以下工具用于构建和运行：

#### Task（任务运行器）

**Linux / macOS：**

```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version  # 验证安装
```

**Windows：**

```powershell
# 1. 使用winget安装（推荐）
# 从以下地址安装winget
# https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# 系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11
winget --version # 验证安装
winget install Task.Task
task --version  # 验证安装

# 2. 使用scoop安装
irm get.scoop.sh | iex
scoop --version # 验证安装
scoop install task
task --version # 验证安装
```

#### Bun（JavaScript 包管理器）

**Linux / macOS：**

```bash
curl -fsSL https://bun.sh/install | bash
bun --version  # 验证安装
```

**Windows：**

```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version  # 验证安装
```

#### uv（Python 包管理器）

**Linux / macOS：**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version  # 验证安装
```

**Windows：**

```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version  # 验证安装
```

> 💡 安装后需要重新加载终端配置：
>
> * Linux/macOS：`source ~/.zshrc`
> * Windows：重新打开 PowerShell 终端

## 快速开始

<div className="fd-steps">
  <div className="fd-step">
    ### 克隆仓库

    ```bash
    git clone https://github.com/TEN-framework/ten-framework.git
    cd ten-framework/ai_agents/agents/examples/websocket-example
    ```
  </div>

  <div className="fd-step">
    ### 安装依赖

    ```bash
    task install
    ```

    > ⏱️ 预计耗时：2-3 分钟

    该命令会自动完成：

    * 安装 TEN 扩展包（45+ 个）
    * 编译 Go 应用和 API 服务器
    * 安装 Python 和前端依赖
  </div>

  <div className="fd-step">
    ### 配置 API Keys

    在 `ai_agents/.env` 文件中配置服务商密钥：

    **Linux / macOS：**

    ```bash
    cd ../../../  # 回到 ai_agents 目录
    vim .env
    ```

    **Windows：**

    ```powershell
    cd ..\..\..   # 回到 ai_agents 目录
    notepad .env
    ```

    从.env.example复制所有内容后，填写以下配置：

    ```bash
    # Deepgram - 语音识别
    DEEPGRAM_API_KEY=your_deepgram_api_key

    # OpenAI - 大语言模型
    OPENAI_API_KEY=your_openai_api_key
    OPENAI_MODEL=your_openai_model

    # ElevenLabs - 语音合成
    ELEVENLABS_TTS_KEY=your_elevenlabs_api_key

    # WeatherAPI - 天气工具（可选）
    WEATHERAPI_API_KEY=your_weatherapi_key
    ```
  </div>

  <div className="fd-step">
    ### 启动服务

    ```bash
    cd agents/examples/websocket-example
    task run
    ```

    服务启动后，自动运行：

    * **前端 UI**：[http://localhost:3000](http://localhost:3000)
    * **API 服务器**：[http://localhost:8080](http://localhost:8080)
    * **TMAN Designer**：[http://localhost:49483](http://localhost:49483)
  </div>

  <div className="fd-step">
    ### 与助手对话

    打开浏览器访问 [http://localhost:3000](http://localhost:3000)，点击麦克风按钮开始语音对话。

    尝试以下对话：

    * "你好，你是谁？"
    * "今天北京的天气怎么样？"（需配置 WeatherAPI）
    * "给我讲个笑话"

    **🎉 恭喜！** 您已成功运行 TEN Agent 语音助手。
  </div>
</div>

## 进阶配置

### 更换扩展

TEN Framework 支持灵活替换 ASR、LLM、TTS 等扩展。

#### 方式 1：使用 TMAN Designer（推荐）

访问 [http://localhost:49483](http://localhost:49483)，通过可视化界面：

1. 点击扩展节点查看详情
2. 选择替代扩展
3. 配置参数
4. 保存并重启

#### 方式 2：编辑配置文件

编辑 `tenapp/property.json`，修改 `predefined_graphs` 中的扩展配置。

**示例：更换 ASR 为 Azure**

```json
{
  "type": "extension",
  "name": "stt",
  "addon": "azure_asr_python",
  "extension_group": "stt",
  "property": {
    "params": {
      "key": "${env:AZURE_STT_KEY}",
      "region": "${env:AZURE_STT_REGION}",
      "language": "en-US"
    }
  }
}
```

**示例：更换 TTS 为 Minimax**

```json
{
  "type": "extension",
  "name": "tts",
  "addon": "minimax_tts_websocket_python",
  "extension_group": "tts",
  "property": {
    "params": {
      "api_key": "${env:MINIMAX_TTS_API_KEY|}",
      "group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
      "model": "speech-02-turbo",
      "audio_setting": {
        "sample_rate": 16000
      },
      "voice_setting": {
        "voice_id": "female-shaonv"
      }
    }
  }
}
```

修改后重启服务（`Ctrl+C` 然后 `task run`）。

### 配置监控

使用 Grafana 监控应用的 Metrics 和 Logs。

<div className="fd-steps">
  <div className="fd-step">
    #### 启动监控栈

    **Linux / macOS：**

    ```bash
    cd tools/grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    ```

    **Windows：**

    ```powershell
    cd tools\grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    # 若docker未安装，请通过https://apps.microsoft.com/detail/xp8cbj40xlbwkx 安装，并重启电脑后，启动docker desktop
    ```

    启动的服务：

    * **Grafana**（端口 3001）- 可视化界面
    * **Prometheus**（端口 9091）- Metrics 存储
    * **Loki**（端口 3100）- Logs 存储
    * **OpenTelemetry Collector**（端口 4317/4318）- OTLP 数据接收
  </div>

  <div className="fd-step">
    #### 配置应用遥测

    编辑 `agents/examples/websocket-example/tenapp/property.json`：

    **配置 Logs 输出**

    在 `ten.log.handlers` 数组中添加：

    ```json
    {
      "matchers": [{ "level": "info" }],
      "formatter": { "type": "json", "colored": false },
      "emitter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
    ```
  </div>
</div>

#### 配置 Metrics 输出

在 `ten.services` 中添加：

```json
{
  "telemetry": {
    "enabled": true,
    "metrics": {
      "enabled": true,
      "exporter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
  }
}
```

<div className="fd-steps">
  <div className="fd-step">
    #### 重启并访问 Grafana

    ```bash
    cd ../../../agents/examples/websocket-example
    task run
    ```

    访问 [http://localhost:3001](http://localhost:3001)（用户名/密码：`admin/admin`）
  </div>
</div>

#### 查看 Metrics

进入 `Dashboards` → **"TEN Framework"** 查看：

* 扩展启动耗时
* Cmd 处理延迟
* 消息队列等待时间

#### 查看 Logs

进入 `Explore` → 选择 `Loki`，使用查询：

* `{service_name="websocket-example"}` - 所有日志
* `{service_name="websocket-example", ten_extension_name="stt"}` - ASR 日志
* `{service_name="websocket-example", ten_extension_name="llm"}` - LLM 日志
* `{service_name="websocket-example", ten_extension_name="tts"}` - TTS 日志

> 💡 使用 `ten_extension_name` 标签精确筛选扩展日志。

停止监控服务：

**Linux / macOS：**

```bash
cd tools/grafana-monitoring
docker compose -f docker-compose.push.yml down
```

**Windows：**

```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```

## 常见问题

<details>
<summary><strong>前端依赖安装失败</strong></summary>

如果 `bun install` 报错版本不存在，切换到 npm 官方源：

```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```

</details>

<details>
<summary><strong>macOS Python 库加载失败</strong></summary>

设置 Python 动态库路径：

```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# 添加到 ~/.zshrc 永久生效
```

</details>

<details>
<summary><strong>Windows Python 库加载失败</strong></summary>

确保 Python 3.10 安装路径在系统 PATH 中：

```powershell
# 检查 Python 路径
python -c "import sys; print(sys.prefix)"

# 将 Python DLL 目录添加到 PATH（根据实际路径调整）
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```

如果安装 Python 时未勾选 "Add Python to PATH"，可以重新运行安装程序选择 "Modify" 来添加。

</details>

<details>
<summary><strong>端口被占用</strong></summary>

查找并终止占用端口的进程：

**Linux / macOS：**

```bash
lsof -i :3000  # 或 :8080
kill -9 <PID>
```

**Windows：**

```powershell
netstat -ano | findstr :3000   # 或 :8080
taskkill /PID <PID> /F
```

</details>

<details>
<summary><strong>Go module相关错误</strong></summary>

```bash
# 设置代理
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"

# 清理 Go module 缓存
go clean -modcache

# 重新安装依赖
cd ai_agents/agents/examples/websocket-example
task install
```

</details>

## 下一步

跟随这些指南，将您的语音 AI 应用带入真实世界。

* **[扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext)** - 开发自定义扩展
* **[更多示例](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - 探索其他 Agent 示例
* **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - 报告问题或请求功能
* **[文档中心](https://theten.ai/cn/docs)** - 完整的 TEN Framework 文档


# 图（Graph） (/cn/docs/ten_framework/0.11.59/graph/graph)

在 TEN framework 中，存在两种类型的图：

1. 动态图
2. 预定义图（`predefined_graph`）

|       | 动态图                          | 预定义图                                      |
| ----- | ---------------------------- | ----------------------------------------- |
| 启动时机  | 当 TEN 应用收到 `start_graph` 命令时 | 当 TEN 应用启动时，或当 TEN 应用收到 `start_graph` 命令时 |
| 图的内容  | 在 `start_graph` 命令中指定        | 在 TEN 应用的属性中预先定义                          |
| 图的 ID | 随机生成的 UUID                   | 随机生成的 UUID                                |

![两种类型的图](https://ten-framework-assets.s3.amazonaws.com/doc-assets/two_types_of_graph.png)

预定义图具有 `auto_start` 属性，用于确定该图是否在 TEN 应用启动时自动启动。

此外，预定义图还有 `singleton` 属性，用于指示该图是否只能在 TEN 应用内生成 *一个* 对应的实例。

## 图 ID 和图名称

对于每个图实例，TEN 应用内部会使用一个唯一的 UUID4 字符串作为标识，称为**图 ID**。

对于预定义图，可以分配一个有意义且易于记忆的名称，称为**图名称**。当需要指定特定预定义图时，可以直接使用其图名称。如果预定义图具有 `singleton` 属性，则表示该预定义图在 TEN 应用内只能存在一个实例。在这种情况下，TEN runtime 会使用图名称来唯一标识从该预定义图生成的单一实例。

## 动态图

当 TEN 应用收到 `start_graph` 命令并创建动态图时，系统会分配一个随机 UUID 作为该图的 ID。如果其他客户端获取了此图 ID，它们也可以连接到这个图。

动态图 ID 示例：

`123e4567-e89b-12d3-a456-426614174000`

## 预定义图

预定义图与动态图的工作原理相似，主要区别在于内容定义方式。动态图的内容包含在 `start_graph` 命令中，而预定义图的内容则由 TEN 应用预先定义。客户端只需在 `start_graph` 命令中指定预定义图的名称即可启动该图。

预定义图的主要优势在于简化使用并保护敏感信息。使用预定义图，客户端无需了解图的详细结构，这既提高了可用性，也避免了暴露图中可能包含的敏感信息。

预定义图名称示例：

`http_server`

当 TEN 应用启动时，所有设置了 `auto_start` 属性的预定义图会自动启动。

## 图定义

无论是动态图还是预定义图，其定义结构都是相同的：

```json
{
  "nodes": [
    // 节点的定义
  ],
  "connections": [
    // 通信链路的定义
  ]
}
```

关键要点：

1. 如果只有一个 TEN 应用，可以省略 `app` 字段。如果有多个应用，则必须明确指定 `app` 字段。

2. `nodes` 字段定义图中的节点，如各种扩展（extension）。

3. 每个节点在 `nodes` 字段中只能出现一次。如果多次出现同一节点，TEN framework 会在验证过程中报错。

4. 在 `nodes` 字段中定义扩展的方式如下，其中 `property` 字段为可选：

   ```json
   {
     "type": "extension",
     "name": "simple_http_server_cpp",
     "addon": "simple_http_server_cpp",
     "extension_group": "default_extension_group",
     "app": "msgpack://127.0.0.1:8001/",
     "property": {
       "root_key": "player",
       "extra_keys": ["playerName"]
     }
   }
   ```

   `addon` 字段表示该扩展是由对应插件生成的实例。

5. `connections` 字段定义图中节点间的通信链路，其中 `extension` 值表示对应节点的名称。

完整示例：

```json
{
  "nodes": [
    {
      "type": "extension",
      "app": "msgpack://127.0.0.1:8001/",
      "name": "simple_http_server_cpp",
      "addon": "simple_http_server_cpp",
      "extension_group": "default_extension_group",
      "property": {
        "root_key": "player",
        "extra_keys": ["playerName"]
      }
    }
  ],
  "connections": [
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "simple_http_server_cpp",
      "cmd": [
        {
          "name": "start",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        },
        {
          "name": "stop",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        }
      ]
    },
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "gateway",
      "cmd": [
        {
          "name": "push_status_online",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "uap"
            }
          ]
        }
      ]
    }
  ]
}
```

## 预定义图的定义

预定义图是在 TEN 应用的配置文件（`property.json`）中的 `predefined_graphs` 字段下定义的，包含 `name`，`auto_start`，`singleton` 等属性：

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        // 完整的图定义
      }
    ]
  }
}
```

完整示例：

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        "nodes": [
          {
            "type": "extension",
            "name": "simple_http_server_cpp",
            "addon": "simple_http_server_cpp",
            "extension_group": "default_extension_group",
            "property": {
              "root_key": "player",
              "extra_keys": [
                "playerName"
              ]
            }
          }
        ],
        "connections": [
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "simple_http_server_cpp",
            "cmd": [
              {
                "name": "start",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              },
              {
                "name": "stop",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              }
            ]
          },
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "gateway",
            "cmd": [
              {
                "name": "push_status_online",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "uap"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## `start_graph` 命令的定义

动态图是通过向 TEN 应用发送 `start_graph` 命令创建的，将图定义放在命令的 `ten` 字段下：

```json
{
  "ten": {
    "type": "start_graph",
    // 完整的图定义
  }
}
```

完整示例：

```json
{
  "ten": {
    "type": "start_graph",
    "nodes": [
      {
        "type": "extension",
        "name": "simple_http_server_cpp",
        "addon": "simple_http_server_cpp",
        "extension_group": "default_extension_group",
        "property": {
          "root_key": "player",
          "extra_keys": ["playerName"]
        }
      }
    ],
    "connections": [
      {
        "app": "msgpack://127.0.0.1:8001/",
        "extension": "simple_http_server_cpp",
        "cmd": [
          {
            "name": "start",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          },
          {
            "name": "stop",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          }
        ]
      },
      {
        "extension": "gateway",
        "cmd": [
          {
            "name": "push_status_online",
            "dest": [
              {
                "extension": "uap"
              }
            ]
          }
        ]
      }
    ]
  }
}
```

### 使用 `import_uri` 引用图定义

在定义预定义图时，可以使用 `import_uri` 字段引用外部图定义文件：

```json
{
  "ten": {
    "uri": "http://localhost:8001",
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": false,
        "import_uri": "../graph.json"
      }
    ]
  }
}
```

`import_uri` 可以是相对路径，绝对路径或 URL：

* 相对路径：相对于 `property.json` 文件所在目录
* 绝对路径：相对于 TEN 应用所在环境的根目录
* URL：直接作为 URL 路径处理

## 图定义的规范

* **`nodes` 字段**：在图定义中必须提供 `nodes` 数组。相比之下，`connections` 数组虽然可选，但建议提供以定义节点间通信。

* **节点 `app` 字段**：`app` 字段不能设置为 `localhost`。在单应用图中，应省略 `app` URI；在多应用图中，`app` 字段值必须与各应用 `property.json` 中的 `ten.uri` 值匹配。

* **节点唯一性**：`nodes` 数组中的每个节点表示一个特定扩展实例，必须通过 `app` 和 `name` 组合唯一标识。不允许出现重复定义，如以下无效示例：

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_1",
        "extension_group": "test"
      },
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_2",
        "extension_group": "test"
      }
    ]
  }
  ```

* **连接定义的一致性**：`connections` 中引用的所有扩展实例必须在 `nodes` 中明确定义。以下示例无效，因为 `ext_2` 未在 `nodes` 中定义：

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "ext_1",
        "addon": "addon_1",
        "extension_group": "some_group"
      }
    ],
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **通信链路整合**：同一源扩展的所有消息应分组在单个部分中，不应分散在多个定义中。以下示例不正确：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      },
      {
        "extension": "ext_1",
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  正确做法是将同一源扩展的所有消息整合在一起：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ],
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **消息目标整合**：对于特定类型的每条消息，所有目标扩展应归类在单个条目下。以下示例不正确：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          },
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  正确做法是将同一消息的所有目标整合在一起：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              },
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  注意，相同名称的消息可以在不同类型（如 `cmd` 和 `data`）中共存，不会导致冲突。

## 子图（Subgraph）

TEN framework 的核心机制是基于图结构，这种结构由节点和连接构成。子图是一种强大的复用机制，可以将复杂的图结构拆分为可重用的模块，从而提升代码的组织性和可维护性。

### 子图设计理念

图的本质是定义扩展之间数据流动的方式。子图在实现上并没有改变这一基本原理，而是作为一种语法糖，最终会被展平到所属的大图中，并使用与普通图相同的机制启动。这种设计既简化了复杂系统的开发，又不增加运行时的复杂度。

#### 设计原则

子图设计遵循以下核心原则：

1. **独立性**：任何一张图都是一个完整的图，可以单独运行，也可以作为组件嵌入到其他图中。

2. **工具友好**：子图提供额外的信息，帮助开发工具更好地理解图结构，提升开发体验，同时不增加运行时的复杂度。

3. **展平机制**：子图最终会被展平为标准的图结构，以确保性能和兼容性。

4. **简洁性**：子图的设计旨在简化开发过程，而不是增加复杂性，避免增加 JSON 结构的复杂度。

子图的设计目的是让开发者可以将一个图当作黑盒使用，无需关注其内部的复杂性。因此，在引用子图的地方，不提供特殊的修补机制来调整子图内部的状态，以减少复杂性。如果需要修改子图的定义，应直接修改原始定义，而不是在引用处进行修补。这么做是为了避免复杂性扩散，增加维护成本。例如，当将子图 A 引入到大图 B 时，如果需要调整子图 A 内部的 extension 属性的 schema 定义，应直接修改该 extension 的 schema 定义，而不是在引用处提供修改机制，因为这可能会导致不良的连锁反应，只修改了 extension 的 schema 定义，而没有修补其他地方，从而导致其他地方出错。

换句话说，子图主要提供的是一种映射机制，用来将子图内的元素暴露给外部，而不是提供机制来修补子图内部的元素。

### 子图实现机制

接下来，我们将详细介绍子图的实现机制，包括子图的定义、引用方式和展平过程。

#### 子图定义示例

以下是 `subgraph.json` 示例，可作为独立图使用，也可作为子图被其他图引用：

```json
{
  "nodes": [
    {
      // 定义名为 ext_c 的扩展节点
      "type": "extension",
      "name": "ext_c",
      "addon": "extension_c"
    },
    {
      // 定义名为 ext_d 的扩展节点
      "type": "extension",
      "name": "ext_d",
      "addon": "extension_d"
    }
  ],
  "connections": [
    {
      // ext_c 将 B 命令传输到 ext_d
      "extension": "ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_d"
            }
          ]
        }
      ]
    }
  ],
  "exposed_messages": [
    // 表示图向外部暴露的消息接口, 主要供开发工具使用，提供智能提示
    {
      // ext_d 的 B 命令暴露给外部
      "extension": "ext_d",
      "type": "cmd_in",
      "name": "B"
    }
  ],
  "exposed_properties": [
    // 表示图向外部暴露的属性
    {
      "extension": "ext_c",
      "name": "a",
      "alias": "a1"
    }
  ]
}
```

#### 子图引用示例

引用子图的 `graph.json` 示例：

```json
{
  "nodes": [
    {
      // 定义名为 ext_a 的扩展节点
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // 定义名为 ext_b 的扩展节点
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 引用子图，命名为 subgraph_1
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "./ten_packages/extension/aaa/subgraph.json",
      "property": {
        // 可以写在这边的 property 字段, 必须要存在于子图的 exposed_properties 字段,
        // 不然会是一个错误情况
        "app_id": "${env:AGORA_APP_ID}",
        "token": "<agora_token>",
        "channel": "ten_agent_test"
      }
    },
    {
      // 引用子图，命名为 subgraph_2
      "type": "subgraph",
      "name": "subgraph_2",
      "import_uri": "./ten_packages/extension/bbb/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // 第一个目标是 ext_b
              "extension": "ext_b"
            },
            {
              // 第二个目标是子图中的 ext_d, 这算是进阶用法
              "extension": "subgraph_1:ext_d"
            },
            {
              // 第三个目标是子图, 但这需要子图有明确 expose 出 B 这个 cmd_in 接口, 不然
              // 会被当成是错误情况
              "subgraph": "subgraph_2"
            }
          ]
        }
      ]
    },
    {
      // 子图中的 ext_c 将 cmd H 传输给 ext_a, 这算是进阶用法
      "extension": "subgraph_1:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 子图 subgraph_2 将 cmd H 传输给 ext_a, 不过這需要子图 expose 出 H 这个
      // cmd_out 接口, 不然会被当成是错误情况
      "subgraph": "subgraph_2",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

#### 关键概念

子图机制引入三个关键概念：

1. **消息暴露**（exposed\_messages）：
   * 子图通过 `exposed_messages` 字段声明对外暴露的接口
   * 主要供开发工具使用，实现智能提示和检查
   * 隐藏子图内部细节，提升开发体验

2. **属性暴露**（exposed\_properties）：
   * 子图通过 `exposed_properties` 字段声明对外暴露的属性
   * 主要供开发工具使用，实现智能提示和检查
   * 隐藏子图内部细节，提升开发体验

3. **子图引用与命名**：
   * 通过 `"type": "subgraph"` 引用其他图文件
   * 每个子图有唯一标识名称，作为命名空间
   * 防止不同子图中的同名元素冲突

4. **跨图连接**：
   * 通过命名空间语法（如 `subgraph_1:ext_d`）引用子图内元素, 或者通过 `subgraph` 字段引用子图
   * 使子图内元素可与主图交互
   * 构建复杂的跨图消息流

总之，`exposed_messages` 以及 `exposed_properties` 字段描述了如何从图的边界向内流动消息，开发工具则利用此字段提供智能提示，便于开发者指定消息如何流向图的边界。

#### 展平机制

最终，引用子图的图会被展平为普通图结构，确保运行时统一性和高效性。不过这个展平的结果不会被用户看到，而是在 TEN runtime 或者 TEN manager 中被展平到内存中使用。注意在展平后的图定义中，就是原始的有哪些 extensions，以及在这些 extensions 之间的连线。

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 子图中的 ext_c 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "extension_c"
    },
    {
      // 子图中的 ext_d 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "extension_d",
      "property": {
        // 在引用子图时所定义的 property 字段, 会被展平到对应的 extension 的 property 字
        // 段
        "app_id": "${env:AGORA_APP_ID}"
      }
    },
    {
      // 子图中的 ext_e 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "extension_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b"
            },
            {
              "extension": "subgraph_1_ext_d"
            },
            {
              "extension": "subgraph_2_ext_e"
            }
          ]
        }
      ]
    },
    {
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 子图内部连接也被展平纳入
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "subgraph_1_ext_d"
            }
          ]
        }
      ]
    }
  ]
  // exposed_messages 以及 exposed_properties 字段在展平过程中被舍弃
}
```

子图展平机制遵循以下规则：

1. 展平前，冒号（`:`）符号表示元素位于子图中（如 `subgraph_1:ext_d`）。

2. 展平后，子图中元素名称添加子图名称作为前缀（如 `subgraph_1_ext_c`），确保全局唯一性。

3. 展平后的图定义不再包含冒号符号，以区分展平前后的状态。

4. 子图中的内部连接会保留并纳入展平后的图中，保证功能完整性。

### 高级功能与应用

随着项目复杂度增加，子图的高级功能可帮助更好地组织和管理系统。

#### 消息转换与子图

子图完全支持消息转换（msg\_conversion）机制，用于处理不同接口间的消息格式转换：

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "http://a.b.c.d/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1:ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "subgraph": "subgraph_2",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

开发工具可利用 `exposed_messages` 信息，提示兼容性并提供消息转换配置界面。配置完成后，转换规则会自动写入图定义，简化开发流程。

展平后，消息转换规则会正确保留，确保运行时行为与设计意图一致：

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "addon_c"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "addon_d"
    },
    {
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "addon_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1_ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_2_ext_e",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

通过这种方式，开发者可灵活处理不同组件间的消息格式差异，无需关心底层实现细节。

## "subgraph\_1:ext\_c" 的意义

在前面的示例中，`subgraph_1:ext_c` 中的 `subgraph_1` 不是表示子图本身，而是引用 `nodes` 字段中的一个元素。在子图应用中，`subgraph_1` 可视为一个命名空间。

通过这种机制，可以在图的连接中指定不同位置的扩展作为源或目标，实现跨多个图的连接。

核心概念是：图的连接本质上是扩展之间的连接。通过这种机制，可以在 `nodes` 字段中指定不同类型的元素作为连接的源或目标，实现跨多个子图甚至多个图的连接。

## 跨多个图的连接

除了子图外，还可以通过类似机制在一个图中指定与其他图的连接。

### 与同一 TEN 应用中的预定义图连接

示例：如何在一个图中连接到同一 TEN 应用中的预定义图：

```json
{
  "nodes": [
    {
      // 定义名为 ext_a 的扩展
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // 定义名为 ext_b 的扩展
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 引用其他图, 图 ID 是 default
      "type": "graph",
      "graph_id": "default"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // 第一个目标是 ext_b
              "extension": "ext_b"
            },
            {
              // 第二个目标是另一张图中的 ext_d
              "extension": "default:ext_d"
            }
          ]
        }
      ]
    },
    {
      // 另一张图中的 ext_c 将 cmd H 传输给 ext_a
      "extension": "default:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 另一张图将 cmd X 传输给 ext_a, 但这需要另一张图 expose 出 X 这个 cmd_out 接口,
      // 不然会被当成是错误情况. 这个错误情况如果可以被静态检查出来, 则会在静态检查的时候报
      // 错, 或者在运行时报错.
      "graph": "default",
      "cmd": [
        {
          "name": "X",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

### graph node 的种类

1. Subgraph

   ```json
   {
     "type": "subgraph",
     "name": "subgraph_1", // namespace 的意义
     "import_uri": "http://a.b.c.d/subgraph.json"
   }
   ```

2. Local graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": true
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": false
   }
   ```

3. Remote graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": true,
     "app": "msgpack:://127.0.0.1:8002/"
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": false,
     "app": "msgpack:://127.0.0.1:8002/"
   }
   ```


# 消息转换 (/cn/docs/ten_framework/0.11.59/graph/msg_conversion)

## 消息转换

TEN Framework 提供了强大的消息转换功能，允许您在消息传递过程中对消息内容进行转换和修改。通过配置转换规则，您可以实现消息字段的重命名、值的替换、以及复杂的数据结构转换。

### 基本配置

消息转换通过 `msg_conversion` 配置项进行设置：

```json
{
  "msg_conversion": {
    "type": "per_property",
    "keep_original": false,
    "rules": [
      {
        "path": "user.name",
        "conversion_mode": "fixed_value",
        "value": "default_user"
      },
      {
        "path": "data.items[0].id",
        "conversion_mode": "from_original",
        "original_path": "original_id"
      }
    ]
  }
}
```

### 配置参数说明

* **type**: 转换类型，目前支持 `per_property`（按属性转换）
* **keep\_original**: 是否保留原始消息内容，设为 `false` 时只保留转换后的字段
* **rules**: 转换规则数组，定义具体的转换逻辑

### 转换规则

每个转换规则包含以下字段：

* **path**: 目标字段路径，指定要设置的消息字段位置
* **conversion\_mode**: 转换模式
  * `fixed_value`: 设置固定值
  * `from_original`: 从原始消息的指定路径复制值
* **value**: 当转换模式为 `fixed_value` 时使用的固定值
* **original\_path**: 当转换模式为 `from_original` 时，指定原始消息中的源字段路径

### 路径表达式

消息转换中的路径表达式支持多种访问方式：

* **对象属性访问**: `user.profile.name` - 访问嵌套对象的属性
* **数组索引访问**: `items[0].name` - 访问数组指定索引的元素
* **混合路径访问**: `user.tags[1].value` - 结合对象和数组访问

### 使用场景

消息转换功能适用于以下场景：

* **字段重命名**: 将消息中的字段名称转换为目标系统要求的格式
* **数据格式转换**: 调整消息的数据结构以适配不同的处理模块
* **默认值设置**: 为缺失的字段提供默认值
* **数据清洗**: 过滤或转换不符合要求的数据内容


# Go (/cn/docs/ten_framework/0.11.59/language_binding/go)

让我们使用 TEN framework 构建一个 Go 应用程序。

## 创建 TEN 应用

首先，我们将通过集成几个预构建的 TEN 包来创建一个基本的 TEN Go 应用。请按照以下步骤操作：

```shell title="终端"
tman install app default_app_go
cd default_app_go

tman install protocol msgpack
tman install extension_group default_extension_group
```

### 安装默认的 TEN 扩展

接下来，安装一个用 Go 编写的默认 TEN 扩展：

```shell title="终端"
tman install extension default_extension_go
```

### 声明预构建的自动启动图

现在，我们将修改 TEN 应用的 `property.json` 文件，以包含图的声明。这将确保默认扩展在 TEN 应用启动时自动启动。

```json title="property.json"
"predefined_graphs": [
  {
    "name": "default",
    "auto_start": true,
    "nodes": [
      {
        "type": "extension_group",
        "name": "default_extension_group",
        "addon": "default_extension_group"
      },
      {
        "type": "extension",
        "name": "default_extension_go",
        "addon": "default_extension_go",
        "extension_group": "default_extension_group"
      }
    ]
  }
]
```

## 构建应用

与标准的 Go 项目不同，TEN Go 应用使用 CGo，因此您需要在构建之前设置某些环境变量。TEN runtime Go 绑定系统包中已经提供了一个构建脚本，因此您可以使用一个命令构建应用：

```shell title="终端"
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```

编译后的二进制文件 `main` 将在 `/bin` 文件夹中生成。

## 启动应用

由于需要设置一些环境变量，建议使用提供的脚本来启动应用：

```shell title="终端"
./bin/start
```

## 调试

如果您使用 Visual Studio Code (VSCode) 作为开发环境，则可以使用以下配置来调试 Go/C 代码。

### 调试 Go 代码

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Go app (launch)",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "cwd": "${workspaceFolder}",
      "output": "${workspaceFolder}/bin/tmp",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

### 调试 C 代码

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "C (launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/main",
      "cwd": "${workspaceFolder}",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

## CGO

### 生成的代码

当 Go 与 C 接口时，`cgo` 工具至关重要。当从 Go 调用 C 函数时，`cgo` 会将 Go 源文件转换为多个输出文件，包括 Go 和 C 源文件。然后，使用 gcc 或 clang 等编译器编译这些生成的 C 源文件。

使用以下命令生成这些 C/Go 源文件：

```bash title="终端"
mkdir out
go tool cgo -objdir out
```

示例：

```bash title="终端"
go tool cgo -objdir out cmd.go error.go
```

生成的文件包括：

```bash title="终端"
├── _cgo_export.c
├── _cgo_export.h
├── _cgo_flags
├── _cgo_gotypes.go
├── _cgo_main.c
├── _cgo_.o
├── cmd.cgo1.go
├── cmd.cgo2.c
├── error.cgo1.go
└── error.cgo2.c
```

* **`_cgo_export.h`** 是 cgo 上下文中 Go 和 C 之间互操作的关键。它包含可以从 C 访问的 Go 函数的必要声明。

  此文件是根据使用 `//export` 指令显式导出的 Go 函数自动生成的。它还包括与这些函数中使用的 Go 类型对应的 C 类型的定义，确保两种语言之间的兼容性。

  类型定义示例：

```c title=".h"
  typedef signed char GoInt8;
  typedef unsigned char GoUint8;
  typedef short GoInt16;
  typedef unsigned short GoUint16;
  typedef int GoInt32;
  typedef unsigned int GoUint32;
  typedef long long GoInt64;
  typedef unsigned long long GoUint64;
  typedef GoInt64 GoInt;
  typedef GoUint64 GoUint;
  typedef size_t GoUintptr;
  typedef float GoFloat32;
  typedef double GoFloat64;
```

* **`_cgo_gotypes.go`** 包含 C 中定义并在 Go 中使用的相应 Go 类型。

  例如，如果您在头文件 `common.h` 中定义了一个结构体 `ten_go_error_t` 并在 Go 中使用 `C.ten_go_error_t`，则在 `_cgo_gotypes.go` 中会有一个相应的 Go 类型：

```go title=".go"
  package ten

  type _Ctype_ten_go_status_t = _Ctype_struct_ten_go_status_t

  type _Ctype_struct_ten_go_status_t struct {
      err_no   _Ctype_int
      msg_size _Ctype_uint8_t
      err_msg  [256]_Ctype_char
      _        [3]byte
  }
```

* **`cmd.cgo1.go`** 是 `cgo` 生成的与原始 Go 源文件 `cmd.go` 对应的文件。它将对 C 函数和类型的直接调用替换为对 `cgo` 提供的生成的 Go 函数和类型的调用。

  示例：

```go title=".go"
  package ten

  func NewCmd(cmdName string) (Cmd, error) {
      // ...
      cStatus := func() _Ctype_struct_ten_go_status_t {
          var _cgo0 _cgo_unsafe.Pointer = unsafe.Pointer(unsafe.StringData(cmdName))
          var _cgo1 _Ctype_int = _Ctype_int(len(cmdName))
          _cgoBase2 := &bridge
          _cgo2 := _cgoBase2
          _cgoCheckPointer(_cgoBase2, 0 == 0)
          return _Cfunc_ten_go_cmd_create_custom_cmd(_cgo0, _cgo1, _cgo2)
      }()
      // ...
  }
```

* **`cmd.cgo2.c`** 是从 Go 调用的原始 C 函数的包装器。

  示例：

```c title=".c"
  CGO_NO_SANITIZE_THREAD void _cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd(void *v) {
      struct {
          const void* p0;
          int p1;
          char __pad12[4];
          ten_go_msg_t** p2;
          ten_go_error_t r;
      } __attribute__((__packed__, __gcc_struct__)) *_cgo_a = v;
      char *_cgo_stktop = _cgo_topofstack();
      __typeof__(_cgo_a->r) _cgo_r;
      _cgo_tsan_acquire();
      _cgo_r = ten_go_cmd_create_cmd(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2);
      _cgo_tsan_release();
      _cgo_a = (void*)((char*)_cgo_a + (_cgo_topofstack() - _cgo_stktop));
      _cgo_a->r = _cgo_r;
      _cgo_msan_write(&_cgo_a->r, sizeof(_cgo_a->r));
  }
```

因此，从 Go 调用 `C.ten_go_cmd_create_cmd()` 的调用顺序为：

```text
_Cfunc_ten_go_cmd_create_custom_cmd (自动生成的 Go)
           |
           V
   _cgo_runtime_cgocall (Go)
           |
           V
      entrysyscall     // 将 Go 堆栈切换到 C 堆栈
           |
           V
_cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd (自动生成的 C)
           |
           V
   ten_go_cmd_create_cmd (C)
           |
           V
      exitsyscall
```

### 不完整类型

如前所述，`cgo` 工具会根据通过 `import "C"` 导入的 C 头文件在 `_cgo_gotypes.go` 中生成相应的 Go 类型。如果结构体在 C 头文件中是不透明的，则 `cgo` 工具会生成一个不完整类型。

不透明 C 结构体的示例：

```c title=".h"
typedef struct ten_go_msg_t ten_go_msg_t;
```

`cgo` 工具将在 Go 中生成一个不完整类型：

```go title=".go"
type _Ctype_ten_go_msg_t = _Ctype_struct_ten_go_msg_t
type _Ctype_struct_ten_go_msg_t _cgopackage.Incomplete
```

#### 如果在 Go 中使用不完整类型会发生什么？

* **无法分配不完整类型。**

  由于 `sys.NotInHeap` 无法在 Go 堆上分配，因此 `new` 或 `make` 等操作将不起作用。尝试在 Go 中创建不透明结构体的新实例将导致编译器错误：

```go title=".go"
  msg := new(C.ten_go_msg_t) // 错误：无法在 Go 中分配；它是不完整的（或不可分配的）
```

* **无法将指向不完整类型的指针直接传递给 C。**

  如果您的 C 函数具有指向不透明结构体的指针作为参数，则根据 cgo 规则，将 Go 指针直接传递给此不完整类型的 C 函数将不起作用。Go 编译器将要求指针“固定”，以确保它符合 Go 的垃圾收集器 (GC) 约束。

### 使用 `C.uintptr_t` 而不是指向不透明结构体的指针的规则

* **优点：**

  * Go 中的 `C.uintptr_t` 和 `uintptr` 是足够大的整数，可以保存 C 指针，从而避免从 Go 传递到 C 时进行内存分配或转换。

* **限制：**

  * `uintptr` 是一个整数，而不是指针。它表示没有指针运算的地址的位模式。
  * `uintptr` 无法在 Go 中解引用。将其转换为 `unsafe.Pointer` 可能会导致 Go 的 GC 出现问题。
  * 由于 `uintptr` 是一个整数，因此无法将 `nil` 或 `NULL` 传递给 C。使用 `0` 而不是 `nil` 来表示空地址。


# Manifest (/cn/docs/ten_framework/0.11.59/metadata_system/manifest)

## 什么是 Manifest 文件

Manifest 文件是 TEN Framework 中用于存储软件包元数据的重要配置文件。它定义了扩展（Extension）或应用（App）的基本信息、依赖关系、API 接口、脚本配置等关键信息。

## Manifest 文件的作用

* **包标识**：定义包的类型、名称和版本信息
* **依赖管理**：声明包所需的依赖项及其版本要求
* **API 定义**：描述包对外提供的数据接口和命令接口
* **打包配置**：指定需要包含在发布版本中的文件
* **脚本管理**：定义构建、测试等自动化脚本

## Extension Manifest 示例

以下是一个 AI 聊天扩展的完整 Manifest 示例：

```json
{
  "type": "extension",
  "name": "openai_chatgpt_python",
  "version": "0.1.0",
  "tags": ["ai", "llm", "python"],
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "^0.8.0"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "*.py",
      "README.md"
    ]
  },
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        }
      }
    },
    "data_in": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "data_out": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_in": [
      {
        "name": "flush"
      },
      {
        "name": "tool_register",
        "property": {
          "properties": {
            "tool": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "description": {
                  "type": "string"
                },
                "parameters": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                }
              },
              "required": ["name", "description", "parameters"]
            }
          }
        },
        "result": {
          "property": {
            "properties": {
              "response": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "tool_call",
        "property": {
          "properties": {
            "name": {
              "type": "string"
            },
            "args": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    ]
  },
  "scripts": {
    "test": "python -m pytest tests/",
    "build": "python setup.py build"
  }
}
```

## App Manifest 示例

以下是一个语音助手应用的 Manifest 示例：

```json
{
  "type": "app",
  "name": "voice_assistant_app",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "^0.8.0"
    }
  ],
  "api": {
    "property": {
      "properties": {
        "app_id": {
          "type": "string"
        },
        "log_level": {
          "type": "uint8"
        },
        "server_config": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "int32"
            },
            "ssl_enabled": {
              "type": "bool"
            }
          },
          "required": ["host", "port"]
        }
      }
    }
  }
}
```

## 主要字段说明

### 基本信息

* **type**：包的类型，可以是 `extension` 或 `app`
* **name**：包的唯一标识名称
* **version**：遵循语义化版本规范的版本号
* **tags**：用于分类和搜索的标签数组

### 依赖配置

* **dependencies**：依赖项列表，每个依赖包含类型、名称和版本要求

### 打包配置

* **package.include**：指定打包时需要包含的文件和目录

### API 配置

* **api.property**：定义扩展的配置属性
* **api.data\_in/data\_out**：定义数据输入和输出接口
* **api.cmd\_in/cmd\_out**：定义命令输入和输出接口

### 脚本配置

* **scripts**：定义各种自动化脚本，如测试、构建等

## 最佳实践

1. **版本管理**：使用语义化版本控制，确保版本号的一致性
2. **依赖声明**：明确声明所有必需的依赖项及其版本范围
3. **API 设计**：清晰定义输入输出接口，便于其他组件集成
4. **文档完整**：确保 Manifest 中的配置与实际实现保持一致


# 元数据系统 (/cn/docs/ten_framework/0.11.59/metadata_system/metadata_system)

TEN framework 在各种类型的包中采用一致的元数据系统。

## 元数据类型

在 TEN 元数据系统中，有两种主要类型的元数据：

### 1. **Manifest（清单）**

* **位置：** 存储在相关 TEN 包的根目录中，文件名为 `manifest.json`。
* **内容：**

  * **包名称：** TEN 包的名称。
  * **包版本：** TEN 包的版本，遵循语义版本控制。
  * **API Schema：** 定义与包的属性和输入/输出消息相关的模式。

  > ⚠️ **注意：** `manifest.json` 中的 API schema *不是* JSON schema。

### 2. **Property（属性）**

* **位置：** 通常存储在 TEN 包根目录的 `property.json` 文件中。
* **目的：** `property.json` 文件存储初始属性值，这些属性值在运行时是可读写的。这意味着可以在 TEN runtime 运行时修改属性。

## 清单文件 (Manifest File)

`manifest.json` 文件充当 TEN 包的蓝图，定义其元数据、属性以及它处理的输入/输出消息。

### `manifest.json` 示例

```json
{
  "type": "app",
  "name": "default_app_cpp",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.0"
    }
  ],
  "api": {
    "property": {
      "exampleInt8": {
        "type": "int8"
      },
      "exampleString": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "detail": {
              "type": "string"
            },
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

### 清单中 TEN 模式的目的

`manifest.json` 中的 TEN 模式为 TEN runtime 提供了关于 extension 的外部 API 的元数据，包括属性和消息的类型信息。这有助于运行平台在操作期间正确处理 extension 的属性和消息。

### 何时使用 TEN 模式

1. **属性验证：** 当 TEN runtime 获取/设置 TEN 包或消息的属性时。
2. **数据转换：** 使用 `from_json` API 将 JSON 文档转换为 TEN 包或消息属性时。
3. **兼容性检查：** 当 TEN 管理器根据 TEN 模式检查消息是否可以从源 extension 输出到目标 extension 时。

## 属性管理

在 TEN framework 中，有两种类型的属性：

1. **消息属性：** 这些属性与框架中 extension 之间交换的消息相关联。消息属性定义消息中携带的特定数据或参数，例如命令参数、数据负载或与音频/视频帧相关的元数据。
2. **TEN 包属性：** 这些属性与 TEN 包本身（例如 extension）相关联。TEN 包属性定义特定于包的配置或设置。例如，extension 可能具有配置其行为的属性，例如运行时设置、初始化参数或其他配置数据。

这两种类型的属性都在 TEN framework 中进行管理，但服务于不同的目的：一种侧重于组件之间的通信（消息属性），另一种侧重于组件本身的配置和操作（TEN 包属性）。

![属性系统](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_system.png)

### 定义 TEN 包属性

`property.json` 文件定义 TEN 包的属性。以下是一个示例：

```json
{
  "prop_1_name": 0,
  "prop_2_name": "prop_2_value",
  "prop_3_name": ["hello", "prop_3_sub_value"],
  "prop_4_name": {
    "prop_4_1_name": 1,
    "prop_4_2_name": "prop_4_sub_value"
  }
}
```

> ⚠️ **注意：** `property.json` 文件中的每个属性名称必须是唯一的。

### 属性的 TEN 模式

您可以在 `manifest.json` 文件中为属性定义 TEN 模式，使 TEN runtime 能够更有效地处理这些属性。如果属性没有相应的 TEN 模式，则运行平台将使用默认的 JSON 处理方法（例如，将所有 JSON 数字视为 `float64`）。如果提供了 TEN 模式，则运行平台将使用它来验证和相应地处理属性。

| 属性 | TEN 模式 | 效果                                          |
| -- | ------ | ------------------------------------------- |
| 是  | 是      | TEN runtime 根据 TEN 模式验证属性值（例如，检查类型）。        |
| 是  | 否      | TEN runtime 使用默认处理，将所有 JSON 数字视为 `float64`。 |

### 从 Extension 访问属性

TEN runtime 为 extension 提供了访问各种属性的 API。

### 从 Extension 访问 App 的属性

> ⚠️ **注意：** 本节所描述的行为还未被实现，是未来的 roadmap。

TEN framework 提供了强大的属性访问机制，让 extension 能够读取和修改 app 的属性。通过 get/set property API，您可以访问所有属性，包括 app 的属性。当 API 参数以 `app:` 前缀开头时，系统会识别这是对 app 属性的操作请求。

各语言实现的 API 设计考量：

* **Go**：提供不带 callback 的同步版本即可，简洁高效。
* **Node.js**：实现为不带 callback 的 async function，符合 Node.js 的异步编程模型。
* **Python**：理想情况下应提供带 callback 和不带 callback 的版本。目前可先实现不带 callback 的同步版本，但在访问 `app:` 属性时会使用 event wait，可能影响性能。
* **Async Python**：实现为不带 callback 的 async function，与 Python 异步模型一致。
* **C++**：应提供带 callback 和不带 callback 的版本。目前可先实现不带 callback 的同步版本，但在访问 `app:` 属性时会使用 event wait，可能影响性能。

这种设计使 app 属性的增删改查通过统一的 get/set\_property API 实现，保持了接口的一致性，且不会影响 message 属性的相关 API。

关于 `ten:` 命名空间字段的处理，采用以下规则：

1. `get/set_property_to/from_json(nullptr)` 操作不处理 `ten:` 字段
2. 要访问 `ten:` 字段，必须明确指定如 `get/set_property("ten:xxx")` 的完整路径
3. 当使用 `get/set_property_to/from_json()` 时，只有明确指定 `"ten:xxx"` 形式的 key 时才会处理 `ten:` 字段

这种设计既保持了 API 的简洁性，又避免了为特定需求创建额外命令的复杂性。

### 在 `start_graph` 命令中指定属性值

与 TEN 包相关的属性值可以在 `start_graph` 命令中指定。TEN runtime 根据 TEN 模式（如果已定义）处理这些属性，并将它们存储在相应的 TEN 包实例中。

#### `start_graph` 命令中属性规范的示例

```json
{
  "nodes": [
    {
      "type": "extension_group",
      "name": "foo_extension_group",
      "addon": "foo_extension_group_addon"
    },
    {
      "type": "extension",
      "name": "bar_extension",
      "extension_group": "foo_extension_group",
      "property": {
        "prop_1_name": 0,
        "prop_2_name": "prop_2_value",
        "prop_3_name": ["hello", "prop_3_sub_value"],
        "prop_4_name": {
          "prop_4_1_name": 1,
          "prop_4_2_name": "prop_4_sub_value"
        }
      }
    }
  ]
}
```


# Property (/cn/docs/ten_framework/0.11.59/metadata_system/property)

## Property 文件概述

Property 文件是 TEN Framework 中用来存储扩展（extension）属性值的配置文件。它定义了扩展的运行时参数和配置选项，是扩展正常运行的重要组成部分。

## Property 文件格式

Property 文件采用 JSON 格式，包含扩展运行所需的各种配置参数，如 API 密钥、模型配置、运行参数等。

### 基础属性示例

以下是一个简单的 Property 文件示例：

```json
{
  "api_key": "your-api-key-here",
  "max_tokens": 1000,
  "temperature": 0.7,
  "model_config": {
    "name": "gpt-4",
    "version": "2024-01-01"
  },
  "allowed_languages": ["en", "zh", "es"]
}
```

### AI 模型配置示例

针对 AI 模型扩展的 Property 文件配置示例：

```json
{
  "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxx",
  "model": "gpt-4",
  "max_tokens": 4096,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "prompt": "You are a helpful AI assistant.",
  "greeting": "Hello! How can I help you today?",
  "max_memory_length": 10,
  "vendor": "openai"
}
```

## 图配置

Property 文件还可以包含图配置（Graph Configuration），定义扩展之间的连接关系和数据流。

### 预定义图配置示例

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "nodes": [
          {
            "type": "extension",
            "name": "my_extension",
            "addon": "my_extension_addon",
            "property": {
              "setting1": "value1",
              "setting2": 42
            }
          }
        ],
        "connections": [
          {
            "extension": "source_extension",
            "cmd": [
              {
                "name": "process_cmd",
                "dest": [
                  {
                    "extension": "target_extension"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## 使用说明

* Property 文件必须是有效的 JSON 格式
* 文件中的属性值会在扩展启动时被加载
* 可以通过修改 Property 文件来调整扩展的运行行为
* 图配置部分定义了扩展间的连接和数据流向


# 插件系统 (/cn/docs/ten_framework/0.11.59/software_packages/addon_system)

TEN framework 是一个软件模块系统，其中每个软件模块都由 **插件 (addon)** 的概念定义。本质上，TEN framework 中的每个软件模块都是一个插件。例如，用户可以定义以下类型的 TEN framework 插件：

1. TEN **扩展 (extension)** 插件
2. TEN **扩展组 (extension group)** 插件
3. TEN **协议 (protocol)** 插件

## 加载插件所需的信息

要将插件加载到 TEN runtime，需要三个信息：

1. **插件名称**

   插件名称在整个 TEN 应用中必须全局唯一。将多个同名插件加载到 TEN runtime 是未定义行为，可能会导致第二个插件实例无法正确加载。

2. **插件文件夹位置**

   在 TEN framework 中，加载插件时，用户必须告知 TEN runtime 插件目录的所在位置。此目录将作为插件的 `base_dir`。TEN runtime 将使用此基本目录加载与插件相关的其他文件，例如插件的 `manifest.json` 和 `property.json` 文件。

   基本上，在设计 TEN framework 的插件注册 API 时，其目标是自动检测插件所在的目录，因此用户无需显式指定 `base_dir`。但是，在某些 TEN framework 无法自动检测 `base_dir` 的特殊情况下，用户将需要为 TEN runtime 显式指定插件所在的文件夹。

3. **插件实例本身**

## 将插件注册到 TEN runtime

有两种方法可以将插件注册到 TEN runtime ：

1. 显式调用 API 以将插件的 `base_dir` 注册到 TEN runtime。

   **注意：** 目前，只有独立测试框架提供此类 API。

2. 将插件放置在 TEN 应用文件夹树中的指定位置。

通过将插件放置在指定位置，TEN 应用将在启动时自动加载这些插件。目前，此指定位置位于 TEN 应用的 `ten_packages/` 目录下。如果插件以此格式放置，它们将在 TEN 应用启动时自动加载。

```text
.
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    └── protocol/
        └── <protocol_a>/
```


# 应用 (/cn/docs/ten_framework/0.11.59/software_packages/app)

应用程序是 TEN 世界中的容器，包含组和扩展。
应用程序可以作为放置在 Docker 容器内的进程（主要在云中使用）。
应用程序也可以作为嵌入在现有应用程序内的线程（在客户端或服务器上）。
⇒ TEN 可以在现有项目上执行本地化修改（小幅增强），而无需根据 TEN 重写整个项目。

![将 TEN 作为进程或线程运行](https://ten-framework-assets.s3.amazonaws.com/doc-assets/run_ten_as_process_or_thread.png)

![应用程序运行的不同模式](https://ten-framework-assets.s3.amazonaws.com/doc-assets/different_mode_of_app_runnings.png)

## TEN 应用程序文件夹结构

在 TEN framework 中，开发人员可以使用各种编程语言创建 TEN 软件包。无论使用哪种语言，TEN 应用程序的整体文件夹结构都保持一致，确保不同实现之间的一致性。

![TEN 软件包文件夹结构](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_package_folder_structure.png)


# 扩展 (/cn/docs/ten_framework/0.11.59/software_packages/extension)

## 生命周期

扩展的生命周期分为以下阶段：

1. `on_configure`
2. `on_init`
3. `on_start`
4. `on_stop`
5. `on_deinit`

在所有这些阶段，扩展都可以发送消息，如果发送的消息是命令类型，则扩展还可以接收其发送的命令的结果。换句话说，不存在扩展**不能**发送消息或接收其自身发送的命令结果的阶段。因此，在所有这些阶段，扩展都可以向其他扩展发送消息，这有助于在所有阶段维持扩展之间的依赖关系。

每个生命周期阶段都对应一个回调函数，并且有一个相应的 `on_xxx_done()` 函数来标记该生命周期阶段的结束。

| 生命周期回调         | 生命周期结束函数            |
| -------------- | ------------------- |
| `on_configure` | `on_configure_done` |
| `on_init`      | `on_init_done`      |
| `on_start`     | `on_start_done`     |
| `on_stop`      | `on_stop_done`      |
| `on_deinit`    | `on_deinit_done`    |

### 阶段转换图

```mermaid
graph TD;
  on_configure((on_configure));
  on_init((on_init));
  on_start((on_start));
  on_stop((on_stop));
  on_deinit((on_deinit));

  on_configure-->on_configure_done
  on_configure_done-->on_init;
  on_configure-->on_stop;
  on_init-->on_init_done;
  on_init_done-->on_start;
  on_init-->on_stop;
  on_start-->on_stop;
  on_start-->on_start_done;
  on_stop-->on_stop_done;
  on_stop_done-->on_deinit;
  on_deinit-->on_deinit_done;
```

### on\_configure

用于设置扩展属性的初始值。这允许在其他生命周期阶段中使用 `ten_env` 的 `get_property` API 来检索在 `on_configure()` 阶段设置的属性。调用 `on_configure_done()` 函数以标记 `on_configure` 阶段的结束。

```c++
  void on_configure(ten::ten_env_t &ten_env) override {
    // 设置扩展属性的初始值。
    ten_env.init_property_from_json(
      R"({
           "ten": {
             "uri": "msgpack://127.0.0.1:8001/",
             "log_level": 2
           }
         })", nullptr);
    ten_env.on_configure_done();
  }
```

### on\_init

用于初始化扩展。由于在 `on_init_done()` 之前扩展尚未准备就绪，TEN runtime 会令发送到扩展的所有消息排队，直到扩展准备就绪。准备就绪后，会将排队的消息传输给扩展。重要的是，如果结果来自扩展自身发送的命令，则*不受*此限制；TEN runtime 会将命令的结果直接传递回扩展。这是因为发送命令是扩展的自我启动操作，因此它必须准备好接收并处理其自身命令的结果。

### on\_start

此阶段标志着运行时开始向扩展发送消息。`on_start_done` 函数没有特殊用途，但为了与其他生命周期阶段保持一致并降低学习复杂性，TEN framework 包含了它。通常，`on_start_done` 在 `on_start` 回调函数的末尾调用。TEN runtime 将在收到其 `on_start_done()` 后开始将来自其他扩展的消息传递到此扩展。

```c++
  void on_start(ten::ten_env_t &ten_env) override {
    // 一些操作。
    ten_env.on_start_done();
  }
```

### on\_stop

在许多情况下需要停止扩展，例如当包含扩展的应用程序或引擎终止时。当扩展即将停止时，TEN runtime 会使用此 `on_stop` 回调通知扩展它已进入 `on_stop` 生命周期阶段。然后，扩展可以执行任何必要的终止操作。

### on\_deinit

在扩展调用 `on_stop_done()` 后，它将进入 `on_deinit` 阶段。在此阶段，由于扩展内的资源可能不再完全可用，因此 TEN runtime 不会将来自其他扩展的任何消息传递到此扩展。

## 不同生命周期阶段的扩展之间的关系

基本上，没有固有的关系。每个扩展都独立运行，在其自身的生命周期阶段之间切换。扩展是彼此独立的，它们之间的任何依赖关系都必须由扩展本身显式实现。TEN runtime 不进行任何假设或保证。例如，如果扩展 A 需要等待扩展 B 完成其初始化才能完成自身的初始化，则扩展 A 可以在其 `on_init` 期间向扩展 B 发送命令。一旦扩展 B 完成初始化并收到命令，它可以回复结果，当扩展 A 收到结果时，它可以调用 `on_init_done`。

```mermaid
sequenceDiagram
  participant A as 扩展 A
  participant B as 扩展 B

  Note over A,B: on_init
  activate A
  activate B
  deactivate B
  Note over B: on_init_done

  Note over B: on_start
  activate B
  deactivate B
  Note over B: on_start_done

  A->>B: cmd
  B-->>A: result

  deactivate A
  Note over A: on_init_done
```

基本上，在所有其他生命周期阶段，如果想要实现不同扩展的顺序，可以使用相同的方法来实现。

## 与 TEN runtime 的接口

扩展主要通过三个接口与 TEN runtime 交互：

1. **生命周期回调**

   * 这些包括像 `on_init`、`on_deinit`、`on_start` 和 `on_stop` 这样的回调。

2. **接收消息的回调**

   * 这些包括诸如 `on_cmd`、`on_data`、`on_audio_frame` 和 `on_video_frame` 之类的回调，用于处理传入扩展的消息。

3. **发送消息的函数**

   * 这些包括诸如 `send_cmd`、`send_data`、`send_audio_frame` 和 `send_video_frame` 之类的函数，用于将消息发送出扩展。

## 生命周期回调函数

扩展生命周期的不同阶段及其与消息处理的联系如下：

* **on\_init \~ on\_init\_done**: 处理扩展自身的初始化。在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。
* **on\_start \~ on\_start\_done**: 在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。由于属性在 `on_configure` 期间初始化，因此您可以执行依赖于这些属性设置的操作。但是，由于这仍然是初始化阶段的一部分，扩展将不会收到其他扩展主动发起的消息，避免了各种检查的需要。允许主动发送消息。
* **After on\_start\_done \~ on\_stop\_done**: 在此阶段，扩展可以正常发送和接收所有类型的消息及其结果。
* **on\_deinit \~ on\_deinit\_done**: 处理扩展的反初始化。与 `on_init` 阶段类似，在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。

## 使用不同语言实现扩展

在 TEN framework 中，可以使用各种语言（例如 C++、Go 和 Python）来实现扩展。开发者可以使用相同的概念方法以不同的语言实现扩展。学习如何使用一种语言开发扩展后，相对容易使用其他语言进行开发。

## 扩展中的异步消息处理

<figure><img src="../../..https://ten-framework-assets.s3.amazonaws.com/doc-assets/asynchronous_message_processing.png" alt=""><figcaption><p>异步消息处理</p></figcaption></figure>

扩展以异步方式处理消息。当 TEN runtime 通过 `on_cmd`、`on_data`、`on_audio_frame` 或 `on_video_frame` 等回调函数将消息传递给扩展时，并不要求扩展立即在回调函数内处理该消息。相反，扩展可以将消息委托给其他线程、进程，甚至其他机器进行处理。这可以充分利用多核和分布式计算资源。

处理完成后，可以通过 `send_cmd`、`send_data`、`send_audio_frame` 或 `send_video_frame` 等回调函数将结果发送回 TEN runtime。整个过程是异步的，这意味着扩展无需在 `on_cmd`、`on_data`、`on_audio_frame` 或 `on_video_frame` 回调函数返回之前就发送处理后的结果。结果只有在真正准备就绪时才可以使用适当的发送函数进行传输。


# 概述 (/cn/docs/ten_framework/0.11.59/software_packages/overview)

一个基本的 TEN framework 软件包的目录结构如下所示。

```text
.
├── bin/
├── src/
├── manifest.json
├── manifest-lock.json
├── property.json
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    ├── protocol/
    │   └── <protocol_a>/
    └── system/
        ├── <system_package_1>/
        └── <system_package_2>/
```

根据编程语言的不同，可能还会存在其他必需的文件。例如，在 C++ 项目中，由于构建系统是 `ten_gn`，因此会存在一个 `BUILD.gn` 文件。在 Go 项目中，会存在诸如 `go.mod` 和 `go.sum` 之类的文件。在 Python 项目中，你可能会找到 `requirements.txt` 文件。

## TEN framework 软件包类型

| 类型                    | 描述              |
| --------------------- | --------------- |
| 应用 (App)              | 包含一个 TEN 应用。    |
| 扩展组 (Extension group) | 包含一个 TEN 扩展组。   |
| 扩展 (Extension)        | 包含一个 TEN 扩展。    |
| 协议 (Protocol)         | 包含一个 TEN 协议。    |
| 系统 (System)           | 包含一个 TEN 系统软件包。 |

## TEN framework 软件包种类

有两种类型的软件包：

* 发布包 (Release package)
* 开发包 (Development package)

这两种软件包之间的关系如下：

* 发布包可以通过构建过程从开发包生成。
* 如果我们将与源代码相关的内容添加到发布包中，它就会变成一个开发包。
* 如果我们从开发包中删除与源代码相关的内容，它就会变成一个发布包。

简而言之，开发包和发布包的目录结构非常相似，区别在于开发包包含与源代码相关的内容，而发布包则不包含。

### 开发包

开发包的主要目的是开发 TEN 软件包。例如，如果您想创建一个新的 TEN 扩展，您可以修改现有的开发包来实现此目的。这是因为开发包包含所有与源代码相关的内容，可以从中创建新的 TEN 软件包。

### 发布包

发布包的主要目的是创建和运行图。由于不需要构建任务，因此可以使用发布包来执行这些操作。但是，开发包也可以用于这些任务。本质上，开发包的应用范围比发布包更广泛。


# Python 异步扩展 (/cn/docs/ten_framework/0.11.59/software_packages/python_async_extension)

**Python 异步扩展**是一个使用 Python 的 `asyncio` 框架的模块化组件。如果您想将使用 `asyncio` 的现有 Python 代码包装到 TEN 扩展中，那么使用 Python 异步扩展是最简单、最方便的选择。

## 示例：默认 Python 异步扩展

以下是 Python 异步扩展的结构方式：

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    async def on_configure(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_json = cmd.to_json()
        ten_env.log_debug(f"DefaultAsyncExtension on_cmd: {cmd_json}")

        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

        # 向其他扩展发送新命令并等待结果。结果将返回给原始发送者。
        new_cmd = Cmd.create("hello")
        cmd_result = await ten_env.send_cmd(new_cmd)
        ten_env.return_result(cmd_result, cmd)
```

每种方法都使用 `await asyncio.sleep()` 模拟延迟。

## 常见问题

### 用于事件处理的异步循环

* 创建一个队列：使用 `asyncio.Queue`。
* 创建一个用于事件处理的异步任务。

这是示例代码：

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    queue = asyncio.Queue()
    loop:asyncio.AbstractEventLoop = None

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        self.loop = asyncio.get_event_loop()

        self.loop.create_task(self._consume())

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        self.queue.put(None)

    async def _consume(self) -> None:
        while True
            try:
                value = await self.queue.get()
                if value is None:
                    self.ten_env.log_info("async loop exit")
                    break

                # 用于处理从队列检索的值的代码。

            except Exception as e:
                self.ten_env.log_error(f"Failed to handle {e}")
```

## 结论

TEN 的 Python 异步扩展提供了一种以异步方式处理长时间运行任务的强大方法。通过集成 Python 的 `asyncio` 框架，扩展能够确保诸如网络调用或文件处理之类的操作是高效且非阻塞的。这使得 TEN 成为构建具有异步功能的可扩展模块化应用程序的绝佳选择。


# 标准接口 (/cn/docs/ten_framework/0.11.59/standard_extension/standard_interface)

在 TEN framework 中，我们鼓励开发者创建自己的扩展（Extension）并将其发布到 TEN 扩展市场，供其他开发者使用。为了让开发者能够轻松地选择和替换扩展，我们需要定义标准化的接口规范，这样开发者就可以根据这些标准接口快速找到符合需求的扩展。

简而言之，标准接口是一种语法糖，它让 TEN 工具（如 TEN Manager）能够更方便地协助开发者进行扩展的选择和替换等操作。

TEN 的接口定义类似于 TypeScript 的 `.d.ts` 文件，让 TypeScript 编译器能够进行类型检查，但在实际运行时，这些接口定义并不会被使用。因此，标准接口设计带来的功能和代码增强，主要体现在 TEN Manager 这类开发工具上，而不是运行时的 TEN Runtime 上。例如，TEN Manager 需要提供相关的 RESTful API 给 tman designer Frontend，让用户可以根据标准接口定义，通过 UI 界面进行扩展的选择和替换。

## 什么是标准接口

在 TEN framework 中，每个扩展的接口都包含以下几个核心组成部分：

1. **Properties（属性）** - 扩展的配置参数
2. **Command（命令）**
   * **Input command** - 输入命令
   * **Output command** - 输出命令
3. **Data（数据）**
   * **Input data** - 输入数据
   * **Output data** - 输出数据
4. **Audio frame（音频帧）**
   * **Input audio frame** - 输入音频帧
   * **Output audio frame** - 输出音频帧
5. **Video frame（视频帧）**
   * **Input video frame** - 输入视频帧
   * **Output video frame** - 输出视频帧

### 标准接口的意义和用途

标准接口就是将这些组成部分进行规范化定义，形成统一的接口标准。开发者可以参考这些接口标准来实现自己的扩展，同时扩展也可以声明自己支持特定的标准接口，这样开发工具就能够根据扩展声明的标准接口提供更好的开发支持，使用户能够根据标准接口来选择或替换所需的扩展。

在标准接口中，Command/Data/Audio frame/Video frame 用于规范扩展之间的交互，实现扩展的互换性和兼容性。而 Properties 则用于规范扩展的配置参数，实现与 tman designer 的配置兼容性。

需要注意的是，规范扩展之间的交互并不是接口本身的功能，这是 Command/Data/Audio frame/Video frame 的功能。接口只是一个用于聚合的语法糖，帮助开发者快速指定扩展的接口定义。

### 接口的检查与报错

在扩展的 Command/Data/Audio frame/Video frame 定义中，如果存在两个同名的消息，且它们的 schema 定义相同，这不算错误，但会被视为重复定义并被忽略。这个规则在标准接口中同样适用，目的是让不同的标准接口可以各自定义共性功能，形成自包含的标准接口定义。当一个扩展实现这些具有相同定义的标准接口时，不会因此产生错误。这个特性在标准接口中非常有用，因为标准接口的定义可以被引用，所以当一个扩展实现多个标准接口时，如果这些标准接口中（或其引用的其他标准接口中）有相同的 message 和相同的 schema，也不会因此产生错误。

由于 TEN framework 采用运行与开发分离的架构（类似于 JavaScript 与 TypeScript 的 .d.ts 文件的关系），标准接口的检查与报错主要发生在开发工具阶段，而不是运行时的 TEN Runtime 上。因此，TEN Runtime 在启动图（start graph）时，不会因为接口错误而报错。

在 tman designer 这个开发工具中，如果扩展的接口定义不完整，会显示某些功能暂时无法在该扩展上使用的提示。例如，在使用某个扩展替代另一个扩展的功能时，如果接口定义不完整，会提示该功能暂时无法在这个扩展上使用，需要等待接口定义完整后才能使用。

另外，由于接口定义可以是一个被引用的独立文件，因此在 TEN Manager 的 publish/install 操作中，接口定义不完整并不算错误。

### 接口的兼容规则

当要将实现接口 A 的扩展替换为实现接口 B 的扩展时，需要接口 A 兼容于接口 B 才能进行替换。接口之间的兼容规则有两种模式：

**严格模式：**

* 接口 A 的所有消息都必须出现在接口 B 中，且接口 A 中消息的 schema 必须兼容于接口 B 中对应消息的 schema。

**宽松模式：**

* 只有存在连线（即有 connection）的接口 A 中的消息必须出现在接口 B 中，且接口 A 中这些消息的 schema 必须兼容于接口 B 中同名消息的 schema。

严格模式和宽松模式供用户选择，适用于不同的场景。例如，如果确定扩展实现的接口 A 中的某个消息 X 一定不会被使用，那么因为消息 X 的不兼容或不存在而导致无法替换为想要的另一个扩展可能过于严格。

## 标准接口的实现方式

TEN 扩展支持的 API 定义在其 `manifest.json` 文件中。标准接口的实现就是将定义好的 API 规范写入独立的 JSON 文件，然后在 `manifest.json` 中引用这个文件。

```json
{
  "type": "extension",
  "name": "foo_extension",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.1"
    }
  ],
  "api": {
    "interface": [
      {
        "import_uri": "https://xxx/asr_interface.json"  // <= 引用标准接口定义文件
      },
      {
        "import_uri": "https://xxx/tts_interface.json"  // <= 引用标准接口定义文件
      }
    ],
    "property": {
      "bar": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "query_vector",
        "property": {
          "collection_name": {
            "type": "string"
          },
          "top_k": {
            "type": "int64"
          },
          "embedding": {
            "type": "array",
            "items": {
              "type": "float64"
            }
          }
        },
        "required": [
          "collection_name",
          "top_k",
          "embedding"
        ],
        "result": {
          "property": {
            "response": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "content": {
                    "type": "string"
                  },
                  "score": {
                    "type": "float64"
                  }
                }
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "publish",
        "property": {
          "message": {
            "type": "buf"
          }
        }
      },
      {
        "name": "set_presence_state"
      }
    ],
    "data_in": [
      {
        "name": "data"
      }
    ]
  }
}
```

### 标准接口定义文件示例

当 TEN 系统（包括 TEN Runtime、TEN Manager 等）解析 `manifest.json` 时，会根据 `import_uri` 下载对应的 JSON 文件，并将其内容合并到 `manifest.json` 中。被引用的标准接口文件内容示例如下，可以看到内容与 manifest.json 中的 `api` 字段相同，只是被独立成一个文件以便引用。并且这个独立的标准接口文件也可以再引用其他的标准接口文件。

```json
{
  "interface": [
    {
      "import_uri": "https://xxx/stt_interface.json"// <= 引用其他的标准接口定义文件
    }
  ],
  "property": {
    "bar": {
      "type": "string"
    }
  },
  "cmd_in": [
    {
      "name": "query_vector",
      "property": {
        "collection_name": {
          "type": "string"
        },
        "top_k": {
          "type": "int64"
        },
        "embedding": {
          "type": "array",
          "items": {
            "type": "float64"
          }
        }
      },
      "required": [
        "collection_name",
        "top_k",
        "embedding"
      ],
      "result": {
        "property": {
          "response": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string"
                },
                "score": {
                  "type": "float64"
                }
              }
            }
          }
        }
      }
    }
  ],
  "cmd_out": [
    {
      "name": "publish",
      "property": {
        "message": {
          "type": "buf"
        }
      }
    },
    {
      "name": "set_presence_state"
    }
  ],
  "data_in": [
    {
      "name": "data"
    }
  ]
}
```

### 接口合并规则

系统会将 `import_uri` 指向的 JSON 文件内容合并到 `manifest.json` 的 `api` 字段中，具体包括：

* `property`
* `cmd_in`、`cmd_out`
* `data_in`、`data_out`
* `audio_frame_in`、`audio_frame_out`
* `video_frame_in`、`video_frame_out`

**重要规则：**

1. **冲突检测**：如果合并过程中发现同名的同类字段（比如同名的 `cmd_in` 或同名的 `property`），如果 schema 相同则不算错误，否则系统会报错。
2. **递归合并**：合并过程是递归进行的，如果检测到循环引用，系统会报错。
3. **多接口支持**：一个扩展可以支持多个标准接口，在 `manifest.json` 中定义多个 `interface` 字段即可。
4. **重复引用检测**：如果不同的 `interface` 字段引用了相同的 `import_uri`，系统会报错。

## 标准接口的存储方式

标准接口文件支持以下几种存储方式：

<div className="fd-steps">
  <div className="fd-step">
    ### TEN 系统包（TEN System Package）

    **概述**：这种方式是将标准接口定义文件打包成 TEN 系统包，然后在 `manifest.json` 中引用。同时也可以在 manifest.json 的 dependencies 字段中依赖这个系统包，这样在安装扩展时就会一并安装所依赖的标准接口系统包，从而提供最佳的开发体验。通过这种方法，可以利用 TEN 包的版本机制来实现标准接口的版本管理，并且可以利用 TEN 包的依赖机制来实现标准接口的自动安装。

    使用 TEN 系统包，除了标准接口的 JSON 定义文件外，还可以包含相关的资源，例如模型文件、公共工具函数库、配套工具等。因此，当所有这些资源出现不兼容的变更时，都可以通过 TEN 系统包的版本号变更来更好地体现这种变化。例如，系统包内 JSON 文件的存放位置或者文件名称的变化，也算是该系统包的破坏性变更，因此应该升级该系统包的主版本号。

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "0.1.0"
        }
      ],
      "api": {
        "interface": [
          {
            "import_uri": "../../../ten_packages/system/foo_interface/interface.json"
          }
        ]
      }
    }
    ```

    **使用 TEN 系统包的优势：**

    1. **易于分发**：系统包可以上传到 TEN 扩展市场，用户可以直接下载使用
    2. **资源丰富**：包中不仅包含接口定义文件，还可以包含相关资源，如：
       * 模型文件
       * 公共工具函数库
       * 配套工具等
    3. **依赖管理**：系统包可以作为扩展的依赖项，当扩展被下载时，相关的系统包也会自动下载
    4. **版本管理**：不仅限于标准接口协议本身，该系统包内的任何资源都是该系统包版本管理的一部分

    需要注意的是，TEN 系统包的版本号并不仅仅代表标准接口的版本号，而是代表整个系统包的版本号。因此，如果系统包内的资源有破坏性变更，就应该升级系统包的主版本号。例如，系统包内 JSON 文件的存放位置或者文件名称的变化，即使没有改变 JSON 文件的内容，也算是该系统包的破坏性变更，因为对使用者来说，使用方式需要改变，所以应该升级该系统包的主版本号。

    因此，不能仅仅通过系统包的版本号来判断标准接口的兼容性。标准接口的兼容性还是需要通过接口的兼容规则来判断。

    #### 支持多版本共存

    > **注意**：这一节所描述的功能，基本上不会实现，只是描述如果之后要做的话，可能的实现方法以及需要考虑的点。

    **概述**：在某些场景下，一个 TEN 应用可能需要同时使用同一标准接口的不同版本。为了支持这种需求，TEN 允许在同一个应用内安装同一个系统包的多个版本。

    **系统包配置：**

    首先，在系统包的 `manifest.json` 中声明支持多版本共存：

    ```json
    {
      "type": "system",
      "name": "foo_interface",
      "version": "0.1.0",
      "install_policy": {
        "coinstallable": "true"
      }
    }
    ```

    **扩展依赖配置：**

    在依赖该系统包的扩展的 `manifest.json` 中，也需要声明使用多版本共存：

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "=0.1.0", // 注意：这里只能是 equal 的语法，不能用 >= 之类的语法
          "install_policy": {
            "coinstallable": "true"
          }
        }
      ],
    }
    ```

    并且在整个依赖树的构建过程中，如果一个系统包的 coinstallable 为 true，则需要所有依赖该系统包的地方都声明是 coinstallable，否则系统会报错。

    **文件系统路径：**

    启用多版本共存后，系统包在文件系统中的路径会变为：

    ```text
    ten_packages/system/<hash_value>_foo_extension_0_1_0/
    ```

    > **注意**：未来可能会引入映射机制，让用户在使用时不需要明确指定 hash\_value，而是通过某种映射机制自动找到对应的系统包。但这种机制如果要好用，还需要修改 TEN 所支持的语言中的模块加载机制，使得开发者可以不用在该语言中的模块加载机制中指定 hash\_value，而是通过某种映射机制自动找到对应的系统包。这种深入语言内部的改动也是这种单包多版本共存的机制基本上不会被实现的原因之一。
    >
    > **注意**：如果真的需要在一个 TEN 应用内使用多个版本的一个标准接口，可以使用下述的几种方式来实现。如果一个标准接口有配套的公用资源，例如模型文件、公共工具函数库、配套工具等，那么这些资源很多无法做到在一个 TEN 应用内的多版本共存，这也是这种单包多版本共存的机制基本上不会被实现的原因之一。
  </div>

  <div className="fd-step">
    ### 网络地址

    **概述**：将标准接口定义文件部署到网络服务器上，通过 HTTP/HTTPS 地址引用：

    ```json
    {
      "interface": [
        {
          "import_uri": "http://xxx/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    > **注意**：TEN Manager 在处理这种引用方式时，可以将文件下载到自己的缓存中，然后通过某种映射机制，让 TEN Manager 能够找到定义。

    **适用场景：**

    * **轻量级接口**：没有配套的公共资源的接口定义
    * **版本灵活性**：需要在一个 `manifest.json` 中引用同一标准接口的不同版本
  </div>

  <div className="fd-step">
    ### 本地路径

    **概述**：引用本地文件系统中的标准接口定义文件：

    **绝对路径：**

    ```json
    {
      "interface": [
        {
          "import_uri": "file://c:/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **相对路径：**

    ```json
    {
      "interface": [
        {
          "import_uri": "../asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **适用场景：**

    与网络地址方式类似：

    * **轻量级接口**：适合没有配套的公共资源的接口定义
    * **版本灵活性**：需要在一个 `manifest.json` 中引用同一标准接口的不同版本
  </div>
</div>

## 如何从云商店中筛选想要的扩展

标准接口与云商店的关系体现在将标准接口包装成 TEN 系统包。有以下两种方式可以搜索出符合特定条件的扩展：

1. **依赖搜索**：搜索依赖某个系统包的扩展。由于云商店中扩展的元数据包含了依赖信息，所以云商店可以据此筛选出依赖某个系统包的所有扩展。

2. **标签搜索**：通过 TEN package 的 tag 来快速搜索出符合某些条件的扩展，选择安装后，再进行替换操作。如果是 TEN 官方认证过的扩展，甚至可以打上 `ten:` 开头的标签来表明这是一个官方认证过的扩展，确实符合所声明的标准接口定义。


# 图检查 (/cn/docs/ten_framework/0.11.59/ten_manager/check_graph)

`tman` 提供了 `check graph` 命令，用于验证预定义图或 `start_graph` 命令的正确性。要查看使用细节，请使用以下命令：

```shell
$ tman check graph -h
检查预定义图或 start_graph 命令的图内容是否正确。有关更详细的用法，请运行“graph -h”

用法：tman check graph [选项] --app <APP>

选项：
      --app <APP>
          图中定义的应用程序的绝对路径。默认情况下，将从列表中的第一个应用程序读取预定义图。
      --predefined-graph-name <PREDEFINED_GRAPH_NAME>
          指定要检查的预定义图的名称，否则将检查所有预定义图。
      --graph <GRAPH>
          指定要检查的“start_graph”命令的 JSON 字符串。如果未指定，则将检查第一个应用程序中的预定义图。
  -h, --help
          打印帮助信息
```

`check graph` 命令旨在处理可能跨越多个应用程序的图，因此不需要从特定应用程序的根目录运行。相反，它可以从任何目录执行，并使用 `--app` 参数指定图中涉及的应用程序的文件夹。

## 使用示例

* **检查 `property.json` 中的所有预定义图：**

  ```shell
  tman check graph --app /home/TEN-Agent/agents
  ```

* **检查特定的预定义图：**

  ```shell
  tman check graph --predefined-graph-name va.openai.azure --app /home/TEN-Agent/agents
  ```

* **检查 `start_graph` 命令：**

  ```shell
  tman check graph --graph '{
    "type": "start_graph",
    "seq_id": "55",
    "nodes": [
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_2__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      },
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_1__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      }
    ]
  }' --app /home/TEN-Agent/agents
  ```

## 前提条件

* **预定义图定义要求：** 如果指定了预定义图的名称，则将从第一个 `--app` 参数指定的应用程序的 `property.json` 文件中提取其定义。
* **包安装要求：** 在运行 `check graph` 命令之前，必须使用 `tman install` 命令安装应用程序依赖的所有扩展。这是必需的，因为验证过程需要有关图中每个扩展的信息，例如其 `manifest.json` 文件中定义的 API。
* **唯一应用程序 URI 要求：** 在多应用程序图中，每个应用程序的 `property.json` 必须定义唯一的 `ten::uri`。此外，`uri` 值不能设置为 `"localhost"`。

## 验证规则

<div className="fd-steps">
  <div className="fd-step">
    ### 节点 (Nodes) 的存在性

    任何图定义中都必须存在 `nodes` 数组。如果缺少该数组，将会抛出错误。

    * **示例（未定义节点）：**

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": []
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        No extension node is defined in graph.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 节点的唯一性

    `nodes` 数组中的每个节点表示一个应用程序组中由指定插件创建的特定扩展实例。因此，每个扩展实例都应该由一个单独的节点唯一表示。必须通过 `app`、`extension_group` 和 `name` 的组合来唯一标识一个节点。不允许出现同一扩展实例的多个条目。

    * **示例（重复节点）：**

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": [
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_2__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          },
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_1__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        Duplicated extension was found in nodes[1], addon: basic_hello_world_1__test_extension, name: test_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在通信链路中使用的扩展应在 nodes 中定义

    `connections` 字段中引用的所有扩展实例（无论是作为源还是目标）都必须在 `nodes` 字段中显式定义。任何未在 `nodes` 数组中定义的实例都会导致验证错误。

    * **示例（未定义源扩展）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0] is not defined in nodes, extension_group: producer, extension: some_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

    * **示例（未定义目标扩展）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                },
                {
                  "type": "extension",
                  "name": "some_extension_1",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        }
                      ]
                    },
                    {
                      "name": "world",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        },
                        {
                          "extension": "consumer"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0].cmd[1] is not defined in nodes extension_group: some_group, extension: consumer.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 `nodes` 中声明的插件必须安装在应用中

    * **示例（`property.json` 中的 `ten::uri` 与 `nodes` 中的 `app` 字段不相等）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ],
          "uri": "http://localhost:8001"
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

      问题在于，应用程序中的所有包都将存储在以应用程序的 `uri` 为键的映射中，并且图中的每个节点都将通过 `app` 字段（默认为 `localhost`）检索。节点中的 `app`（即 `localhost`）与应用程序的 `uri`（即 `http://localhost:8001`）不匹配。

    * **示例（由于未执行 `tman install` 而导致 `ten_packages` 目录不存在）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ]
        }
      }
      ```

      并且 `ten_packages` 目录**不存在**。

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 通信链路中，从一个扩展发送的消息应在同一部分中定义

    * **示例：**

      假设所有包都已安装。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          },
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello_2",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        extension 'some_extension' is defined in connection[0] and connection[1], merge them into one section.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 通信链路中，从一个扩展发出的消息在每个类型中应具有唯一的名称

    * **示例：**

      假设所有包都已安装。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              },
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "some_extension"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        - connection[0]:
         - Merge the following cmd into one section:
            'hello' is defined in flow[0] and flow[1].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 通信链路中声明的消息应兼容

    根据扩展的 `manifest.json` 文件中的模式定义，将检查在 `connections` 中的每个消息流中声明的消息的模式是否兼容。消息兼容的规则如下。

    * 如果在源扩展和目标扩展中都找不到消息模式，则消息是兼容的。

    * 如果仅在其中一个扩展中找到消息模式，则该消息不兼容。

    * 只有满足以下条件，消息才是兼容的：

      * 属性类型兼容。
        * 如果属性是一个对象，则源模式和目标模式中的字段都必须具有兼容的类型。
        * 如果在目标模式中定义了 `required` 关键字，则源模式中必须存在一个 `required` 关键字，该关键字是目标中 `required` 的超集。

    * **示例：**

      假设所有包都已安装。

      `addon_a` 中的 `manifest.json` 的内容如下。

      ```json
      {
        "type": "extension",
        "name": "addon_a",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_out": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "string"
                }
              }
            }
          ]
        }
      }
      ```

      并且，`addon_b` 中的 `manifest.json` 的内容如下。

      ```json
      {
        "type": "extension",
        "name": "addon_b",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_in": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "int8"
                }
              }
            }
          ]
        }
      }
      ```

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        - connections[0]:
          - cmd[0]:  Schema incompatible to [extension_group: some_group, extension: another_ext], { .foo: type is incompatible, source is [string], but target is [int8] }

      All is done.
      ❌ Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 节点中的 `app` 必须明确

    每个节点中的 `app` 字段必须满足以下规则。

    * `app` 字段必须等于相应 TEN 应用程序的 `ten::uri`。

    * 要么所有节点都应该声明 `app`，要么都不应该声明。

    * `app` 字段不能为 `localhost`。

    * `app` 字段不能为空字符串。

    * **示例（某些节点指定了 `app` 字段）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        Either all nodes should have 'app' declared, or none should, but not a mix of both.
      ```

    * **示例（所有节点均未指定 `app`，但某些源扩展在通信链路中指定了 `app`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
         {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].the 'app' should not be declared, as not any node has declared it
      ```

    * **示例（所有节点均未指定 `app`，但某些目标扩展在通信链路中指定了 `app`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext",
                    "app": "http://localhost:8000"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].cmd[0].dest[0]: the 'app' should not be declared, as not any node has declared it
      ```

    * **示例 (节点中的 `app` 字段与应用程序的 `ten::uri` 不相等)：**

      与 [规则 4](#id-4.-the-addons-declared-in-the-nodes-must-be-installed-in-the-app) 相同。

    * **示例（单应用图中 `app` 字段为 `localhost`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, and the graph seems to be a single-app graph, just remove the 'app' field
      ```

    * **示例（多应用图中 `app` 字段为 `localhost`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, change the content of 'app' field to be consistent with 'ten::uri'
      ```
  </div>
</div>


# 概述 (/cn/docs/ten_framework/0.11.59/ten_manager/overview)

在 TEN 生态系统中，几乎所有事物都是组件。例如，一个 TEN 应用是一种组件，一个 TEN 扩展是另一种组件，而一个 TEN 协议又是另一种组件。为了帮助 TEN 开发者管理与这些组件相关的所有操作，我们提供了一个至关重要的工具，称为 TEN 管理器。

<Callout type="info">
  `tman`

   是 TEN 管理器 (TEN Manager) 的缩写。
</Callout>

TEN 管理器包含一个 TEN 软件包管理器，旨在简化 TEN 软件包的管理。此软件包管理器处理诸如确定哪些依赖包是必需的、获取这些依赖包以及安装它们等任务。简而言之，TEN 管理器简化了从头开始创建新的 TEN 软件包的过程，并使从 TEN 云商店安装 TEN 软件包更加容易。


# 调试 (/cn/docs/ten_framework/0.11.59/testing_and_debugging/debugging)

## 适用于 TEN framework 的开发人员

如果您使用 VS Code 开发 TEN framework，则 TEN framework 源代码树中有一个 `launch.json` 文件，其中提供了默认的调试目标。

## 适用于 TEN framework 的用户

如果您使用 VS Code 开发基于 TEN framework 的自己的应用程序，您可以将配置添加到您的 `.vscode/launch.json` 文件，以便您可以使用断点和变量检查来调试您的应用程序，无论您使用的是哪种编程语言（C++/Golang/Python）。

### 在 C++ 应用程序中调试

如果应用程序是用 C++ 编写的，这意味着扩展可以使用 C++ 或 Python 编写。

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

* **使用 lldb 调试 C++ 代码**

  ```json
  {
    "name": "app (C++) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    }
  }
  ```

* **使用 gdb 调试 C++ 代码**

  ```json
  {
    "name": "app (C++) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### 使用 debugpy 调试 Python 代码

您可以使用 `debugpy` 调试 Python 代码。 但是，由于程序不是直接通过 Python 解释器启动的，因此 Python 代码由嵌入的 Python 解释器执行。 因此，您只能将调试器附加到正在运行的进程。

首先，您需要在 Python 环境中安装 `debugpy`：

```shell
pip install debugpy
```

然后，您需要使用设置的环境变量 `TEN_ENABLE_PYTHON_DEBUG` 和 `TEN_PYTHON_DEBUG_PORT` 启动应用程序。 例如：

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

如果环境变量 `TEN_ENABLE_PYTHON_DEBUG` 设置为 `true`，则应用程序将阻塞，直到附加调试器。 如果设置了环境变量 `TEN_PYTHON_DEBUG_PORT`，则调试器服务器将侦听指定端口上的传入连接。

然后，您可以将调试器附加到正在运行的进程。 以下是一个配置示例：

**使用 debugpy 调试 Python 代码**

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### 同时调试 C++ 和 Python 代码

如果您想在 VSCode 中一键启动同时调试 C++ 和 Python 代码，您可以执行以下操作：

1. 定义一个任务，在附加调试器之前延迟几秒钟：

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. 将以下配置添加到 `launch.json` 文件：

   ```json
   "configurations": [
        {
            "name": "app (C++) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch", // "launch" or "attach"
            "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
            "cwd": "${workspaceFolder}/",
            "env": {
                "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
                "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (C++) (debugpy, attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "justMyCode": false,
            "preLaunchTask": "delay 3 seconds"
        }
   ],
   "compounds": [
        {
            "name": "app (C++) (lldb, launch) and app (C++) (debugpy, attach)",
            "configurations": [
                "app (C++) (lldb, launch with debugpy)",
                "app (C++) (debugpy, attach with delay)"
            ]
        }
   ]
   ```

然后，您可以使用复合配置 `app (C++) (lldb, launch) and app (C++) (debugpy, attach)` 启动调试 C++ 和 Python 代码。

### 在 Go 应用程序中调试

如果应用程序是用 Go 编写的，这意味着扩展可以使用 Go 或 C++ 或 Python 编写。

#### 使用 delve 调试 Go 代码

您可以使用以下配置在您的 Go 扩展中进行调试：

```json
{
  "name": "app (golang) (go, launch)",
  "type": "go",
  "request": "launch",
  "mode": "exec",
  "cwd": "${workspaceFolder}/",
  "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
  "env": {
    "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
    "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
    "TEN_APP_BASE_DIR": "${workspaceFolder}"
  }
}
```

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

* **使用 lldb 调试 C++ 代码**

  ```json
  {
    "name": "app (Go) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    },
    "initCommands": ["process handle SIGURG --stop false --pass true"]
  }
  ```

* **使用 gdb 调试 C++ 代码**

  ```json
  {
    "name": "app (Go) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### 使用 debugpy 调试 Python 代码

请参阅上面的部分，了解如何使用 debugpy 调试 Python 代码。 使用设置的环境变量 `TEN_ENABLE_PYTHON_DEBUG` 和 `TEN_PYTHON_DEBUG_PORT` 启动应用程序。

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

然后，您可以将调试器附加到正在运行的进程。 以下是一个配置示例：

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### 同时调试 C++、Go 和 Python 代码

如果您想在 VSCode 中一键启动同时调试 C++、Go 和 Python 代码，您可以执行以下操作：

1. 定义一个任务，在附加调试器之前延迟几秒钟：

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. 将以下配置添加到 `launch.json` 文件：

   ```json
   "configurations": [
        {
            "name": "app (golang) (debugpy, remote attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "delay 3 seconds",
            "justMyCode": false
        },
        {
            "name": "app (golang) (go, launch with debugpy)",
            "type": "go",
            "request": "launch",
            "mode": "exec",
            "cwd": "${workspaceFolder}/",
            "program": "${workspaceFolder}/bin/worker",
            "env": {
                "TEN_APP_BASE_DIR": "${workspaceFolder}",
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (golang) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/bin/worker",
            "cwd": "${workspaceFolder}/",
            "env": {
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            },
            "initCommands": [
                "process handle SIGURG --stop false --pass true"
            ]
        },
   ],
   "compounds": [
        {
            "name": "Mixed Go/Python",
            "configurations": [
                "app (golang) (go, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ],
        },
        {
            "name": "Mixed Go/Python/C++ (lldb)",
            "configurations": [
                "app (golang) (lldb, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ]
        }
   ]
   ```

如果您想同时调试 C++ 和 Python 代码，您可以使用复合配置 `Mixed Go/Python/C++ (lldb)`。 使用此配置，您可以同时对 C++、Go 和 Python 代码执行断点调试。 但是，变量检查仅支持 C++ 和 Python 代码，而不支持 Go 代码。

如果您想同时调试 Go 和 Python 代码，您可以使用复合配置 `Mixed Go/Python`。 使用此配置，您可以同时对 Go 和 Python 代码执行断点调试，并检查 Go 和 Python 代码的变量。

### 在 Python 应用程序中调试

如果应用程序是用 Python 编写的，这意味着扩展可以使用 Python 或 C++ 编写。

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

```json
{
  "name": "app (Python) (cpp, launch)",
  "type": "cppdbg", //
  "request": "launch", // "launch" or "attach"
  "program": "/usr/bin/python3", // Python 解释器路径
  "args": ["main.py"],
  "cwd": "${workspaceFolder}/",
  "environment": [
    {
      "name": "PYTHONPATH",
      "value": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface"
    },
    {
      "name": "TEN_APP_BASE_DIR",
      "value": "${workspaceFolder}/"
    }
  ],
  "MIMode": "gdb" // "gdb" 或 "lldb"
}
```

#### 使用 debugpy 调试 Python 代码

请参阅上面的部分，了解如何使用 debugpy 调试 Python 代码。 您可以使用以下配置来调试 Python 代码：

```json
{
  "name": "app (Python) (debugpy, launch)",
  "type": "debugpy",
  "python": "/usr/bin/python3",
  "request": "launch",
  "program": "${workspaceFolder}/main.py",
  "console": "integratedTerminal",
  "cwd": "${workspaceFolder}/",
  "env": {
    "PYTHONPATH": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface",
    "TEN_APP_BASE_DIR": "${workspaceFolder}/",
    "TEN_ENABLE_PYTHON_DEBUG": "true"
  }
}
```

#### 同时调试 C++ 和 Python 代码

如果您想同时启动 C++ 和 Python 代码的调试，您可以使用配置 `app (Python) (debugpy, launch)` 启动应用程序，并使用以下配置将调试器附加到正在运行的进程：

```json
{
  "name": "app (Python) (cpp, attach)",
  "type": "cppdbg",
  "request": "attach", // "launch" 或 "attach"
  "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
  "cwd": "${workspaceFolder}/",
  "MIMode": "gdb",
  "environment": [
    {
      "name": "LD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    },
    {
      "name": "DYLD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    }
  ]
}
```


# 面向 TEN framework 开发者 (/cn/docs/ten_framework/0.11.59/testing_and_debugging/for_developers_of_ten_framework)

TEN framework 包括三种类型的测试套件：

* 单元测试
* 冒烟测试
* 集成测试

TEN framework 使用 `gtest` 作为单元测试和冒烟测试的测试框架，使用 `pytest` 作为集成测试的测试框架。

## 单元测试

单元测试的源目录是 `tests/ten_runtime/unit`。如果您需要添加新的单元测试用例，请将它们放在 `tests/ten_runtime/unit directory` 中。

对于 C++ 项目，在成功编译后，您可以导航到输出目录（例如，`out/linux/x64/tests/standalone/`，具体取决于目标操作系统和 CPU）并执行以下命令：

```shell
./ten_runtime_unit_test
```

## 冒烟测试

冒烟测试的源目录是 `tests/ten_runtime/smoke`。如果您需要添加新的冒烟测试用例，请将它们放在 `tests/ten_runtime/smoke` 目录中。

导航到输出目录（例如，`out/linux/x64/tests/standalone`，具体取决于目标操作系统和 CPU）并执行以下命令：

```shell
./ten_runtime_smoke_test
```

## 循环测试

您可以使用以下命令执行多轮测试。

仅运行单元测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;
```

仅运行冒烟测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

同时运行单元测试和冒烟测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

## 集成测试

集成测试是 TEN framework 的黑盒测试。集成测试的源目录是 `tests/ten_runtime/integration`。这些测试用于验证实际的执行场景。

集成测试的文件结构如下：

```text
tests/ten_runtime/integration/
  ├── test_1/
  │    ├── test_case.py
  │    └── ...
  ├── test_2/
  │    ├── test_case.py
  │    └── ...
  └── ...
```

要执行集成测试，请使用以下命令：

```shell
cd path/to/out/
pytest tests/ten_runtime/integration/
```

## 测试技巧

### Linux

在 Linux 上，您可以使用以下技术来减慢 TEN 的执行速度，这有助于触发与时序相关的错误。

```shell
sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...
```

* `taskset 0x1`: 仅使用 1 个 CPU 核心来运行程序。
* `taskset 0x3`: 仅使用 2 个 CPU 核心来运行程序。


# 面向 TEN framework 用户 (/cn/docs/ten_framework/0.11.59/testing_and_debugging/for_users_of_ten_framework)

## 扩展的独立测试

TEN framework 提供了一种独立的扩展测试机制。该机制允许开发者在不依赖其他 TEN 概念（例如其他 TEN 扩展、TEN 图或 TEN 应用）的情况下测试扩展。当开发者需要在不运行整个 TEN 应用的情况下测试扩展的行为时，这种独立的扩展测试机制尤其有用。

TEN 扩展的独立测试框架遵循两个关键原则：

1. **与所用语言的任何原生测试框架兼容。**

   例如，如果使用 C++ 开发 TEN 扩展，则 Google gtest/gmock 测试框架可以与 TEN 扩展独立测试框架一起使用，以实现 C++ TEN 扩展的独立测试。

2. **用户无需修改被测扩展中的任何代码。**

   可以使用此独立测试框架对运行时使用的完全相同的扩展代码进行全面测试。

3. **以不同语言开发的扩展的独立测试流程遵循相同的设计原则和使用方法。**

   这确保一旦您学习了一种语言的独立测试概念和方法，就可以快速掌握其他语言的 TEN 扩展的独立测试概念和方法。

对于用户来说，TEN 扩展独立测试框架引入了两个主要概念：

1. **`extension_tester`**
2. **`ten_env_tester`**

`extension_tester` 的作用类似于一个测试驱动程序，负责设置和执行整个测试过程。另一方面，`ten_env_tester` 的行为类似于典型的 TEN 扩展的 `ten_env` 实例，使用户能够从 `extension_tester` 的回调中调用独立测试框架中的功能，例如向被测扩展发送消息和从被测扩展接收消息。

从 `extension_tester` 和 `ten_env_tester` 的 API 设计中，您可以看到它们与 **TEN 扩展** 和 `ten_env` 非常相似，这是有意为之的。主要目的是让熟悉扩展开发的用户能够快速适应独立测试框架，然后为扩展本身开发独立的测试用例。

然而，出于测试目的，不可避免地会有专门用于测试的 API 和功能。为了防止这些在实际运行时不需要的、专用于测试的功能和 API 污染运行平台 API 集，独立测试框架被设计为不直接使用或扩展 **extension** 和 `ten_env` 的 API 集。相反，它引入了独立于测试的类型和 API 集。这种分离确保了实际运行时使用的类型和 API 集与测试期间使用的类型和 API 集互不干扰，避免了任何负面影响。

## 独立测试框架内部机制

在内部，TEN 扩展独立测试框架会隐式启动一个测试应用，加载要测试的扩展插件（我们称之为扩展 A），并初始化一个包含要测试的扩展 A 和另一个专门用于测试其他扩展的扩展（我们称之为扩展 B）的图。扩展 A 的所有输入和输出消息都被重定向到扩展 B，允许用户在测试过程中自定义输入和输出，并完成实际的测试工作。

```mermaid
sequenceDiagram
  participant tester as Extension Tester
  participant testing as Extension Testing
  participant tested as Extension Tested

  tester->>testing: send messages
  testing->>tested: send messages

  tested-->>testing: send results
  testing-->>tester: send results
```

基本上，您可以将此隐藏在测试框架中的测试扩展视为被测试扩展和测试器之间的代理。它充当 TEN 图中的代理扩展，使用 TEN 环境的语言促进被测扩展和测试器之间的消息交换。

## 基本测试流程

基本测试流程和逻辑如下：

1. 创建一个扩展测试器来管理整个独立测试过程。
2. 向独立测试框架提供包含待测扩展的文件夹信息。
3. 设置测试模式，例如用于测试单个扩展的模式。
4. 开始测试。

## C++

以下是使用 Google gtest 进行 TEN 扩展独立测试的示例：

```c++
class extension_tester_basic : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    auto new_cmd = ten::cmd_t::create("hello_world");
    ten_env.send_cmd(std::move(new_cmd),
                     [](ten::ten_env_tester_t &ten_env,
                        std::unique_ptr<ten::cmd_result_t> result) {
                       if (result->get_status_code() == TEN_STATUS_CODE_OK) {
                         ten_env.stop_test();
                       }
                     });
  }
};

TEST(Test, Basic) {
  // 1. Create an extension tester to manage the entire standalone testing process.
  auto *tester = new extension_tester_basic();

  // 2. Set a testing mode, such as a mode for testing a single extension.
  tester->set_test_mode_single("default_extension_cpp");

  // 3. Start the testing.
  tester->run();

  delete tester;
}
```

## Golang

TODO: 待补充。

## Python

```python
class ExtensionTesterBasic(ExtensionTester):
    def check_hello(self, ten_env: TenEnvTester, result: CmdResult):
        statusCode = result.get_status_code()
        print("receive hello_world, status:" + str(statusCode))

        if statusCode == StatusCode.OK:
            ten_env.stop_test()

    def on_start(self, ten_env: TenEnvTester) -> None:
        new_cmd = Cmd.create("hello_world")

        print("send hello_world")
        ten_env.send_cmd(
            new_cmd,
            lambda ten_env, result: self.check_hello(ten_env, result),
        )

        print("tester on_start_done")
        ten_env.on_start_done()


def test_basic():
    # 1. Create an extension tester to manage the entire standalone testing process.
    tester = ExtensionTesterBasic()

    # 2. Set a testing mode, such as a mode for testing a single extension.
    tester.set_test_mode_single("default_extension_python")

    # 3. Start the testing.
    tester.run()
```


# 如何使用日志进行调试 (/cn/docs/ten_framework/0.11.59/testing_and_debugging/how_to_debug_with_logs)

我们将介绍如何查看日志，现在它们已移植到 stdout，因此查看它们非常容易。

例如，如果 TEN Agent 在 localhost:3000 上运行，您可能会在日志中看到类似于以下的输出：

```bash
...
[GIN] 2024/09/29 - 09:29:42 | 200 |     634.458µs |    192.168.65.1 | POST     "/ping"
[agora_5bnxv1] 09-29 09:29:42.895  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ How about you?], text_recv_ts: 1727602181374572, read_bytes: 37200, sent_frames: 117, first_frame_latency: 306ms, finish_latency: 428ms, synthesis_first_byte_latency: 300ms, synthesis_finish_latency: 332ms
[agora_5bnxv1] 09-29 09:29:43.350  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Anything exciting happening?], text_recv_ts: 1727602181383864, read_bytes: 63600, sent_frames: 199, first_frame_latency: 300ms, finish_latency: 452ms, synthesis_first_byte_latency: 295ms, synthesis_finish_latency: 332ms
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_5bnxv1 requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
2024/09/29 09:29:45 INFO handlerPing end worker="&{ChannelName:agora_5bnxv1 HttpServerPort:10024 LogFile:/tmp/astra/app-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.json Pid:7455 QuitTimeoutSeconds:60 CreateTs:1727602125 UpdateTs:1727602185}" requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |    1.038208ms |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
2024/09/29 09:29:45 ERROR handlerPing channel not existed channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |     442.375µs |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:48 INFO handlerPing start channelName=agora_5bnxv1 requestId=6fb07d85-
...
```

一旦您与 TEN Agent 交互，您应该会看到如下条目：

```bash title="Bash"
[agora_5bnxv1] 09-29 09:35:25.049  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ and the World Cup in Russia.], text_recv_ts: 1727602522313774, read_bytes: 64000, sent_frames: 200, first_frame_latency: 418ms, finish_latency: 568ms, synthesis_first_byte_latency: 413ms, synthesis_finish_latency: 458ms
[agora_5bnxv1] 09-29 09:35:25.570  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ In tech,], text_recv_ts: 1727602522347918, read_bytes: 29200, sent_frames: 92, first_frame_latency: 427ms, finish_latency: 515ms, synthesis_first_byte_latency: 419ms, synthesis_finish_latency: 460ms
[agora_5bnxv1] 09-29 09:35:26.502  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ we saw some big advancements like the rise of AI and electric cars becoming more mainstream.], text_recv_ts: 1727602522461866, read_bytes: 178800, sent_frames: 559, first_frame_latency: 510ms, finish_latency: 928ms, synthesis_first_byte_latency: 503ms, synthesis_finish_latency: 800ms
[agora_5bnxv1] 09-29 09:35:27.029  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Personally,], text_recv_ts: 1727602522468652, read_bytes: 34400, sent_frames: 108, first_frame_latency: 442ms, finish_latency: 524ms, synthesis_first_byte_latency: 436ms, synthesis_finish_latency: 463ms
```

当您看到这样的日志时，这意味着系统工作正常并且正在记录您说的每句话。


# Doc Window (/cn/docs/ten_framework/0.11.59/tman_designer/doc_window)

TEN 设计师内建文档窗口，可以查看 TEN Framework 的观念跟术语. 例如在 App 下拉选单中选择 `About App` 即可打开文档窗口并显示 App 的说明.

![TMAN Designer Doc Window](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_doc_window.png)


# 概述 (/cn/docs/ten_framework/0.11.59/tman_designer/overview)

TEN Framework 具有高度灵活性，提供了丰富的功能，包括 TEN 云商店、软件包管理、多语言支持和多平台支持等。然而，这些功能可能需要较长的学习时间和较高的技术门槛，才能熟练操作 TEN Framework 完成 AI Agent 的开发和定制化。为便于开发者使用，TEN Framework 提供了一个名为 TEN Manager 的命令行工具，使开发者能够简便地修改和定制基于 TEN Framework 构建的 AI Agent。

由于命令行工具不如图形用户界面友好，因此 TEN Manager 除了命令行界面外，还内置了基于 Web UI 的可视化开发界面 — TMAN Designer，提供丰富的功能帮助开发者开发和调试 TEN Apps 及 TEN Extension。这意味着开发者只需获取 TEN Manager 这一个工具，就能同时拥有命令行和图形用户界面的完整 TEN Framework 开发环境。

您可以通过以下链接获取 TMAN Designer：

```text
https://github.com/TEN-framework/ten-framework/releases
```

TMAN Designer 支持以下操作系统及架构，请根据您的平台获取对应版本：

* Windows
  * x86\_64
* Linux
  * x86\_64
* MacOS
  * x86\_64
  * arm64

下载并解压后，使用以下命令启动 TMAN Designer：

```shell
tman designer
```

您可以在任何 TEN App 的根目录下执行此命令，也可以在任意位置执行。如果在 TEN App 的根目录下执行，TMAN Designer 会默认将该 TEN App 载入。您也可以在 TMAN Designer 中手动载入任何 TEN App。因此，在特定 TEN App 根目录下启动 TMAN Designer 并非必要操作，但可以简化初始手动载入 TEN App 的步骤。

启动后将看到如下信息，表示 TMAN Designer 默认在端口 49483 上启动：

```text
🏆  Starting server at http://0.0.0.0:49483
```

您可以使用以下 URL 与 TMAN Designer 交互：

```text
http://127.0.0.1:49483/
```

> 注意：如果您的 TMAN Designer 运行在远程机器上，请使用远程机器的 IP 地址访问 TMAN Designer。

## 概览

使用浏览器打开 TMAN Designer 页面后，会看到如下界面：

![TMAN Designer 概观](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_overview.png)

在 TMAN Designer 右上方可以看到几个功能按钮：

* 最右侧按钮显示当前 TMAN Designer 的版本号，并自动检测是否有新版本发布。如有更新，会显示一个向上的箭头，点击后将打开新版本的下载页面。
* 左侧第一个按钮是 [TEN Agent](https://agent.theten.ai/) 按钮，点击后将打开 TEN Agent 页面。TEN Agent 是使用 TEN Framework 构建的完整 AI Agent 实现，TMAN Designer 的主要目标之一是让开发者能够轻松地通过 TMAN Designer 开发、调试和定制 TEN Agent。
* 左侧第二个按钮是 [TEN Framework](https://github.com/TEN-framework/ten-framework) 的 GitHub 页面，点击后将打开 TEN Framework 的 GitHub 仓库。如需了解更多关于 TEN Framework 的信息、源代码或提交问题，可以访问此页面。
* 最左侧有两个按钮，分别用于切换明暗模式和选择语言。

在 TMAN Designer 的左上方有几个下拉菜单，开发者可以通过这些菜单使用 TMAN Designer 的各项功能。

## 处理 TEN App

### 载入已有的 TEN App

通过 App 菜单的 Load App 按钮，可以载入已有的 TEN App。点击按钮后，会看到如下界面：

![载入已有的 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_load_app.png)

通过此文件浏览对话框，开发者可以指定已有 TEN App 的根目录来载入应用。

### 管理已载入的 TEN App

通过 App 菜单的 Manage Loaded App(s) 按钮，可以管理已载入的 TEN App。点击按钮后，会看到如下 Apps Manager 界面：

![管理已经载入的 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_manage_loaded_apps.png)

开发者可以通过此对话框：

* 卸载已载入的 TEN App
* 重新载入指定的 TEN App
* 重新载入所有已载入的 TEN App
* 安装指定 App 的所有依赖
* 运行指定 App 的功能

### 安装指定 App 的所有依赖

开发者可以在 TEN App 中安装各种 TEN 软件包，一个 TEN App 所依赖的所有 TEN 软件包都会记录在 `manifest.json` 文件中。以下是一个 `manifest.json` 文件的示例：

```json
{
  "type": "app",
  "name": "aaa",
  "version": "0.8.18",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.8.18"
    },
    {
      "type": "extension",
      "name": "ffmpeg_muxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_demuxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_client",
      "version": "^0.1.0"
    },
    {
      "type": "protocol",
      "name": "msgpack",
      "version": "^0.8.18"
    }
  ],
  "scripts": {
    "start": "python3 tools/run_script.py start",
    "build": "python3 tools/run_script.py build"
  }
}
```

从上述配置可以看出，该 TEN App 依赖以下几个 TEN 软件包：

* `ten_runtime`：TEN Framework 的系统软件包
* `ffmpeg_demuxer`：用于解码音频和视频的 TEN Extension 软件包
* `ffmpeg_muxer`：用于编码音频和视频的 TEN Extension 软件包
* `ffmpeg_client`：用于使用另外两个 ffmpeg extension 软件包处理音频和视频的 TEN Extension 软件包
* `msgpack`：用于使用 MessagePack 协议进行消息序列化和反序列化的 TEN Protocol 软件包

实际上，在完成 TEN App 所有依赖的安装后，系统中的 TEN 软件包文件结构如下：

```text
├── ten_packages
│   ├── extension
│   │   ├── ffmpeg_client
│   │   ├── ffmpeg_demuxer
│   │   └── ffmpeg_muxer
│   ├── protocol
│   │   └── msgpack
│   └── system
│       ├── ffmpeg
│       └── ten_runtime
```

值得注意的是，尽管没有在 `manifest.json` 中显式声明，系统中还多了一个 `ffmpeg` 系统软件包。这是因为 `ffmpeg_demuxer` 和 `ffmpeg_muxer` 依赖了 `ffmpeg` 这个系统软件包，TEN Manager 会自动识别并安装这些传递依赖。

通过 TMAN Designer 的 Apps Manager 界面，开发者可以对指定的 App 执行 Install All 操作，轻松完成上述所有依赖的安装过程。点击按钮后，会看到如下界面，显示安装进度：

![安装指定 App 的所有依赖](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_install_all.png)

### 从云商店安装 TEN Extension

通过 Graph 菜单的 Open Extension Store 按钮，可以打开 TEN 云商店，并从云商店安装 TEN Extension 到当前的 TEN App。点击按钮后，会看到如下界面：

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_store.png)

在 TEN 云商店中，开发者可以搜索所需的 TEN Extension，并点击 Install 按钮安装到当前的 TEN App。安装完成后，可以在当前 TEN App 中查看已安装的 TEN Extension。

### 运行 TEN App

在 Apps Manager 中，可以对指定的 App 执行预设的运行操作。在 Apps Manager 中点击 Run 按钮后，会看到如下界面：

![运行 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_run.png)

在对话框中，开发者可以从下拉菜单中选择要执行的预设操作。通常包括以下几种操作：

* 运行 App：启动 App 并在 TMAN Designer 的消息窗口中显示 App 的输出信息。
* 编译 App：对 App 进行编译。由于 TEN Framework 支持多种编程语言，有些语言需要编译，有些则无需编译。例如，基于 C++ 开发的 TEN App 需要编译，而基于 Python 开发的 TEN App 则无需编译。

以下是一个基于 C++ 开发的 TEN App 的编译界面：

![编译 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_build.png)

编译完成后，可以继续执行 TEN App 的启动操作，系统会弹出一个窗口，实时显示 TEN App 的运行日志：

![启动 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_start.png)

## 处理 TEN Graph

### 载入已有的 TEN Graph

通过 Graph 菜单的 Open Existing Graph 按钮，可以载入已有的 TEN Graph。载入成功后，会看到类似如下界面：

![TEN Graph](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_graph_overview.png)

以这个 Graph 为例，它包含 3 个 TEN Extensions，分别是：

* `ffmpeg_demuxer`
* `ffmpeg_muxer`
* `ffmpeg_client`

每个 TEN Extension 都有其对应的输入和输出。以 `ffmpeg_demuxer` 为例，它具有以下输入与输出：

* 2 个 cmd 输入
* 1 个 cmd 输出
* 1 个 audio\_frame 输出
* 1 个 video\_frame 输出

点击输入输出的数字按钮，可以打开如下对话框，用于管理输入输出的连接：

![TEN Message Input/Output Dialog](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_msg_info_dialog.png)

### 自动排布 Graph

当开发者手动调整 TEN Graph 中的 TEN Extension 位置后，可以点击 Graph 菜单中的 Auto Layout 按钮，自动排布 TEN Graph 中的 TEN Extension。

### 打开 TEN Extension 的上下文菜单

在 Graph 中，对每个 TEN Extension 右键点击，可以打开 TEN Extension 的上下文菜单：

![TEN extension context menu](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_context_menu.png)

在上下文菜单中，可以打开 TEN Extension 的 `manifest.json` 文件、`property.json` 文件，也可以直接在 TEN Extension 的根目录下启动终端，方便开发者对该 TEN Extension 进行修改和定制：

![Open Manifest](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_context_menu_effect.png)


# 如何在 Python 扩展中运行本地 AI 模型 (/cn/docs/ten_framework/0.11.60/ai_model/how_to_run_local_model_in_python_extensions)

在 TEN framework 中，扩展可以使用第三方 AI 服务或在本地运行 AI 模型，以提高性能并降低成本。本教程介绍了如何在 Python 扩展中运行本地 AI 模型，以及如何在扩展中与之交互。

## 步骤 1：检查硬件要求

在本地运行 AI 模型之前，请确保您的硬件满足必要的要求。要验证的关键组件包括：

* **CPU/GPU**：检查模型是否需要特定的处理能力。
* **内存**：确保有足够的内存来加载和运行模型。

验证您的系统是否可以支持模型的需求，以确保顺利运行。

## 步骤 2：安装必要的软件和依赖项

硬件准备就绪后，安装所需的软件和依赖项。请按照以下步骤操作：

1. **操作系统**：确保与您的模型兼容。大多数 AI 框架都支持 Windows、macOS 和 Linux，尽管可能需要特定版本。

2. **Python 版本**：确保与 TEN Python 运行平台和模型兼容。

3. **所需的库**：安装必要的库，例如：

   * TensorFlow
   * PyTorch
   * Numpy
   * vllm

   您可以将所需的依赖项列在 `requirements.txt` 文件中，以便轻松安装。

4. **下载模型**：获取您计划运行的 AI 模型的本地版本。

## 步骤 3：实现您的 Python 扩展

以下示例说明了如何在 Python 扩展中使用 `vllm` 推理引擎实现基本的文本生成功能。

首先，在扩展中初始化本地模型：

```python
from ten_runtime import (
    Extension,
    TenEnv,
    Cmd,
    CmdResult,
)
from vllm import LLM

class TextGenerationExtension(Extension):
    def on_init(self, ten_env: TenEnv) -> None:
        self.llm = LLM(model="<model_path>")
        ten_env.on_init_done()
```

接下来，实现 `on_cmd` 方法以根据提供的输入处理文本生成：

```python
    def on_cmd(self, ten_env: TenEnv, cmd: Cmd) -> None:
        prompt = cmd.get_property_string("prompt")

        outputs = self.llm.generate(prompt)
        generated_text = outputs[0].outputs[0].text

        cmd_result = CmdResult.create(StatusCode.OK)
        cmd_result.set_property_string("result", generated_text)
        ten_env.return_result(cmd_result, cmd)
```

在此代码中，`on_cmd` 方法检索 `prompt`，使用模型生成文本，并将生成的文本作为命令结果返回。

您可以通过处理相关的输入类型来调整此方法，以实现其他功能，例如图像识别或语音转文本。

## 步骤 4：卸载模型

在扩展清理期间卸载模型以释放资源非常重要：

```python
import gc
import torch

class TextGenerationExtension(Extension):
    ...

    def on_deinit(self, ten_env: TenEnv) -> None:
        del self.llm
        gc.collect()
        torch.cuda.empty_cache()
        torch.distributed.destroy_process_group()
        print("Successfully deleted the LLM pipeline and freed GPU memory!")
        ten_env.on_deinit_done()
```

这确保了有效的内存管理，尤其是在使用 GPU 资源时。

## 总结

在 TEN Python 扩展中运行本地模型类似于原生 Python 开发。通过在适当的扩展生命周期方法中加载和卸载模型，您可以轻松集成本地 AI 模型并有效地与之交互。


# API (/cn/docs/ten_framework/0.11.60/api/api)

在 TEN Framework 中，extension 的 API 定义包含在 `manifest.json` 文件中。TEN Framework 定义 API 时，不使用 JSON Schema，而是使用一种类似 JSON 的语法来定义 API Schema。

## 为什么不使用 JSON Schema

TEN Framework 选择不使用标准 JSON Schema 的原因如下：

* **类型系统差异**：TEN Framework 的类型系统包含了许多 JSON Schema 没有的类型，例如 `int8`、`int16`、`int32`、`int64`、`uint8`、`uint16`、`uint32`、`uint64`、`float32`、`float64`、`buf`、`ptr` 等。

* **减少冗余**：在 TEN Framework 的 API 定义中，许多地方不需要具有弹性的设定。例如，message 的 schema 一定是 object 类型，不能是其他类型。如果要求开发者明确写 `"type": "object"` 反而显得多余，且容易引起混淆。

基于以上原因，TEN Framework 采用了一种更自然的定义方式，使用类似 JSON 的语法来定义 API Schema。

## API 顶层定义

API Schema 的定义放在 `manifest.json` 文件中的 `api` 字段中：

```json
{
  "api": {
    "property": {
      "properties": {...},
      "required": [...]
    },
    "cmd_in": [{
      "name": "foo",
      "property": {
        "properties": {...},
        "required": [...]
      },
      "result": {
        "property": {
          "properties": {...},
          "required": [...]
        }
      }
    }]
  }
}
```

## API Schema 定义模式

### 表达 object 类型字段

使用 `properties` 字段来描述 object 类型字段内每个属性的类型：

```json
{
  "my_object": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "age": {
        "type": "int32"
      }
    }
  }
}
```

### 表达必需和可选字段

与 object 类型字段同级的 `required` 字段，用来描述必需和可选字段：

```json
{
  "some_concept": {
    "properties": {
      "foo": {},
      "bar": {}
    },
    "required": ["foo"]
  }
}
```

合理使用 `required` 字段确保关键数据的存在：

```json
{
  "name": "critical_cmd",
  "property": {
    "properties": {
      "essential_param": {
        "type": "string"
      },
      "optional_param": {
        "type": "int32"
      }
    },
    "required": ["essential_param"]
  }
}
```

### 表达 array 类型字段

使用 `items` 字段来描述 array 类型字段内每个元素的类型：

```json
{
  "my_array": {
    "type": "array",
    "items": {
      "type": "string"
    }
  }
}
```

### type 字段的使用原则

如果一个字段只能是特定类型（由 TEN Framework 本身约束），就不需要明确写 `type` 字段。如果字段可以是多种类型，则需要明确写 `type` 字段。

例如，下面的 `some_concept` 字段如果只能是 object 类型，就不需要明确写 `type` 字段。而其中的 `foo` 字段可以是多种类型，则需要明确写 `type` 字段：

```json
{
  "some_concept": {
    "properties": {
      "foo": {
        "type": "string"
      },
      "bar": {
        "type": "object",
        "properties": {
          "baz": {
            "type": "string"
          }
        },
        "required": ["baz"]
      }
    },
    "required": ["foo"]
  }
}
```

## 表达 enum 类型字段

使用 `enum` 字段来描述枚举类型的所有可能值：

```json
{
  "api": {
    "property": {
      "properties": {
        "status": {
          "type": "string",
          "enum": ["CREATING", "ACTIVE", "DELETING", "FAILED"]
        }
      }
    }
  }
}
```

## Extension Property Schema

定义 extension 属性的完整示例：

```json
{
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        },
        "config": {
          "type": "object",
          "properties": {
            "enabled": {
              "type": "bool"
            },
            "settings": {
              "type": "array",
              "items": {
                "type": "string"
              }
            }
          },
          "required": ["enabled"]
        }
      },
      "required": ["api_key", "max_tokens", "temperature", "config"]
    }
  }
}
```

## Command-like Message Schema

命令支持输入 (`cmd_in`) 和输出 (`cmd_out`)：

```json
{
  "api": {
    "cmd_in": [
      {
        "name": "process_text",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            },
            "options": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "max_length": {
                  "type": "int32"
                }
              }
            }
          },
          "required": ["text"]
        },
        "result": {
          "property": {
            "properties": {
              "processed_text": {
                "type": "string"
              },
              "confidence": {
                "type": "float32"
              }
            },
            "required": ["processed_text"]
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "notify_completion",
        "property": {
          "properties": {
            "task_id": {
              "type": "string"
            },
            "status": {
              "type": "string"
            }
          },
          "required": ["task_id", "status"]
        }
      }
    ]
  }
}
```

## Data-like Message Schema

数据类型消息的 schema 定义：

```json
{
  "api": {
    "data_in": [
      {
        "name": "input_data",
        "property": {
          "properties": {
            "content": {
              "type": "string"
            },
            "metadata": {
              "type": "object",
              "properties": {
                "timestamp": {
                  "type": "int64"
                }
              }
            }
          },
          "required": ["content"]
        }
      }
    ],
    "data_out": [
      {
        "name": "output_data",
        "property": {
          "properties": {
            "result": {
              "type": "string"
            }
          }
        }
      }
    ],
    "audio_frame_in": [
      {
        "name": "audio_input"
      }
    ],
    "audio_frame_out": [
      {
        "name": "audio_output"
      }
    ],
    "video_frame_in": [
      {
        "name": "video_input"
      }
    ],
    "video_frame_out": [
      {
        "name": "video_output"
      }
    ]
  }
}
```

## 常见错误和解决方案

<div className="fd-steps">
  <div className="fd-step">
    ### 属性名不符合规范

    **错误示例**：属性名包含连字符或以数字开头

    ```json
    {
      "api-key": {  // 错误：包含连字符
        "type": "string"
      },
      "2nd_param": {  // 错误：以数字开头
        "type": "int32"
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "api_key": {
        "type": "string"
      },
      "second_param": {
        "type": "int32"
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### 数组类型缺少 items 定义

    **错误示例**：

    ```json
    {
      "my_array": {
        "type": "array"
        // 缺少 items 定义
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "my_array": {
        "type": "array",
        "items": {
          "type": "string"
        }
      }
    }
    ```
  </div>

  <div className="fd-step">
    ### 对象类型缺少 properties 定义

    **错误示例**：

    ```json
    {
      "my_object": {
        "type": "object"
        // 缺少 properties 定义
      }
    }
    ```

    **正确示例**：

    ```json
    {
      "my_object": {
        "type": "object",
        "properties": {
          "field1": {
            "type": "string"
          }
        }
      }
    }
    ```
  </div>
</div>

## 总结

TEN Framework 的 Schema 系统提供了一个强大且结构化的方式来定义和验证数据结构，确保组件之间的一致性和类型安全。通过遵循设计原则和最佳实践，开发者可以创建健壮、可维护的 TEN 应用程序和扩展。

该系统支持丰富的类型定义、复杂的 API 描述以及灵活的消息转换功能，为构建大规模的实时多模态应用提供了坚实的基础。


# 日志 (/cn/docs/ten_framework/0.11.60/api/log)

TEN framework 允许使用不同语言开发的 extension 在同一进程中运行。这使得需要以统一的日志格式和信息在统一的日志中查看所有这些 extension 的日志，从而简化调试。

为了解决这个问题，`ten_env` 提供了一个日志 API。在 extension 的每个回调中，都可以访问 `ten_env` 的实例。使用此实例，您可以调用日志 API，将使用各种语言开发的 extension 的日志输出到统一的日志输出中。

## 日志级别

TEN framework 支持以下日志级别，按严重程度递增：

* **DEBUG** (1): 调试信息
* **INFO** (2): 一般信息
* **WARN** (3): 警告信息
* **ERROR** (4): 错误信息

## API 接口

### C++ 语言

```cpp
#include "ten_runtime/binding/cpp/ten.h"

class Extension {
public:
    void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr<ten::cmd_t> cmd) {
        // 使用便捷宏
        TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
        TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
        TEN_ENV_LOG_WARN(ten_env, "Warning message");
        TEN_ENV_LOG_ERROR(ten_env, "Error occurred");

        // 直接调用方法，支持 category 和 fields 参数
        ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
                   "Direct log call", nullptr, nullptr);
    }
};
```

C++ 提供了以下便捷宏：

* `TEN_ENV_LOG_DEBUG(ten_env, msg)`
* `TEN_ENV_LOG_INFO(ten_env, msg)`
* `TEN_ENV_LOG_WARN(ten_env, msg)`
* `TEN_ENV_LOG_ERROR(ten_env, msg)`

### Python 语言

```python
from ten_runtime import Extension, TenEnv

class MyExtension(Extension):
    def on_cmd(self, ten_env: TenEnv, cmd):
        # 使用不同级别的日志方法
        ten_env.log_debug("调试信息")
        ten_env.log_info("一般信息")
        ten_env.log_warn("警告信息")
        ten_env.log_error("错误信息")

        # 支持格式化字符串和额外参数
        test_value = "example"
        ten_env.log_info(f"处理命令，测试值: {test_value}")

        # 支持 category 和 fields 参数
        ten_env.log_info("带分类的日志", category="my_extension")
```

Python 提供了以下方法：

* `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
* `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`

### Go 语言

```go
package main

import "ten_runtime/ten"

type MyExtension struct {
    ten.DefaultExtension
}

func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    // 使用不同级别的日志方法
    tenEnv.LogDebug("调试信息")
    tenEnv.LogInfo("一般信息")
    tenEnv.LogWarn("警告信息")
    tenEnv.LogError("错误信息")

    // 支持格式化
    cmdName := cmd.GetName()
    tenEnv.LogInfo("处理命令: " + cmdName)

    // 使用完整的 Log 方法支持更多参数
    category := "my_extension"
    tenEnv.Log(ten.LogLevelInfo, "带分类的日志", &category, nil, nil)
}
```

Go 提供了以下方法：

* `LogDebug(msg string) error`
* `LogInfo(msg string) error`
* `LogWarn(msg string) error`
* `LogError(msg string) error`
* `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`

### Node.js/TypeScript 语言

```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";

class MyExtension extends Extension {
    async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
        // 使用不同级别的日志方法
        tenEnv.logDebug("调试信息");
        tenEnv.logInfo("一般信息");
        tenEnv.logWarn("警告信息");
        tenEnv.logError("错误信息");

        // 支持字符串拼接
        const cmdName = cmd.getName();
        tenEnv.logInfo("处理命令: " + cmdName);

        // 支持 category 和 fields 参数
        tenEnv.logInfo("带分类的日志", "my_extension");
    }
}
```

Node.js 提供了以下方法：

* `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
* `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
* `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
* `logError(message: string, category?: string, fields?: Value): TenError | undefined`

## 使用示例

### 基本使用

```python
# Python 示例
def on_start(self, ten_env: TenEnv):
    ten_env.log_info("extension 启动")
    ten_env.on_start_done()

def on_cmd(self, ten_env: TenEnv, cmd):
    cmd_name = cmd.get_name()
    ten_env.log_debug(f"收到命令: {cmd_name}")

    # 处理命令逻辑
    if cmd_name == "hello":
        ten_env.log_info("处理 hello 命令", category="command_handler")
        # ... 处理逻辑
    else:
        ten_env.log_warn(f"未知命令: {cmd_name}")
```

### 在线程中使用

```python
import threading
import time

def log_routine(self, ten_env: TenEnv):
    """在单独线程中记录日志"""
    while not self.stop_flag:
        self.log_count += 1
        ten_env.log_info(f"日志消息 {self.log_count}")
        time.sleep(0.05)

def on_start(self, ten_env: TenEnv):
    # 启动日志线程
    self.log_thread = threading.Thread(
        target=self.log_routine, args=(ten_env,)
    )
    self.log_thread.start()
    ten_env.on_start_done()
```

### 错误处理

```python
def on_cmd(self, ten_env: TenEnv, cmd):
    try:
        # 业务逻辑
        result = self.process_command(cmd)
        ten_env.log_info("命令处理成功")
    except ValueError as e:
        ten_env.log_error(f"参数错误: {str(e)}", category="error_handling")
    except Exception as e:
        ten_env.log_error(f"严重错误: {str(e)}", category="error_handling")
        raise
```

## 日志格式和输出

TEN framework 的日志系统会自动包含以下信息：

* **时间戳**: 日志记录的时间
* **日志级别**: DEBUG、INFO、WARN、ERROR
* **函数名**: 调用日志的函数名（如果可用）
* **文件名**: 调用日志的源文件名（如果可用）
* **行号**: 调用日志的代码行号（如果可用）
* **分类**: 可选的日志分类信息
* **消息内容**: 用户提供的日志消息

输出格式类似：

```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] 用户日志消息
```

## 高级日志配置

TEN framework 支持通过 `property.json` 文件配置高级日志功能。这允许您自定义日志的输出格式、目标和过滤规则。

### 配置结构

在 `property.json` 文件中，您可以在 `ten.log` 部分定义高级日志配置：

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "my_extension"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

### 配置组件说明

<div className="fd-steps">
  <div className="fd-step">
    #### 日志处理器 (Handlers)

    每个处理器包含三个主要组件：

    * **matchers**: 定义哪些日志消息会被此处理器处理
    * **formatter**: 定义日志的输出格式
    * **emitter**: 定义日志的输出目标
  </div>

  <div className="fd-step">
    #### 匹配器 (Matchers)

    匹配器用于过滤日志消息：

    ```json
    {
      "level": "info",      // 日志级别: "off", "debug", "info", "warn", "error"
      "category": "auth"    // 可选：日志分类（字符串）
    }
    ```

    支持的日志级别：

    * `"off"`: 关闭日志
    * `"debug"`: 调试级别
    * `"info"`: 信息级别
    * `"warn"`: 警告级别
    * `"error"`: 错误级别
  </div>

  <div className="fd-step">
    #### 格式器 (Formatter)

    控制日志的输出格式：

    ```json
    {
      "type": "plain",      // 格式类型: "plain" 或 "json"
      "colored": true       // 可选：是否启用颜色输出（仅对 plain 格式有效）
    }
    ```

    * **plain**: 人类可读的纯文本格式
    * **json**: JSON 结构化格式，便于日志分析工具处理
  </div>

  <div className="fd-step">
    #### 发射器 (Emitter)

    定义日志的输出目标：

    ##### 控制台输出

    ```json
    {
      "type": "console",
      "config": {
        "stream": "stdout",  // "stdout" 或 "stderr"
        "encryption": {      // 可选：加密配置
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### 文件输出

    ```json
    {
      "type": "file",
      "config": {
        "path": "/path/to/logfile.log",  // 日志文件路径
        "encryption": {                  // 可选：加密配置
          "enabled": true,
          "algorithm": "AES-CTR",
          "params": {
            "key": "your-encryption-key",
            "nonce": "your-nonce"
          }
        }
      }
    }
    ```

    ##### OTLP 输出

    OTLP (OpenTelemetry Protocol) 输出允许您将日志发送到支持 OpenTelemetry 的后端系统，如 Jaeger、Prometheus、Grafana 等。

    ```json
    {
      "type": "otlp",
      "config": {
        "endpoint": "http://localhost:4317",  // OTLP 接收端点地址
        "protocol": "grpc",                   // 可选：传输协议，"grpc"（默认）或 "http"
        "headers": {                          // 可选：自定义 HTTP 头
          "Authorization": "Bearer token",
          "X-Custom-Header": "value"
        },
        "service_name": "my-service"          // 可选：服务名称，用于标识日志来源
      }
    }
    ```

    配置参数说明：

    * **endpoint**: OTLP 接收端点的 URL（必需）
    * **protocol**: 传输协议，支持 `"grpc"` 或 `"http"`，默认为 `"grpc"`
    * **headers**: 自定义 HTTP 请求头，用于认证或其他元数据传递（可选）
    * **service\_name**: 服务名称，用于在日志聚合系统中标识不同的服务（可选）
  </div>
</div>

### 配置示例

#### 示例 1: 多级别日志分离

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/errors.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "plain",
            "colored": true
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 2: 按分类分离日志

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "debug",
              "category": "database"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/var/log/app/database.log"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "info",
              "category": "api"
            }
          ],
          "formatter": {
            "type": "plain"
          },
          "emitter": {
            "type": "console",
            "config": {
              "stream": "stdout"
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 3: 加密日志输出

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "file",
            "config": {
              "path": "/secure/logs/app.log",
              "encryption": {
                "enabled": true,
                "algorithm": "AES-CTR",
                "params": {
                  "key": "your-32-character-encryption-key",
                  "nonce": "your-16-character-nonce"
                }
              }
            }
          }
        }
      ]
    }
  }
}
```

#### 示例 4: OTLP 日志输出到 OpenTelemetry 收集器

```json
{
  "ten": {
    "log": {
      "handlers": [
        {
          "matchers": [
            {
              "level": "info"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "http://localhost:4317",
              "protocol": "grpc",
              "service_name": "ten-framework-app"
            }
          }
        },
        {
          "matchers": [
            {
              "level": "error"
            }
          ],
          "formatter": {
            "type": "json"
          },
          "emitter": {
            "type": "otlp",
            "config": {
              "endpoint": "https://api.monitoring.example.com/v1/logs",
              "protocol": "http",
              "headers": {
                "Authorization": "Bearer your-api-token",
                "X-Environment": "production"
              },
              "service_name": "ten-framework-app"
            }
          }
        }
      ]
    }
  }
}
```

### 配置注意事项

1. **多个处理器**: 可以定义多个处理器来实现复杂的日志路由策略
2. **匹配优先级**: 按照配置文件中的顺序匹配处理器
3. **加密安全**: 加密密钥和 nonce 应该安全存储，不要硬编码在配置文件中
4. **文件权限**: 确保日志文件路径具有适当的写入权限
5. **性能考虑**: JSON 格式可能会比 plain 格式产生更大的日志文件
6. **OTLP 端点**:
   * 使用 gRPC 协议时，默认端口通常为 4317
   * 使用 HTTP 协议时，默认端口通常为 4318
   * 确保 OTLP 收集器端点可访问且配置正确
7. **认证**: 使用 OTLP 时，如果后端需要认证，请在 headers 中添加相应的认证信息

## 最佳实践

1. **选择合适的日志级别**：
   * 使用 `DEBUG` 记录详细的调试信息
   * 使用 `INFO` 记录正常的业务流程
   * 使用 `WARN` 记录潜在问题
   * 使用 `ERROR` 记录错误信息

2. **提供有意义的日志消息**：
   * 包含上下文信息（如命令名称、参数值等）
   * 使用描述性的消息而不是简单的标识符
   * 合理使用 category 参数对日志进行分类

3. **避免过度日志记录**：
   * 不要在高频循环中使用过多的日志
   * 在生产环境中适当调整日志级别
   * 使用高级日志配置的匹配器来过滤不必要的日志

4. **线程安全**：
   * TEN framework 的日志 API 是线程安全的
   * 可以在任何线程中安全调用日志方法

5. **高级配置建议**：
   * 为不同的模块使用不同的 category 便于日志分析
   * 在生产环境中考虑使用文件输出以便于日志持久化
   * 对敏感日志使用加密功能
   * 定期轮转日志文件以控制磁盘使用
   * 使用 JSON 格式便于与日志分析工具集成
   * 考虑使用 OTLP 输出将日志集成到现代可观测性平台（如 Grafana、Jaeger、Datadog 等）
   * OTLP 输出与 OpenTelemetry 生态系统无缝集成，便于实现分布式追踪和日志关联

整体效果如下图所示：

![多语言 extension 的统一日志输出](https://ten-framework-assets.s3.amazonaws.com/doc-assets/log.png)


# required (/cn/docs/ten_framework/0.11.60/api/required)

## `required` 字段的原则

1. **通信链路中，源的 `required` 字段必须是目标 `required` 字段的超集**

   这意味着如果目标 extension 要求某个字段为必需，那么源 extension 也必须将该字段标记为必需。这确保了消息在传递过程中不会缺少必要的字段。

2. **源和目标 `required` 字段中的相同字段名称必须具有兼容的类型**

   框架会验证字段类型的兼容性，确保数据可以正确传递和解析。类型兼容性检查包括基本类型匹配、数组项类型匹配以及对象属性类型匹配。

## `required` 字段的使用

### Extension 发送消息时

当 extension 调用 `send_<foo>(msg_X)` 或 `return_result(result_Y)` 时，框架会根据 extension 中的相应 schema 检查 `msg_X` 或 `result_Y`。如果 `msg_X` 或 `result_Y` 缺少 schema 中标记为 `required` 的任何字段，则 schema 检查将失败。

以下三种情况的处理方式相同：

1. **发送 command 且 schema 检查失败时**

   `send_<foo>` 将立即返回 `false`，错误讯息将包含 schema 检查失败的错误消息。

2. **`return_result` 未通过 schema 检查时**

   `return_result` 将返回 `false`，错误讯息将包含 schema 检查失败的错误消息，通常会显示缺少的必需字段，例如："the required properties are absent: 'foo'"。

3. **发送 data-like message（如数据、音频帧或视频帧）时**

   `send_<foo>` 将返回 `false`，错误讯息将包含 schema 检查失败的错误消息。

**注意：** 框架的 schema 系统仅在字段存在时生效。如果某个字段在 schema 中定义但在消息中不存在，且该字段未在 `required` 中列出，则不会进行类型检查。只有当字段同时存在于消息中和 schema 定义中时，才会进行类型验证。

### Extension 接收消息时

在 TEN runtime 将 `msg_X` 或 `result_Y` 传递给 extension 的 `on_<foo>()` 或结果处理程序之前，它会检查 `msg_X` 或 `result_Y` 的 schema 中定义的所有 `required` 字段是否都存在。如果缺少任何 `required` 字段，则 schema 检查将失败。

1. **传入消息是 command 时**

   TEN runtime 会将错误返回给上一个 extension，错误消息会指明缺少的必需字段。

2. **传入消息是 TEN 命令结果时**

   TEN runtime 会将结果的 `status_code` 更改为错误，添加缺少的 `required` 字段，并根据其类型将这些字段的值设置为默认值。

3. **传入消息是数据类 TEN 消息时**

   TEN runtime 将直接丢弃该数据类消息。

### 错误消息格式

当 `required` 字段验证失败时，框架会提供详细的错误信息：

* 单个缺失字段：`"the required properties are absent: 'field_name'"`
* 多个缺失字段：`"the required properties are absent: 'field1', 'field2'"`
* 嵌套对象中的缺失字段：`".nested_object: the required properties are absent: 'field_name'"`


# 类型系统 (/cn/docs/ten_framework/0.11.60/api/type_system)

## 类型

TEN framework 类型系统是 TEN framework 内用于定义值的数据类型的系统。开发者可以通过 TEN schema 声明 message 或 extension property 的类型。

TEN framework 类型系统包括基本类型和复合类型。基本类型包括以下内容：

| 类型      | 描述                     |
| ------- | ---------------------- |
| null    | 空值类型。                  |
| bool    | 布尔值，true 或 false。      |
| int8    | 8 位有符号整数。              |
| int16   | 16 位有符号整数。             |
| int32   | 32 位有符号整数。             |
| int64   | 64 位有符号整数。             |
| uint8   | 8 位无符号整数。              |
| uint16  | 16 位无符号整数。             |
| uint32  | 32 位无符号整数。             |
| uint64  | 64 位无符号整数。             |
| float32 | 单精度（32 位）IEEE 754 浮点数。 |
| float64 | 双精度（64 位）IEEE 754 浮点数。 |
| string  | Unicode (UTF-8) 字符序列。  |
| buf     | 8 位无符号字节序列。            |
| ptr     | 指向内存地址的指针。             |

复合类型包括以下内容：

| 类型     | 描述                   |
| ------ | -------------------- |
| array  | 相同类型的元素集合。           |
| object | 表示复杂的键/值对。键类型始终为字符串。 |

对于基本类型，可以使用 `get_property()` / `set_property()` 等方法访问或设置属性。例如：

```cpp
// 获取属性值
int32_t value = cmd.get_property_int32("property_name");

// 设置属性值
cmd.set_property("property_name", 100);
```

对于复合类型，通常需要使用相关的序列化方法。例如：

```go
type MyProp struct {
    Name string `json:"name"`
}

var prop MyProp
bytes, _ := json.Marshal(&prop)
cmd.SetPropertyFromJSONBytes("property_name", bytes)
```

## JSON 类型映射

当从 JSON 创建 TEN value 时，TEN framework 使用以下映射规则：

| JSON 类型 | TEN 类型映射规则                                   |
| ------- | -------------------------------------------- |
| 整数      | 如果值在 int32 范围内，优先映射为 int32；否则映射为 int64       |
| 浮点数     | 如果值在 float32 范围内，优先映射为 float32；否则映射为 float64 |
| 字符串     | 映射为 string                                   |
| 布尔值     | 映射为 bool                                     |
| null    | 映射为 null                                     |
| 数组      | 映射为 array                                    |
| 对象      | 映射为 object                                   |

需要注意的是，JSON 中的数字类型在没有明确 schema 定义的情况下，会根据数值大小自动选择合适的 TEN 类型。这确保了数据的精度和效率。

## 类型和 schema

如果指定了 TEN schema，则将根据相应的 TEN schema 确定属性类型。如果未指定 TEN schema，则将根据初始值赋值的类型确定属性类型。例如，如果初始赋值为 C++ 的 `int32_t`，则属性类型为 `int32`；如果使用 JSON 完成初始赋值，则将根据 JSON 处理规则确定类型。

当有 TEN schema 定义时，JSON 数据会被强制转换为 schema 指定的类型，如果转换失败则会报错。

## 转换规则

TEN framework 支持不同类型值之间的灵活自动转换。转换分为安全转换和不安全转换两种类型。

### 类型兼容性

TEN framework 使用类型兼容性规则来确定两个类型之间是否可以进行转换：

* **整数类型兼容**：所有整数类型（int8, int16, int32, int64, uint8, uint16, uint32, uint64）之间相互兼容
* **浮点数兼容整数**：浮点数类型可以接受整数类型的值（当整数值在浮点数表示范围内时）
* **其他类型**：bool, ptr, string, array, object, buf, null 类型只与自身兼容

### 安全转换（必须成功）

将较低精度类型转换为较高精度类型始终是安全的，并且保证成功，因为较高精度类型可以完全容纳较低精度类型的值而不会丢失数据。

在 TEN 类型系统中，安全转换规则如下：

1. 在有符号整数类型中，从较低精度到较高精度
2. 在无符号整数类型中，从较低精度到较高精度
3. 从较小范围的无符号整数到较大范围的有符号整数（当值在目标类型范围内时）
4. 在浮点数类型中，从 float32 到 float64

| 从       | 到（允许的安全转换）                                       |
| ------- | ------------------------------------------------ |
| int8    | int16 / int32 / int64                            |
| int16   | int32 / int64                                    |
| int32   | int64                                            |
| uint8   | uint16 / uint32 / uint64 / int16 / int32 / int64 |
| uint16  | uint32 / uint64 / int32 / int64                  |
| uint32  | uint64 / int64                                   |
| float32 | float64                                          |

例如：

```cpp
// 设置属性值。TEN runtime 中 `property_name` 的类型为 `int32`。
cmd.set_property("property_name", 100);

// 获取属性值。正确。
int32_t value = cmd.get_property_int32("property_name");

// 获取属性值。正确。TEN 类型系统会自动将类型转换为 `int64`。
int64_t value2 = cmd.get_property_int64("property_name");

// 获取属性值。不正确，会抛出一个错误，因为这是不安全转换。
int16_t error_type = cmd.get_property_int16("property_name");
```

### 不安全转换（可能失败）

将较高精度类型转换为较低精度类型是不安全的，因为较高精度类型的值可能超出较低精度类型的范围，从而导致数据丢失。此转换称为不安全转换。执行不安全转换时，TEN runtime 会进行严格的范围检查。如果发生溢出或精度丢失，TEN 类型系统将抛出一个错误。

在 TEN framework 类型系统中，不安全转换包括：

1. 将较高精度整数转换为较低精度整数
2. 将有符号整数转换为无符号整数（当值为负数时）
3. 将无符号整数转换为有符号整数（当值超出有符号整数范围时）
4. 将 `float64` 转换为 `float32`
5. 将浮点数转换为整数（当有小数部分或超出整数范围时）
6. 整数到浮点数的转换（当可能发生精度丢失时）

| 转换类型              | 成功条件                   |
| ----------------- | ---------------------- |
| 高精度 → 低精度         | 值必须在目标类型范围内            |
| 有符号 → 无符号         | 值必须非负且在目标类型范围内         |
| 无符号 → 有符号         | 值必须在目标类型范围内            |
| float64 → float32 | 值必须在 float32 范围内且不丢失精度 |
| 浮点数 → 整数          | 值必须在目标整数类型范围内且无小数部分    |
| 整数 → 浮点数          | 整数值必须能被浮点数精确表示，不发生精度丢失 |

#### 范围检查细节

TEN framework 实现了精确的范围检查算法：

* **整数范围检查**：使用标准的 MIN/MAX 常量进行比较
* **浮点数范围检查**：使用 `ldexp` 函数计算精确的边界值
* **精度检查**：对于整数到浮点数的转换，会进行往返转换验证以确保精度不丢失

### 转换触发时机

TEN runtime 在以下情况下执行类型转换：

1. **加载 `property.json` 时**
   * JSON 整数根据数值大小解析为相应的整数类型
   * JSON 浮点数根据数值大小解析为相应的浮点数类型
   * 如果属性有定义的 TEN schema，会根据 schema 进行强制类型转换（可能为不安全转换）

2. **调用 `set_property_from_json()` 等方法时**
   * 传递序列化的 JSON 字符串时，会根据 schema 执行类型转换

3. **属性访问时**
   * 使用 `get_property_xxx()` 方法获取属性时，会尝试将存储类型转换为请求类型
   * 这时可能触发安全或不安全转换

4. **消息传递时**
   * 当消息属性需要在不同类型之间转换时

### 错误处理

当类型转换失败时，TEN framework 会：

* 返回错误状态码和详细的错误信息
* 在错误对象中提供源类型、目标类型和失败原因
* 对于超出范围的转换，错误消息格式为："out of range of \[target\_type]"
* 对于不兼容的类型转换，错误消息格式为："unsupported conversion from `[source_type]` to `[target_type]`"

#### 常见错误示例

```cpp
// 范围溢出错误
int8_t small_value = cmd.get_property_int8("large_number");  // 如果 large_number = 200
// 错误：out of range of int8

// 类型不兼容错误
std::string str_value = cmd.get_property_string("integer_prop");
// 错误：unsupported conversion from `int32` to `string`

// 精度丢失错误
float float_value = cmd.get_property_float32("precise_double");  // 如果 precise_double 超出 float32 精度
// 错误：out of range of float32
```

## 最佳实践

1. **使用合适的类型**：根据数据的实际范围选择合适的整数和浮点数类型
2. **明确定义 schema**：在 manifest.json 中明确定义属性类型以避免类型转换歧义
3. **处理转换错误**：在代码中适当处理类型转换可能产生的错误
4. **注意 JSON 映射**：了解 JSON 数据到 TEN 类型的映射规则，避免意外的类型转换


# 开发流程 (/cn/docs/ten_framework/0.11.60/development/development_process)

基本上，通过使用 TEN framework 来完成所需的场景，有两种开发方法或视角：

1. 从小扩展到大场景
2. 从大场景到小扩展

**注意：** 无论您采取这两种开发方法中的哪一种，您都可以从头开始编写自己的扩展，或者从 TEN 云商店下载预构建的扩展。

在第一种方法中，**从小扩展到大场景**，开发过程通常遵循以下步骤：

1. 开发扩展。
2. 将扩展组装成图，并进一步创建agent，例如 [TEN-Agent](https://github.com/TEN-framework/TEN-Agent)。

在第二种方法中，**从大场景到小扩展**，开发过程通常遵循以下步骤：

1. 选择所需的agent模板，例如 [TEN-Agent](https://github.com/TEN-framework/TEN-Agent)。
2. 将所选agent模板中的某些扩展替换为您自己的扩展。

在第二种开发方法中，通过使用框架提供的扩展**独立测试**功能，您可以确保替换后的扩展行为与原扩展一致。在此过程中，开发人员只需了解与 TEN 扩展相关的概念，包括如何独立开发和测试 TEN 扩展。其他与 TEN 相关的概念（扩展开发和测试之外）则无需关注，因为这些概念已封装在所选的 TEN agent中。


# TEN 扩展开发完整指南 (/cn/docs/ten_framework/0.11.60/development/how_to_develop_with_ext)

TEN Framework 提供了丰富的扩展模板，帮助开发者快速创建扩展并完成从开发到测试的完整流程。本指南将通过实际操作演示，详细介绍如何使用 C++，Go，Python，Node.js 四种语言进行扩展开发的全流程。

## 开发前准备

### 环境要求

在开始扩展开发之前，请确保您的开发环境已正确配置。通过以下命令验证安装：

```bash
tman --version
```

正常情况下，您应该看到类似以下的版本信息输出：

```bash
TEN Framework version: <version>
```

> **重要提示**：请确保您使用的 `tman` 版本 >= 0.10.12，如果版本过低，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 安装最新版本。

### 开发流程概览

无论使用哪种编程语言，TEN 扩展开发都遵循以下标准化流程：

1. **项目创建** - 使用官方模板生成扩展项目骨架
2. **依赖安装** - 配置运行环境并安装必要的依赖包
3. **核心开发** - 实现扩展的业务逻辑和功能代码
4. **构建测试** - 编译项目（如需要）并执行单元测试
5. **调试优化** - 使用专业调试工具定位并解决问题

***

## C++ 扩展开发

C++ 扩展适用于对性能要求极高的应用场景，如实时音视频处理、高频数据计算、底层系统操作等。

### 创建项目

使用 TEN 官方提供的 C++ 扩展模板快速创建新项目：

```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```

项目创建成功后，您会得到以下完整的项目结构：

```bash
my_example_ext_cpp/
├── BUILD.gn              # GN 构建系统配置文件
├── manifest.json         # 扩展元数据和配置信息
├── property.json         # 扩展属性和参数配置
├── src/
│   └── main.cc           # 扩展核心实现代码
├── tests/
│   ├── basic.cc          # 基础功能测试用例
│   └── gtest_main.cc     # 测试框架程序入口
├── include/              # 头文件存放目录
├── tools/                # 辅助工具和脚本
└── .vscode/
    └── launch.json       # VSCode 调试配置文件
```

### 环境配置验证

进入项目目录并验证构建工具是否正常工作：

```bash
cd my_example_ext_cpp
tgn --version
```

> **期望输出**：

```bash
0.1.0
```

如果命令无法执行，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 检查 TEN Framework 开发环境是否正确安装。

### 依赖包安装

安装扩展运行时所需的全部依赖包：

```bash
tman install --standalone
```

安装成功后，您将看到类似以下的详细安装日志：

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done

🏆  Install successfully in 1 second
```

### 项目构建

TEN framework 提供了两种便捷的构建方式供您选择：

#### 方式一：手动分步构建

```bash
# 第一步：生成构建配置文件（启用独立测试模式）
tgn gen linux x64 debug -- ten_enable_standalone_test=true

# 第二步：执行项目构建
tgn build linux x64 debug
```

#### 方式二：使用 tman 快捷命令

```bash
tman run build
```

构建完成后，检查生成的可执行测试文件：

```bash
ls -la bin/
# 应该能看到：my_example_ext_cpp_test
```

### 运行测试

验证扩展功能是否正常工作：

#### 方式一：直接执行测试文件

```bash
./bin/my_example_ext_cpp_test
```

#### 方式二：使用 tman 统一命令

```bash
tman run test
```

测试执行成功时的输出示例：

```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN      ] Test.Basic
[       OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[  PASSED  ] 1 test.
```

> **成功标志**：当您看到 `[  PASSED  ] 1 test.` 时表示测试全部通过。

### 核心代码结构详解

#### 扩展主体实现（src/main.cc）

C++ 扩展的核心类需要继承自 `ten::extension_t` 基类，并实现完整的生命周期管理方法：

```cpp
class my_example_ext_cpp_t : public ten::extension_t {
 public:
  // 扩展初始化阶段 - 进行基础配置和资源预分配
  void on_init(ten::ten_env_t &ten_env) override;

  // 扩展启动阶段 - 正式开始处理业务逻辑
  void on_start(ten::ten_env_t &ten_env) override;

  // 命令处理器 - 处理来自其他扩展或应用的命令请求
  void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr<ten::cmd_t> cmd) override;

  // 数据处理器 - 处理通用数据流
  void on_data(ten::ten_env_t &ten_env, std::unique_ptr<ten::data_t> data) override;

  // 音频帧处理器 - 处理实时音频流数据
  void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::audio_frame_t> frame) override;

  // 视频帧处理器 - 处理实时视频流数据
  void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr<ten::video_frame_t> frame) override;

  // 扩展停止阶段 - 清理资源和优雅关闭
  void on_stop(ten::ten_env_t &ten_env) override;
};
```

#### 测试框架实现（tests/basic.cc）

使用 TEN 专用测试框架编写完整的单元测试：

```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    // 创建测试用的命令对象
    auto new_cmd = ten::cmd_t::create("foo");

    // 发送命令到扩展并验证响应结果
    ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
      // 在这里验证测试结果的正确性
    });
  }
};
```

### 调试环境配置

#### VSCode 集成调试

1. 安装 VSCode 的 **CodeLLDB** 扩展插件
2. 在源代码中设置断点
3. 选择 "standalone test (lldb, launch)" 调试配置
4. 按 F5 键启动调试会话

调试配置文件 `.vscode/launch.json` 的内容：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (lldb, launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

#### 命令行调试

使用经典的 GDB 调试器进行命令行调试：

```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt  # 查看完整的函数调用栈
```

***

## Go 扩展开发

Go 扩展在高性能和开发效率之间提供了很好的平衡，特别适合构建网络服务、并发处理、微服务架构等应用场景。

### 创建项目

使用 Go 扩展模板创建新项目：

```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```

项目创建成功后会显示：

```bash
🏆  Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```

完整的项目结构如下：

```bash
my_example_ext_go/
├── extension.go             # 扩展核心实现代码
├── go.mod                   # Go 模块依赖管理配置
├── manifest.json            # 扩展元数据信息
├── property.json            # 扩展属性配置
├── README.md                # 项目说明文档
├── tests/                   # 测试相关文件
│   ├── basic_tester.go      # 测试逻辑实现
│   ├── basic_tester_test.go # 测试用例定义
│   ├── main_test.go         # 测试程序入口
│   └── bin/start            # 测试启动脚本
└── .vscode/launch.json      # VSCode 调试配置
```

### 依赖包安装

安装项目运行所需的依赖包：

```bash
tman install --standalone
```

### 运行测试

验证扩展功能的正确性：

#### 方式一：使用启动脚本

```bash
./tests/bin/start
```

#### 方式二：使用 tman 命令

```bash
tman run test
```

测试成功执行的输出示例：

```bash
=== RUN   TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```

### 核心代码结构详解

#### 扩展实现（extension.go）

Go 扩展的核心实现结构：

```go
import (
    ten "ten_framework/ten_runtime"
)

type ExampleExtension struct {
    ten.DefaultExtension
}

// 扩展启动生命周期
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStart")
    tenEnv.OnStartDone()
}

// 命令处理逻辑
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
    tenEnv.LogDebug("OnCmd")
    cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
    tenEnv.ReturnResult(cmdResult, nil)
}

// 扩展停止生命周期
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
    tenEnv.LogDebug("OnStop")
    tenEnv.OnStopDone()
}
```

#### 测试框架（tests/basic\_tester.go）

Go 扩展的测试框架实现：

```go
type BasicExtensionTester struct {
    ten.DefaultExtensionTester
}

func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
    // 创建用于测试的命令对象
    cmdTest, _ := ten.NewCmd("test")

    // 发送命令并处理响应结果
    tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
        if err != nil {
            panic(err)
        }
        statusCode, _ := cr.GetStatusCode()
        if statusCode != ten.StatusCodeOk {
            panic(statusCode)
        }
        tenEnvTester.StopTest(nil)
    })

    tenEnvTester.OnStartDone()
}
```

### 开发环境优化

为了提升开发体验和调试便利性，建议在扩展根目录创建 `go.work` 工作空间文件：

```go
go 1.18

use (
    .
    .ten/app/ten_packages/system/ten_runtime_go/interface
)
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Go 官方扩展，然后使用以下调试配置：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (go, launch)",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}/tests/",
      "env": {
        "CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
        "LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
      },
      "args": ["-test.v"]
    }
  ]
}
```

***

## Python 扩展开发

Python 扩展具有最高的开发效率，特别适合快速原型开发、AI/ML 应用集成、复杂业务逻辑实现等场景。

### 创建项目

使用 Python 异步扩展模板创建项目：

```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```

完整的项目结构：

```bash
my_example_ext_python/
├── extension.py         # 扩展核心实现代码
├── addon.py             # 扩展插件注册入口
├── __init__.py          # Python 包初始化文件
├── requirements.txt     # Python 依赖包清单
├── manifest.json        # 扩展元数据配置
├── property.json        # 扩展属性配置
├── tests/               # 测试相关文件
│   ├── test_basic.py    # 基础测试用例
│   ├── conftest.py      # pytest 配置文件
│   └── bin/start        # 测试启动脚本
└── .vscode/launch.json  # VSCode 调试配置
```

### 依赖包安装

安装项目所需的 Python 依赖包：

```bash
tman install --standalone
```

### 运行测试

验证 Python 扩展的功能：

#### 方式一：使用启动脚本

```bash
./tests/bin/start
```

#### 方式二：使用 tman 命令

```bash
tman run test
```

测试成功执行的输出示例：

```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py .                                                                                [100%]
============================================ 1 passed in 0.04s =======================================
```

### 核心代码结构详解

#### 扩展实现（extension.py）

Python 扩展推荐使用现代异步编程模式，以获得更好的性能和并发处理能力：

```python
from ten_runtime import (
    AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
    Cmd, StatusCode, CmdResult, Data
)

class ExampleExtension(AsyncExtension):
    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_init")

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_start")
        # TODO: 在这里读取配置文件，初始化必要的资源

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_name = cmd.get_name()
        ten_env.log_debug(f"on_cmd name {cmd_name}")

        # TODO: 在这里实现具体的业务逻辑处理
        cmd_result = CmdResult.create(StatusCode.OK, cmd)
        await ten_env.return_result(cmd_result)

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        ten_env.log_debug("on_stop")
        # TODO: 在这里清理资源，进行优雅关闭
```

#### 插件注册入口（addon.py）

扩展插件的注册和创建逻辑：

```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension

@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
    def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
        ten_env.log_info("on_create_instance")
        ten_env.on_create_instance_done(ExampleExtension(name), context)
```

#### 测试实现（tests/test\_basic.py）

完整的异步测试框架实现：

```python
from ten_runtime import (
    AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
    TenError, TenErrorCode
)

class ExtensionTesterBasic(AsyncExtensionTester):
    async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
        # 创建用于测试的命令对象
        new_cmd = Cmd.create("hello_world")

        ten_env.log_debug("send hello_world")
        result, err = await ten_env.send_cmd(new_cmd)

        # 验证测试结果的正确性
        if (err is not None or result is None
            or result.get_status_code() != StatusCode.OK):
            ten_env.stop_test(TenError.create(
                TenErrorCode.ErrorCodeGeneric,
                "Failed to send hello_world"
            ))
        else:
            ten_env.stop_test()

def test_basic():
    tester = ExtensionTesterBasic()
    tester.set_test_mode_single("my_example_ext_python")
    err = tester.run()
    if err is not None:
        assert False, err.error_message()
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Python 扩展和 debugpy 调试器，使用以下配置进行调试：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (debugpy, launch)",
      "type": "debugpy",
      "request": "launch",
      "python": "/usr/bin/python3",
      "module": "pytest",
      "args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
      "env": {
        "TEN_ENABLE_PYTHON_DEBUG": "true",
        "PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
      },
      "console": "integratedTerminal"
    }
  ]
}
```

***

## Node.js 扩展开发

Node.js 扩展提供了现代 JavaScript/TypeScript 开发体验，特别适合 Web 应用集成、快速原型开发、前端技术栈扩展等场景。得益于 Node.js 的异步特性和丰富的生态系统，开发者可以轻松构建高效的实时应用。

### 创建项目

使用 TEN 官方提供的 Node.js 扩展模板快速创建新项目：

```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```

项目创建成功后，您会得到以下完整的项目结构：

```bash
my_example_ext_nodejs/
├── manifest.json         # 扩展元数据和配置信息
├── property.json         # 扩展属性和参数配置
├── package.json          # Node.js 依赖包管理配置
├── tsconfig.json         # TypeScript 编译器配置
├── src/
│   └── index.ts          # 扩展核心实现代码
├── tests/                # 测试相关文件
│   ├── src/
│   │   ├── index.ts      # 测试器实现
│   │   ├── index.spec.ts # 测试用例定义
│   │   └── main.spec.ts  # 测试框架配置
│   ├── bin/start         # 测试启动脚本
│   ├── package.json      # 测试依赖配置
│   └── tsconfig.json     # 测试 TypeScript 配置
├── tools/                # 辅助工具和脚本
└── .vscode/
    └── launch.json       # VSCode 调试配置文件
```

### 环境配置验证

进入项目目录并验证开发环境：

```bash
cd my_example_ext_nodejs
node --version
npm --version
```

> **期望输出**：

```bash
v18.0.0 或更高版本
8.0.0 或更高版本
```

### 依赖包安装

安装扩展运行时所需的全部依赖包：

```bash
tman install --standalone
```

安装成功后，您将看到类似以下的详细安装日志：

```bash
📦  Get all installed packages...
🔍  Filter compatible packages...
🔒  Creating manifest-lock.json...
📥  Installing packages...
  [00:00:00] [########################################]       2/2       Done

🏆  Install successfully in 1 second
```

> **重要提示**：`tman install --standalone` 会在项目目录下创建 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录，后续的构建和测试操作都需要在这个目录下进行。

### 项目构建

Node.js 扩展使用 TypeScript 进行开发，需要先安装独立模式依赖，然后编译为 JavaScript：

#### 方式一：手动构建

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 安装独立模式的 Node.js 依赖
npm run standalone-install

# 编译 TypeScript 代码
npm run build
```

#### 方式二：使用 tman 快捷命令

```bash
tman run build
```

构建完成后，检查生成的编译结果：

```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# 应该能看到：index.js 和相关映射文件
```

### 运行测试

验证扩展功能是否正常工作：

#### 方式一：使用启动脚本

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 运行测试
./tests/bin/start
```

#### 方式二：使用 tman 统一命令

```bash
tman run test
```

测试执行成功时的输出示例：

```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
  ✓ case2 (1010ms)

  2 passing (2s)
```

> **成功标志**：当您看到所有测试用例显示 `✓` 并且最后显示 `passing` 时表示测试全部通过。

### 核心代码结构详解

#### 扩展主体实现（src/index.ts）

Node.js 扩展的核心类需要继承自 `Extension` 基类，并实现完整的生命周期管理方法：

```typescript
import {
  Addon,
  RegisterAddonAsExtension,
  Extension,
  TenEnv,
  Cmd,
  CmdResult,
  StatusCode,
} from "ten-runtime-nodejs";

class DefaultExtension extends Extension {
  constructor(name: string) {
    super(name);
  }

  // 扩展配置阶段 - 进行配置参数的读取和验证
  async onConfigure(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onConfigure");
  }

  // 扩展初始化阶段 - 进行基础配置和资源预分配
  async onInit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onInit");
  }

  // 扩展启动阶段 - 正式开始处理业务逻辑
  async onStart(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStart");
  }

  // 命令处理器 - 处理来自其他扩展或应用的命令请求
  async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
    console.log("DefaultExtension onCmd", cmd.getName());

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("detail", "This is a demo");
    tenEnv.returnResult(cmdResult);
  }

  // 扩展停止阶段 - 清理资源和优雅关闭
  async onStop(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onStop");
  }

  // 扩展销毁阶段 - 最终清理和资源释放
  async onDeinit(_tenEnv: TenEnv): Promise<void> {
    console.log("DefaultExtension onDeinit");
  }
}
```

#### 插件注册入口

扩展插件的注册和创建逻辑：

```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
  async onCreateInstance(
    _tenEnv: TenEnv,
    instanceName: string,
  ): Promise<Extension> {
    return new DefaultExtension(instanceName);
  }
}
```

#### 测试框架实现（tests/src/index.ts）

使用 TEN 专用测试框架编写完整的单元测试：

```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";

export class MyExtensionTester extends ExtensionTester {
  async onStart(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStart");

    // 模拟异步操作和测试逻辑
    new Promise((resolve) => {
      setTimeout(() => {
        resolve(true);
      }, 1000);
    }).then(() => {
      // 停止测试并返回结果
      tenEnvTester.stopTest();
    });
  }

  async onStop(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onStop");
  }

  async onDeinit(tenEnvTester: TenEnvTester) {
    console.log("MyExtensionTester onDeinit");
  }
}
```

#### 测试用例定义（tests/src/index.spec.ts）

使用 Mocha 测试框架编写具体的测试用例：

```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";

const test_addon_name = "default_extension_nodejs";

describe("MyExtensionTester", () => {
  it("case1", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });

  it("case2", async () => {
    const extensionTester = new MyExtensionTester();
    extensionTester.setTestModeSingle(test_addon_name, "{}");
    const result = await extensionTester.run();
    assert(result === null, "result should be null");

    console.log("deinit done");
  });
});
```

### TypeScript 配置

Node.js 扩展使用现代 TypeScript 配置，支持最新的语言特性：

```json
{
  "compilerOptions": {
    "allowJs": false,
    "composite": true,
    "module": "NodeNext",
    "target": "es6",
    "moduleResolution": "NodeNext",
    "outDir": "build",
    "removeComments": false,
    "sourceMap": true,
    "noImplicitAny": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "strictNullChecks": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "rootDir": "src",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
  },
  "exclude": [
    "node_modules"
  ],
  "include": [
    "src/**/*"
  ]
}
```

### 调试环境配置

#### VSCode 集成调试

确保已安装 Node.js 官方扩展，然后使用以下调试配置：

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "standalone test (nodejs) (mocha, launch)",
      "type": "node",
      "request": "launch",
      "program": "node_modules/mocha/bin/_mocha",
      "stopOnEntry": true,
      "args": [
        "--no-timeouts",
        "--package",
        "package.json",
      ],
      "cwd": "${workspaceFolder}/tests",
      "env": {
        "NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
      },
      "runtimeArgs": [
        "--expose-gc",
        "--loader",
        "ts-node/esm",
        "--no-warnings",
      ]
    }
  ]
}
```

#### 命令行调试

使用 Node.js 内置调试器进行命令行调试：

```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 启动调试模式
node --inspect-brk ./tests/bin/start

# 或者使用 Chrome DevTools
node --inspect ./tests/bin/start
```

### 完整开发流程总结

为了帮助开发者快速上手，这里提供一个完整的 Node.js 扩展开发流程总结：

```bash
# 1. 创建扩展项目
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example

# 2. 进入项目目录
cd my_example_ext_nodejs

# 3. 安装依赖
tman install --standalone

# 4. 进入扩展安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs

# 5. 安装独立模式依赖
npm run standalone-install

# 6. 构建项目
npm run build

# 7. 运行测试
./tests/bin/start
```

> **工作目录说明**：
>
> * 扩展源代码位于项目根目录的 `src/` 文件夹
> * 实际的构建、测试和运行操作都在 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录下进行
> * 这种设计确保了扩展的独立性和依赖管理的正确性

### 开发最佳实践

#### 异步编程

Node.js 扩展充分利用异步编程模式，提供更好的性能：

```typescript
// 推荐：使用 async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    const result = await processCommand(cmd);
    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("result", result);
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

#### 错误处理

实现完善的错误处理机制：

```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise<void> {
  try {
    // 业务逻辑处理
    const result = await this.handleBusinessLogic(cmd);

    const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
    cmdResult.setPropertyString("data", JSON.stringify(result));
    tenEnv.returnResult(cmdResult);
  } catch (error) {
    console.error("Command processing error:", error);

    const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
    cmdResult.setPropertyString("error", error.message);
    tenEnv.returnResult(cmdResult);
  }
}
```

***

## 开发总结

通过遵循本指南提供的完整开发流程，您可以高效地进行 TEN 扩展的开发、测试和调试工作。无论选择 C++，Go，Python 还是 Node.js，TEN Framework 都为您提供了完整的工具链和最佳实践，帮助您充分发挥 TEN Framework 的强大功能，构建出高性能、高可靠性的扩展应用。

每种语言都有其独特的优势和适用场景：

* **C++**：适用于对性能要求极高的场景，如实时音视频处理、高频计算
* **Go**：在高性能和开发效率之间提供平衡，适合网络服务、并发处理
* **Python**：具有最高的开发效率，特别适合AI/ML应用、快速原型开发
* **Node.js**：提供现代 Web 开发体验，适用于前端技术栈扩展、实时应用

请根据您的具体需求和团队技术栈选择最合适的开发方案。在开发过程中，建议充分利用 TEN Framework 提供的调试工具和测试框架，以确保扩展的质量和稳定性。


# TEN Framework 快速入门指南 (/cn/docs/ten_framework/0.11.60/getting-started/quick-start)

## TEN Framework 快速入门指南

> 🎯 **目标**：5分钟内搭建开发环境并运行第一个 TEN 应用

## 系统要求

**支持的操作系统**：

* Linux (x64)
* Linux (arm64)
* macOS Intel (x64)
* macOS Apple Silicon (arm64)
* Windows (x64)

**必需的软件环境**：

* Python 3.10
* Go 1.20+
* Node.js / npm（用于安装和管理 JavaScript 依赖）

> 💡 **Windows 用户注意**：推荐使用 PowerShell 执行本指南中的命令。部分命令在 CMD 中可能不兼容。

## 第一步：检查环境

在开始之前，请确保你的系统已安装以下软件：

### Python 3.10

```bash
python3 --version
# 应显示: Python 3.10.x
```

> 💡 **重要**：TEN Framework 目前仅支持 Python 3.10。推荐使用 `pyenv` 或 `venv` 创建 Python 虚拟环境：
>
> **Linux / macOS：**
>
> ```bash
> # 使用 pyenv 安装和管理 Python 3.10（推荐）
> pyenv install 3.10.18
> pyenv local 3.10.18
>
> # 或使用 venv 创建虚拟环境
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows：**
>
> 从 [https://www.python.org/downloads/release/python-31011/](https://www.python.org/downloads/release/python-31011/) 底部表格中下载 Windows installer，启动后安装 Python 3.10。
>
> 注意，点击Install Now前务必勾选 "Add Python to PATH"
>
> ```bash
> # Windows平台需要配置python3命令：
>
> # 首先，确认python的安装路径：
> where.exe python
> # 输出示例: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # 然后，以管理员身份打开 PowerShell，在python.exe同目录下创建symlink：
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # 请将上面的路径替换为 where.exe python 的实际输出路径
>
> # 验证：
> python3 --version
> ```
>
> ```powershell
> # 推荐安装后使用 venv 创建虚拟环境，在该环境中工作
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # 激活环境
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # 若有权限错误，关闭终端/IDE，右键选择“以管理员身份运行”重新打开
> # 或者改变执行策略来允许ps1脚本执行
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```

### Go 1.20+

```bash
go version
# 应显示: go version go1.20 或更高版本

$env:CGO_ENABLED = "1"
# Windows下需要显式启用CGO
```

### Node.js / npm

```bash
node --version
npm --version
# 确保 node 和 npm 命令可用
```

### GCC (MinGW)

💡仅在windows上需要安装

```bash
# 首先确定winget存在。
winget --version

# 若不存在则需要从 <https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US> 安装。
#（系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11）

#安装MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT

# 或查找后自行选择合适版本安装
winget search "mingw"

#检查安装
gcc --version
```

> 💡 **提示**：如果缺少上述环境，请先安装对应版本后再继续。

## 第二步：安装 TEN Manager (tman)

TEN Manager (tman) 是 TEN Framework 的命令行工具，用于创建项目、管理依赖和运行应用。

方式一：通过包管理器安装（推荐）

**Linux (Ubuntu/Debian):**

```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```

**macOS:**

```bash
brew install TEN-framework/ten-framework/tman
```

**Windows:**

```powershell
winget install TEN-framework.tman
```

方式二：通过安装脚本

**Linux / macOS：**

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```

**Windows：**

```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```

方式三：如果你已经克隆了仓库，使用仓库内的安装脚本

```bash
cd ten-framework

# Linux/MacOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```

> 💡 **提示**：如果系统中已经安装了 tman，安装脚本会询问是否重新安装/升级，输入 `y` 继续安装，输入 `n` 取消。
>
> **非交互式安装**（适用于自动化脚本或 CI 环境，Windows不可用）：
>
> ```bash
> # 远程安装
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # 本地安装
> yes y | bash tools/tman/install_tman.sh
> ```

**验证安装**：

```bash
tman --version
```

> 💡 **提示**：如果出现类似
>
> * `tman: command not found`（Linux/macOS）
> * `无法将"tman"项识别为 cmdlet、函数、脚本文件或可运行程序的名称`（Windows）
>
>   的提示，请确保 tman 所在目录在你的 PATH 中：
>
> **Linux / macOS：**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc  # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc   # macOS
> source ~/.bashrc  # 或 source ~/.zshrc
> ```
>
> **Windows：**
>
> ```powershell
> # 检查 PATH 中是否包含 tman
> $env:Path -split ";" | Select-String "tman"
> # 如果不包含，请
> # 1. 通过 "系统属性 → 环境变量" 手动添加，然后重启终端以应用
> # 2. 运行以下命令来临时添加路径
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```

## 第三步：创建并运行示例应用

<div className="fd-steps">
  <div className="fd-step">
    ### 创建应用

    ```bash
    # 创建一个新的 transcriber_demo 应用
    tman install app transcriber_demo
    cd transcriber_demo
    ```
  </div>

  <div className="fd-step">
    ### 安装依赖

    ```bash
    # 安装 TEN 包依赖
    tman install

    # 安装 Python 和 npm 包的依赖
    tman run install_deps
    ```

    > ⏱️ **预计时间**：1-2 分钟
  </div>

  <div className="fd-step">
    ### 构建应用

    ```bash
    tman run build
    ```

    > ⏱️ **预计时间**：30 秒
  </div>

  <div className="fd-step">
    ### 配置环境变量

    在运行应用前，需要配置 ASR（语音识别）服务的密钥。当前示例使用 Azure ASR extension，你需要在 `transcriber_demo/.env` 文件中填写相关配置：

    **Linux / macOS：**

    ```bash
    # 创建 .env 文件
    cat > .env << EOF
    # Azure Speech Service 配置
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region      # 例如：eastus
    AZURE_STT_LANGUAGE=en-US                # 根据你的音频语种或实时录音语种设置，如：zh-CN, ja-JP, ko-KR 等
    EOF
    ```

    **Windows（PowerShell）：**

    ```powershell
    # 创建 .env 文件
    @"
    # Azure Speech Service 配置
    AZURE_STT_KEY=your_azure_speech_api_key
    AZURE_STT_REGION=your_azure_region
    AZURE_STT_LANGUAGE=en-US
    "@ | Out-File -Encoding utf8 .env
    ```

    > 💡 **提示**：Windows 用户也可以直接用文本编辑器创建 `.env` 文件并填写上述配置。
    >
    > 💡 **提示**：如果你想使用其他 ASR extension（如 OpenAI Whisper、Google Speech 等），可以从云商店下载并替换，同样将相应的 API key 等环境变量配置在 `.env` 文件中。
  </div>

  <div className="fd-step">
    ### 运行应用

    ```bash
    tman run start
    ```

    如果一切正常，你应该看到类似以下的输出：

    ```text
    [web_audio_control_go] Web server started on port 8001
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```
  </div>

  <div className="fd-step">
    ### 体验 Demo

    打开浏览器访问：

    ```text
    http://localhost:8001
    ```

    你应该能看到 Transcriber Demo 的 Web 界面，可以尝试：

    * 点击麦克风按钮进行实时语音转录
    * 上传音频文件进行转录
    * 查看实时转录以及字幕结果
  </div>
</div>

## 恭喜！🎉

你已经成功运行了第一个 TEN 应用！

### 了解应用架构

这个 `transcriber_demo` 应用展示了 TEN Framework 的多语言扩展能力，它由以下组件构成：

* **Go** - WebSocket 服务器扩展 (`web_audio_control_go`)
* **Python** - ASR 语音识别扩展 (`azure_asr_python`)
* **TypeScript** - VTT 字幕生成和音频录制扩展 (`vtt_nodejs`)

🎯 **你已经可以运行这些多语言插件了！**

### 下一步

现在你可以：

1. **从云商店探索和下载更多插件，设计和编排你的应用**

   ```bash
   tman designer  # 启动 TMAN Designer，在云商店中探索插件、下载插件并设计编排你的应用
   ```

2. **选择一个语言，开发自己的插件**
   * 支持 Go、Python、TypeScript/JavaScript、C++ 等多种语言
   * 查看 [TEN 扩展开发完整指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) 了解详情

## 进阶：开发和构建 C++ 插件

如果你想开发和使用 C++ 扩展，推荐安装 TEN 构建工具链（tgn）。以下是完整的步骤：

<div className="fd-steps">
  <div className="fd-step">
    ### 安装 tgn 构建工具

    tgn 是 TEN Framework 的 C/C++ 构建系统，基于 Google 的 GN。

    方式一：一键安装（推荐）

    **Linux / macOS：**

    ```bash
    curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
    ```

    **Windows：**

    ```powershell
    irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
    ```

    方式二：从克隆的仓库安装

    **Linux / macOS：**

    ```bash
    # 如果你已经克隆了 TEN Framework 仓库
    cd ten-framework
    bash tools/tgn/install_tgn.sh
    ```

    **Windows：**

    ```powershell
    # 如果你已经克隆了 TEN Framework 仓库
    & ".\tools\tgn\install_tgn.ps1"
    ```

    安装完成后，确认 tgn 已添加到 PATH：

    **Linux / macOS：**

    ```bash
    # 临时添加到当前会话
    export PATH="/usr/local/ten_gn:$PATH"

    # 或永久添加到 shell 配置（推荐）
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc  # Linux
    echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc   # macOS
    source ~/.bashrc  # 或 source ~/.zshrc
    ```

    **Windows：**

    ```powershell
    # 安装脚本会自动添加到用户 PATH，如需手动添加：
    $env:Path += ";$env:LOCALAPPDATA\ten_gn"
    ```

    验证安装：

    ```bash
    tgn --help
    ```
  </div>

  <div className="fd-step">
    ### 安装 C++ 扩展

    以 WebRTC VAD（语音活动检测）扩展为例，从云商店安装 C++ 扩展：

    ```bash
    cd transcriber_demo
    tman install extension webrtc_vad_cpp
    ```

    > 💡 **提示**：`webrtc_vad_cpp` 是一个用 C++ 实现的语音活动检测扩展，可以在实时语音识别场景中筛选出语音部分。
  </div>

  <div className="fd-step">
    ### 编译 C++ 扩展

    安装 C++ 扩展后，需要重新构建应用以编译 C++ 代码为动态库：

    ```bash
    tman run build
    ```

    > ⏱️ **预计时间**：首次编译 C++ 扩展可能需要 1-3 分钟，具体取决于你的机器性能。
  </div>

  <div className="fd-step">
    ### 运行带有 VAD 功能的应用

    ```bash
    tman run start_with_vad
    ```

    如果一切正常，你应该看到：

    ```text
    [web_audio_control_go] Web server started on port 8001
    [vad] WebRTC VAD initialized with mode 2
    [audio_file_player_python] AudioFilePlayerExtension on_start
    ```

    现在打开浏览器访问 `http://localhost:8001`，进入麦克风实时转写页面，你会看到经过vad后的silence状态变化，当silence状态为true时，表示当前音频中没有语音。
  </div>
</div>

### C++ 开发环境要求

开发和编译 C++ 扩展需要安装 C++ 编译器（gcc 或 clang）：

**Linux:**

```bash
# Ubuntu/Debian
sudo apt-get install gcc g++

# 或使用 clang
sudo apt-get install clang
```

**macOS:**

```bash
# 安装 Xcode Command Line Tools (包含 clang)
xcode-select --install
```

**Windows:**

```powershell
# 方式一：安装 Visual Studio Build Tools（推荐）
# 从 https://visualstudio.microsoft.com/visual-cpp-build-tools/ 下载安装

# 方式二：使用 winget 安装
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

# 💡注意：
#  - 安装时选择 "使用 C++ 的桌面开发" 工作负载
#  - 且务必勾选“适用于Windows的C++ Clang工具”
```

验证编译器安装：

```bash
# 检查 gcc（Linux/macOS/MSYS2）
gcc --version
g++ --version

# 或检查 clang（Linux/macOS）
clang --version
```

**Windows（使用 MSVC）：**

```powershell
# 在 "Developer PowerShell for VS" 中执行
clang-cl --version
```

### 常见问题（C++ 扩展）

1. tgn 命令找不到

   确保已经执行安装脚本并将 tgn 添加到 PATH：

   ```bash
   export PATH="/usr/local/ten_gn:$PATH"
   ```

2. 编译失败：找不到编译器

   请参考上面的"C++ 开发环境要求"部分安装编译器。

### 了解更多

* [ten\_gn 构建系统](https://github.com/TEN-framework/ten_gn) - TEN 的 C/C++ 构建工具
* [C++ 扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) - 完整的 C++ 扩展开发文档

## 常见问题

<div className="fd-steps">
  <div className="fd-step">
    ### macOS 上 Python 库加载失败

    **问题**：运行应用时提示找不到 `libpython3.10.dylib`

    **解决方案**：

    ```bash
    export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
    ```

    建议将这行添加到 `~/.zshrc` 或 `~/.bash_profile` 中。
  </div>

  <div className="fd-step">
    ### tman 下载失败或速度很慢

    **问题**：网络连接 GitHub 受限

    **解决方案**：

    * 手动下载：访问 [Releases 页面](https://github.com/TEN-framework/ten-framework/releases) 下载对应平台的 `tman` 二进制文件
  </div>

  <div className="fd-step">
    ### 端口 8001 已被占用

    **问题**：启动时提示端口冲突

    **解决方案**：

    * 查找占用端口的进程：
      * Linux/macOS：`lsof -i :8001`
      * Windows：`netstat -ano | findstr :8001`
    * 杀掉该进程：
      * Linux/macOS：`kill -9 <PID>`
      * Windows：`taskkill /PID <PID> /F`
    * 或修改应用配置文件（`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`）中的端口号
  </div>

  <div className="fd-step">
    ### Go build 失败

    **问题**：构建时提示 Go module 相关错误

    **解决方案**：

    ```bash
    # 设置代理
    # Linux/macOS
    export GOPROXY=https://goproxy.cn,direct
    # Windows PowerShell
    $env:GOPROXY = "https://goproxy.cn,direct"

    # 清理 Go module 缓存
    go clean -modcache

    # 重新安装依赖
    cd transcriber_demo
    tman run build
    ```
  </div>

  <div className="fd-step">
    ### Python 依赖安装失败

    **问题**：pip 安装超时或失败

    **解决方案**：使用国内镜像源

    ```bash
    pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
    ```
  </div>

  <div className="fd-step">
    ### Azure语音服务报错

    **问题**：Azure语音服务相关错误，比如认证失败

    **解决方案**：检查.env文件中的配置是否正确，确保 AZURE\_STT\_KEY 和 AZURE\_STT\_REGION 填写无误
  </div>

  <div className="fd-step">
    ### Windows下权限问题

    **问题**：Windows下在访问文件时报PermissionError

    **解决方案**：右键点击PowerShell，选择“以管理员身份运行”
  </div>
</div>

## 获取帮助

* **GitHub Issues**：[https://github.com/TEN-framework/ten-framework/issues](https://github.com/TEN-framework/ten-framework/issues)
* **文档**：[https://theten.ai/cn/docs](https://theten.ai/cn/docs)
* **贡献指南**：[contributing.md](../code-of-conduct/contributing.md)


# 基于WebSocket的语音助手快速入门 (/cn/docs/ten_framework/0.11.60/getting-started/websocket-voice-assistant-quick-start)

## 基于 WebSocket 的语音助手快速入门

在不到 10 分钟内构建并运行一个基于 WebSocket 的实时语音助手。

## 简介

本指南将帮助您快速运行一个完整的语音助手 Demo，支持实时语音对话和工具调用。该示例集成了：

| 组件   | 服务商        | 用途       |
| ---- | ---------- | -------- |
| ASR  | Deepgram   | 语音转文字    |
| LLM  | OpenAI     | 对话生成     |
| TTS  | ElevenLabs | 文字转语音    |
| Tool | WeatherAPI | 天气查询（可选） |

## 环境要求

### 前置依赖

请先参考 [TEN Framework 快速入门指南](./quick-start.cn.md) 完成基础环境安装，确保以下命令通过检查：

```bash
tman check env
```

预期输出（至少 Python 和 Go 为 Ready）：

```text
✅ Operating System: Supported
✅ Python:   Ready
✅ Go:       Ready
```

### 额外工具

安装以下工具用于构建和运行：

#### Task（任务运行器）

**Linux / macOS：**

```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version  # 验证安装
```

**Windows：**

```powershell
# 1. 使用winget安装（推荐）
# 从以下地址安装winget
# https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# 系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11
winget --version # 验证安装
winget install Task.Task
task --version  # 验证安装

# 2. 使用scoop安装
irm get.scoop.sh | iex
scoop --version # 验证安装
scoop install task
task --version # 验证安装
```

#### Bun（JavaScript 包管理器）

**Linux / macOS：**

```bash
curl -fsSL https://bun.sh/install | bash
bun --version  # 验证安装
```

**Windows：**

```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version  # 验证安装
```

#### uv（Python 包管理器）

**Linux / macOS：**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version  # 验证安装
```

**Windows：**

```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version  # 验证安装
```

> 💡 安装后需要重新加载终端配置：
>
> * Linux/macOS：`source ~/.zshrc`
> * Windows：重新打开 PowerShell 终端

## 快速开始

<div className="fd-steps">
  <div className="fd-step">
    ### 克隆仓库

    ```bash
    git clone https://github.com/TEN-framework/ten-framework.git
    cd ten-framework/ai_agents/agents/examples/websocket-example
    ```
  </div>

  <div className="fd-step">
    ### 安装依赖

    ```bash
    task install
    ```

    > ⏱️ 预计耗时：2-3 分钟

    该命令会自动完成：

    * 安装 TEN 扩展包（45+ 个）
    * 编译 Go 应用和 API 服务器
    * 安装 Python 和前端依赖
  </div>

  <div className="fd-step">
    ### 配置 API Keys

    在 `ai_agents/.env` 文件中配置服务商密钥：

    **Linux / macOS：**

    ```bash
    cd ../../../  # 回到 ai_agents 目录
    vim .env
    ```

    **Windows：**

    ```powershell
    cd ..\..\..   # 回到 ai_agents 目录
    notepad .env
    ```

    从.env.example复制所有内容后，填写以下配置：

    ```bash
    # Deepgram - 语音识别
    DEEPGRAM_API_KEY=your_deepgram_api_key

    # OpenAI - 大语言模型
    OPENAI_API_KEY=your_openai_api_key
    OPENAI_MODEL=your_openai_model

    # ElevenLabs - 语音合成
    ELEVENLABS_TTS_KEY=your_elevenlabs_api_key

    # WeatherAPI - 天气工具（可选）
    WEATHERAPI_API_KEY=your_weatherapi_key
    ```
  </div>

  <div className="fd-step">
    ### 启动服务

    ```bash
    cd agents/examples/websocket-example
    task run
    ```

    服务启动后，自动运行：

    * **前端 UI**：[http://localhost:3000](http://localhost:3000)
    * **API 服务器**：[http://localhost:8080](http://localhost:8080)
    * **TMAN Designer**：[http://localhost:49483](http://localhost:49483)
  </div>

  <div className="fd-step">
    ### 与助手对话

    打开浏览器访问 [http://localhost:3000](http://localhost:3000)，点击麦克风按钮开始语音对话。

    尝试以下对话：

    * "你好，你是谁？"
    * "今天北京的天气怎么样？"（需配置 WeatherAPI）
    * "给我讲个笑话"

    **🎉 恭喜！** 您已成功运行 TEN Agent 语音助手。
  </div>
</div>

## 进阶配置

### 更换扩展

TEN Framework 支持灵活替换 ASR、LLM、TTS 等扩展。

#### 方式 1：使用 TMAN Designer（推荐）

访问 [http://localhost:49483](http://localhost:49483)，通过可视化界面：

1. 点击扩展节点查看详情
2. 选择替代扩展
3. 配置参数
4. 保存并重启

#### 方式 2：编辑配置文件

编辑 `tenapp/property.json`，修改 `predefined_graphs` 中的扩展配置。

**示例：更换 ASR 为 Azure**

```json
{
  "type": "extension",
  "name": "stt",
  "addon": "azure_asr_python",
  "extension_group": "stt",
  "property": {
    "params": {
      "key": "${env:AZURE_STT_KEY}",
      "region": "${env:AZURE_STT_REGION}",
      "language": "en-US"
    }
  }
}
```

**示例：更换 TTS 为 Minimax**

```json
{
  "type": "extension",
  "name": "tts",
  "addon": "minimax_tts_websocket_python",
  "extension_group": "tts",
  "property": {
    "params": {
      "api_key": "${env:MINIMAX_TTS_API_KEY|}",
      "group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
      "model": "speech-02-turbo",
      "audio_setting": {
        "sample_rate": 16000
      },
      "voice_setting": {
        "voice_id": "female-shaonv"
      }
    }
  }
}
```

修改后重启服务（`Ctrl+C` 然后 `task run`）。

### 配置监控

使用 Grafana 监控应用的 Metrics 和 Logs。

<div className="fd-steps">
  <div className="fd-step">
    #### 启动监控栈

    **Linux / macOS：**

    ```bash
    cd tools/grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    ```

    **Windows：**

    ```powershell
    cd tools\grafana-monitoring
    docker compose -f docker-compose.push.yml up -d
    # 若docker未安装，请通过https://apps.microsoft.com/detail/xp8cbj40xlbwkx 安装，并重启电脑后，启动docker desktop
    ```

    启动的服务：

    * **Grafana**（端口 3001）- 可视化界面
    * **Prometheus**（端口 9091）- Metrics 存储
    * **Loki**（端口 3100）- Logs 存储
    * **OpenTelemetry Collector**（端口 4317/4318）- OTLP 数据接收
  </div>

  <div className="fd-step">
    #### 配置应用遥测

    编辑 `agents/examples/websocket-example/tenapp/property.json`：

    **配置 Logs 输出**

    在 `ten.log.handlers` 数组中添加：

    ```json
    {
      "matchers": [{ "level": "info" }],
      "formatter": { "type": "json", "colored": false },
      "emitter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
    ```
  </div>
</div>

#### 配置 Metrics 输出

在 `ten.services` 中添加：

```json
{
  "telemetry": {
    "enabled": true,
    "metrics": {
      "enabled": true,
      "exporter": {
        "type": "otlp",
        "config": {
          "endpoint": "http://localhost:4317",
          "service_name": "websocket-example"
        }
      }
    }
  }
}
```

<div className="fd-steps">
  <div className="fd-step">
    #### 重启并访问 Grafana

    ```bash
    cd ../../../agents/examples/websocket-example
    task run
    ```

    访问 [http://localhost:3001](http://localhost:3001)（用户名/密码：`admin/admin`）
  </div>
</div>

#### 查看 Metrics

进入 `Dashboards` → **"TEN Framework"** 查看：

* 扩展启动耗时
* Cmd 处理延迟
* 消息队列等待时间

#### 查看 Logs

进入 `Explore` → 选择 `Loki`，使用查询：

* `{service_name="websocket-example"}` - 所有日志
* `{service_name="websocket-example", ten_extension_name="stt"}` - ASR 日志
* `{service_name="websocket-example", ten_extension_name="llm"}` - LLM 日志
* `{service_name="websocket-example", ten_extension_name="tts"}` - TTS 日志

> 💡 使用 `ten_extension_name` 标签精确筛选扩展日志。

停止监控服务：

**Linux / macOS：**

```bash
cd tools/grafana-monitoring
docker compose -f docker-compose.push.yml down
```

**Windows：**

```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```

## 常见问题

<details>
<summary><strong>前端依赖安装失败</strong></summary>

如果 `bun install` 报错版本不存在，切换到 npm 官方源：

```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```

</details>

<details>
<summary><strong>macOS Python 库加载失败</strong></summary>

设置 Python 动态库路径：

```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# 添加到 ~/.zshrc 永久生效
```

</details>

<details>
<summary><strong>Windows Python 库加载失败</strong></summary>

确保 Python 3.10 安装路径在系统 PATH 中：

```powershell
# 检查 Python 路径
python -c "import sys; print(sys.prefix)"

# 将 Python DLL 目录添加到 PATH（根据实际路径调整）
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```

如果安装 Python 时未勾选 "Add Python to PATH"，可以重新运行安装程序选择 "Modify" 来添加。

</details>

<details>
<summary><strong>端口被占用</strong></summary>

查找并终止占用端口的进程：

**Linux / macOS：**

```bash
lsof -i :3000  # 或 :8080
kill -9 <PID>
```

**Windows：**

```powershell
netstat -ano | findstr :3000   # 或 :8080
taskkill /PID <PID> /F
```

</details>

<details>
<summary><strong>Go module相关错误</strong></summary>

```bash
# 设置代理
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"

# 清理 Go module 缓存
go clean -modcache

# 重新安装依赖
cd ai_agents/agents/examples/websocket-example
task install
```

</details>

## 下一步

跟随这些指南，将您的语音 AI 应用带入真实世界。

* **[扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext)** - 开发自定义扩展
* **[更多示例](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - 探索其他 Agent 示例
* **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - 报告问题或请求功能
* **[文档中心](https://theten.ai/cn/docs)** - 完整的 TEN Framework 文档


# 图（Graph） (/cn/docs/ten_framework/0.11.60/graph/graph)

在 TEN framework 中，存在两种类型的图：

1. 动态图
2. 预定义图（`predefined_graph`）

|       | 动态图                          | 预定义图                                      |
| ----- | ---------------------------- | ----------------------------------------- |
| 启动时机  | 当 TEN 应用收到 `start_graph` 命令时 | 当 TEN 应用启动时，或当 TEN 应用收到 `start_graph` 命令时 |
| 图的内容  | 在 `start_graph` 命令中指定        | 在 TEN 应用的属性中预先定义                          |
| 图的 ID | 随机生成的 UUID                   | 随机生成的 UUID                                |

![两种类型的图](https://ten-framework-assets.s3.amazonaws.com/doc-assets/two_types_of_graph.png)

预定义图具有 `auto_start` 属性，用于确定该图是否在 TEN 应用启动时自动启动。

此外，预定义图还有 `singleton` 属性，用于指示该图是否只能在 TEN 应用内生成 *一个* 对应的实例。

## 图 ID 和图名称

对于每个图实例，TEN 应用内部会使用一个唯一的 UUID4 字符串作为标识，称为**图 ID**。

对于预定义图，可以分配一个有意义且易于记忆的名称，称为**图名称**。当需要指定特定预定义图时，可以直接使用其图名称。如果预定义图具有 `singleton` 属性，则表示该预定义图在 TEN 应用内只能存在一个实例。在这种情况下，TEN runtime 会使用图名称来唯一标识从该预定义图生成的单一实例。

## 动态图

当 TEN 应用收到 `start_graph` 命令并创建动态图时，系统会分配一个随机 UUID 作为该图的 ID。如果其他客户端获取了此图 ID，它们也可以连接到这个图。

动态图 ID 示例：

`123e4567-e89b-12d3-a456-426614174000`

## 预定义图

预定义图与动态图的工作原理相似，主要区别在于内容定义方式。动态图的内容包含在 `start_graph` 命令中，而预定义图的内容则由 TEN 应用预先定义。客户端只需在 `start_graph` 命令中指定预定义图的名称即可启动该图。

预定义图的主要优势在于简化使用并保护敏感信息。使用预定义图，客户端无需了解图的详细结构，这既提高了可用性，也避免了暴露图中可能包含的敏感信息。

预定义图名称示例：

`http_server`

当 TEN 应用启动时，所有设置了 `auto_start` 属性的预定义图会自动启动。

## 图定义

无论是动态图还是预定义图，其定义结构都是相同的：

```json
{
  "nodes": [
    // 节点的定义
  ],
  "connections": [
    // 通信链路的定义
  ]
}
```

关键要点：

1. 如果只有一个 TEN 应用，可以省略 `app` 字段。如果有多个应用，则必须明确指定 `app` 字段。

2. `nodes` 字段定义图中的节点，如各种扩展（extension）。

3. 每个节点在 `nodes` 字段中只能出现一次。如果多次出现同一节点，TEN framework 会在验证过程中报错。

4. 在 `nodes` 字段中定义扩展的方式如下，其中 `property` 字段为可选：

   ```json
   {
     "type": "extension",
     "name": "simple_http_server_cpp",
     "addon": "simple_http_server_cpp",
     "extension_group": "default_extension_group",
     "app": "msgpack://127.0.0.1:8001/",
     "property": {
       "root_key": "player",
       "extra_keys": ["playerName"]
     }
   }
   ```

   `addon` 字段表示该扩展是由对应插件生成的实例。

5. `connections` 字段定义图中节点间的通信链路，其中 `extension` 值表示对应节点的名称。

完整示例：

```json
{
  "nodes": [
    {
      "type": "extension",
      "app": "msgpack://127.0.0.1:8001/",
      "name": "simple_http_server_cpp",
      "addon": "simple_http_server_cpp",
      "extension_group": "default_extension_group",
      "property": {
        "root_key": "player",
        "extra_keys": ["playerName"]
      }
    }
  ],
  "connections": [
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "simple_http_server_cpp",
      "cmd": [
        {
          "name": "start",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        },
        {
          "name": "stop",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "gateway"
            }
          ]
        }
      ]
    },
    {
      "app": "msgpack://127.0.0.1:8001/",
      "extension": "gateway",
      "cmd": [
        {
          "name": "push_status_online",
          "dest": [
            {
              "app": "msgpack://127.0.0.1:8001/",
              "extension": "uap"
            }
          ]
        }
      ]
    }
  ]
}
```

## 预定义图的定义

预定义图是在 TEN 应用的配置文件（`property.json`）中的 `predefined_graphs` 字段下定义的，包含 `name`，`auto_start`，`singleton` 等属性：

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        // 完整的图定义
      }
    ]
  }
}
```

完整示例：

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": true,
        "singleton": true,
        "nodes": [
          {
            "type": "extension",
            "name": "simple_http_server_cpp",
            "addon": "simple_http_server_cpp",
            "extension_group": "default_extension_group",
            "property": {
              "root_key": "player",
              "extra_keys": [
                "playerName"
              ]
            }
          }
        ],
        "connections": [
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "simple_http_server_cpp",
            "cmd": [
              {
                "name": "start",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              },
              {
                "name": "stop",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "gateway"
                  }
                ]
              }
            ]
          },
          {
            "app": "msgpack://127.0.0.1:8001/",
            "extension": "gateway",
            "cmd": [
              {
                "name": "push_status_online",
                "dest": [
                  {
                    "app": "msgpack://127.0.0.1:8001/",
                    "extension": "uap"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## `start_graph` 命令的定义

动态图是通过向 TEN 应用发送 `start_graph` 命令创建的，将图定义放在命令的 `ten` 字段下：

```json
{
  "ten": {
    "type": "start_graph",
    // 完整的图定义
  }
}
```

完整示例：

```json
{
  "ten": {
    "type": "start_graph",
    "nodes": [
      {
        "type": "extension",
        "name": "simple_http_server_cpp",
        "addon": "simple_http_server_cpp",
        "extension_group": "default_extension_group",
        "property": {
          "root_key": "player",
          "extra_keys": ["playerName"]
        }
      }
    ],
    "connections": [
      {
        "app": "msgpack://127.0.0.1:8001/",
        "extension": "simple_http_server_cpp",
        "cmd": [
          {
            "name": "start",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          },
          {
            "name": "stop",
            "dest": [
              {
                "app": "msgpack://127.0.0.1:8001/",
                "extension": "gateway"
              }
            ]
          }
        ]
      },
      {
        "extension": "gateway",
        "cmd": [
          {
            "name": "push_status_online",
            "dest": [
              {
                "extension": "uap"
              }
            ]
          }
        ]
      }
    ]
  }
}
```

### 使用 `import_uri` 引用图定义

在定义预定义图时，可以使用 `import_uri` 字段引用外部图定义文件：

```json
{
  "ten": {
    "uri": "http://localhost:8001",
    "predefined_graphs": [
      {
        "name": "default",
        "auto_start": false,
        "import_uri": "../graph.json"
      }
    ]
  }
}
```

`import_uri` 可以是相对路径，绝对路径或 URL：

* 相对路径：相对于 `property.json` 文件所在目录
* 绝对路径：相对于 TEN 应用所在环境的根目录
* URL：直接作为 URL 路径处理

## 图定义的规范

* **`nodes` 字段**：在图定义中必须提供 `nodes` 数组。相比之下，`connections` 数组虽然可选，但建议提供以定义节点间通信。

* **节点 `app` 字段**：`app` 字段不能设置为 `localhost`。在单应用图中，应省略 `app` URI；在多应用图中，`app` 字段值必须与各应用 `property.json` 中的 `ten.uri` 值匹配。

* **节点唯一性**：`nodes` 数组中的每个节点表示一个特定扩展实例，必须通过 `app` 和 `name` 组合唯一标识。不允许出现重复定义，如以下无效示例：

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_1",
        "extension_group": "test"
      },
      {
        "type": "extension",
        "name": "some_ext",
        "addon": "addon_2",
        "extension_group": "test"
      }
    ]
  }
  ```

* **连接定义的一致性**：`connections` 中引用的所有扩展实例必须在 `nodes` 中明确定义。以下示例无效，因为 `ext_2` 未在 `nodes` 中定义：

  ```json
  {
    "nodes": [
      {
        "type": "extension",
        "name": "ext_1",
        "addon": "addon_1",
        "extension_group": "some_group"
      }
    ],
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **通信链路整合**：同一源扩展的所有消息应分组在单个部分中，不应分散在多个定义中。以下示例不正确：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      },
      {
        "extension": "ext_1",
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  正确做法是将同一源扩展的所有消息整合在一起：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ],
        "data": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

* **消息目标整合**：对于特定类型的每条消息，所有目标扩展应归类在单个条目下。以下示例不正确：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              }
            ]
          },
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  正确做法是将同一消息的所有目标整合在一起：

  ```json
  {
    "connections": [
      {
        "extension": "ext_1",
        "cmd": [
          {
            "name": "hello",
            "dest": [
              {
                "extension": "ext_2"
              },
              {
                "extension": "ext_3"
              }
            ]
          }
        ]
      }
    ]
  }
  ```

  注意，相同名称的消息可以在不同类型（如 `cmd` 和 `data`）中共存，不会导致冲突。

## 子图（Subgraph）

TEN framework 的核心机制是基于图结构，这种结构由节点和连接构成。子图是一种强大的复用机制，可以将复杂的图结构拆分为可重用的模块，从而提升代码的组织性和可维护性。

### 子图设计理念

图的本质是定义扩展之间数据流动的方式。子图在实现上并没有改变这一基本原理，而是作为一种语法糖，最终会被展平到所属的大图中，并使用与普通图相同的机制启动。这种设计既简化了复杂系统的开发，又不增加运行时的复杂度。

#### 设计原则

子图设计遵循以下核心原则：

1. **独立性**：任何一张图都是一个完整的图，可以单独运行，也可以作为组件嵌入到其他图中。

2. **工具友好**：子图提供额外的信息，帮助开发工具更好地理解图结构，提升开发体验，同时不增加运行时的复杂度。

3. **展平机制**：子图最终会被展平为标准的图结构，以确保性能和兼容性。

4. **简洁性**：子图的设计旨在简化开发过程，而不是增加复杂性，避免增加 JSON 结构的复杂度。

子图的设计目的是让开发者可以将一个图当作黑盒使用，无需关注其内部的复杂性。因此，在引用子图的地方，不提供特殊的修补机制来调整子图内部的状态，以减少复杂性。如果需要修改子图的定义，应直接修改原始定义，而不是在引用处进行修补。这么做是为了避免复杂性扩散，增加维护成本。例如，当将子图 A 引入到大图 B 时，如果需要调整子图 A 内部的 extension 属性的 schema 定义，应直接修改该 extension 的 schema 定义，而不是在引用处提供修改机制，因为这可能会导致不良的连锁反应，只修改了 extension 的 schema 定义，而没有修补其他地方，从而导致其他地方出错。

换句话说，子图主要提供的是一种映射机制，用来将子图内的元素暴露给外部，而不是提供机制来修补子图内部的元素。

### 子图实现机制

接下来，我们将详细介绍子图的实现机制，包括子图的定义、引用方式和展平过程。

#### 子图定义示例

以下是 `subgraph.json` 示例，可作为独立图使用，也可作为子图被其他图引用：

```json
{
  "nodes": [
    {
      // 定义名为 ext_c 的扩展节点
      "type": "extension",
      "name": "ext_c",
      "addon": "extension_c"
    },
    {
      // 定义名为 ext_d 的扩展节点
      "type": "extension",
      "name": "ext_d",
      "addon": "extension_d"
    }
  ],
  "connections": [
    {
      // ext_c 将 B 命令传输到 ext_d
      "extension": "ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_d"
            }
          ]
        }
      ]
    }
  ],
  "exposed_messages": [
    // 表示图向外部暴露的消息接口, 主要供开发工具使用，提供智能提示
    {
      // ext_d 的 B 命令暴露给外部
      "extension": "ext_d",
      "type": "cmd_in",
      "name": "B"
    }
  ],
  "exposed_properties": [
    // 表示图向外部暴露的属性
    {
      "extension": "ext_c",
      "name": "a",
      "alias": "a1"
    }
  ]
}
```

#### 子图引用示例

引用子图的 `graph.json` 示例：

```json
{
  "nodes": [
    {
      // 定义名为 ext_a 的扩展节点
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // 定义名为 ext_b 的扩展节点
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 引用子图，命名为 subgraph_1
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "./ten_packages/extension/aaa/subgraph.json",
      "property": {
        // 可以写在这边的 property 字段, 必须要存在于子图的 exposed_properties 字段,
        // 不然会是一个错误情况
        "app_id": "${env:AGORA_APP_ID}",
        "token": "<agora_token>",
        "channel": "ten_agent_test"
      }
    },
    {
      // 引用子图，命名为 subgraph_2
      "type": "subgraph",
      "name": "subgraph_2",
      "import_uri": "./ten_packages/extension/bbb/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // 第一个目标是 ext_b
              "extension": "ext_b"
            },
            {
              // 第二个目标是子图中的 ext_d, 这算是进阶用法
              "extension": "subgraph_1:ext_d"
            },
            {
              // 第三个目标是子图, 但这需要子图有明确 expose 出 B 这个 cmd_in 接口, 不然
              // 会被当成是错误情况
              "subgraph": "subgraph_2"
            }
          ]
        }
      ]
    },
    {
      // 子图中的 ext_c 将 cmd H 传输给 ext_a, 这算是进阶用法
      "extension": "subgraph_1:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 子图 subgraph_2 将 cmd H 传输给 ext_a, 不过這需要子图 expose 出 H 这个
      // cmd_out 接口, 不然会被当成是错误情况
      "subgraph": "subgraph_2",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

#### 关键概念

子图机制引入三个关键概念：

1. **消息暴露**（exposed\_messages）：
   * 子图通过 `exposed_messages` 字段声明对外暴露的接口
   * 主要供开发工具使用，实现智能提示和检查
   * 隐藏子图内部细节，提升开发体验

2. **属性暴露**（exposed\_properties）：
   * 子图通过 `exposed_properties` 字段声明对外暴露的属性
   * 主要供开发工具使用，实现智能提示和检查
   * 隐藏子图内部细节，提升开发体验

3. **子图引用与命名**：
   * 通过 `"type": "subgraph"` 引用其他图文件
   * 每个子图有唯一标识名称，作为命名空间
   * 防止不同子图中的同名元素冲突

4. **跨图连接**：
   * 通过命名空间语法（如 `subgraph_1:ext_d`）引用子图内元素, 或者通过 `subgraph` 字段引用子图
   * 使子图内元素可与主图交互
   * 构建复杂的跨图消息流

总之，`exposed_messages` 以及 `exposed_properties` 字段描述了如何从图的边界向内流动消息，开发工具则利用此字段提供智能提示，便于开发者指定消息如何流向图的边界。

#### 展平机制

最终，引用子图的图会被展平为普通图结构，确保运行时统一性和高效性。不过这个展平的结果不会被用户看到，而是在 TEN runtime 或者 TEN manager 中被展平到内存中使用。注意在展平后的图定义中，就是原始的有哪些 extensions，以及在这些 extensions 之间的连线。

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 子图中的 ext_c 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "extension_c"
    },
    {
      // 子图中的 ext_d 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "extension_d",
      "property": {
        // 在引用子图时所定义的 property 字段, 会被展平到对应的 extension 的 property 字
        // 段
        "app_id": "${env:AGORA_APP_ID}"
      }
    },
    {
      // 子图中的 ext_e 被展平，名称前缀为子图名称
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "extension_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b"
            },
            {
              "extension": "subgraph_1_ext_d"
            },
            {
              "extension": "subgraph_2_ext_e"
            }
          ]
        }
      ]
    },
    {
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 子图内部连接也被展平纳入
      "extension": "subgraph_1_ext_c",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "subgraph_1_ext_d"
            }
          ]
        }
      ]
    }
  ]
  // exposed_messages 以及 exposed_properties 字段在展平过程中被舍弃
}
```

子图展平机制遵循以下规则：

1. 展平前，冒号（`:`）符号表示元素位于子图中（如 `subgraph_1:ext_d`）。

2. 展平后，子图中元素名称添加子图名称作为前缀（如 `subgraph_1_ext_c`），确保全局唯一性。

3. 展平后的图定义不再包含冒号符号，以区分展平前后的状态。

4. 子图中的内部连接会保留并纳入展平后的图中，保证功能完整性。

### 高级功能与应用

随着项目复杂度增加，子图的高级功能可帮助更好地组织和管理系统。

#### 消息转换与子图

子图完全支持消息转换（msg\_conversion）机制，用于处理不同接口间的消息格式转换：

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "subgraph",
      "name": "subgraph_1",
      "import_uri": "http://a.b.c.d/subgraph.json"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1:ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "subgraph": "subgraph_2",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

开发工具可利用 `exposed_messages` 信息，提示兼容性并提供消息转换配置界面。配置完成后，转换规则会自动写入图定义，简化开发流程。

展平后，消息转换规则会正确保留，确保运行时行为与设计意图一致：

```json
{
  "nodes": [
    {
      "type": "extension",
      "name": "ext_a",
      "addon": "addon_a"
    },
    {
      "type": "extension",
      "name": "ext_b",
      "addon": "addon_b"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_c",
      "addon": "addon_c"
    },
    {
      "type": "extension",
      "name": "subgraph_1_ext_d",
      "addon": "addon_d"
    },
    {
      "type": "extension",
      "name": "subgraph_2_ext_e",
      "addon": "addon_e"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              "extension": "ext_b",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_1_ext_d",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            },
            {
              "extension": "subgraph_2_ext_e",
              "msg_conversion": {
                "type": "per_property",
                "rules": [
                  {
                    "path": "extra_data",
                    "conversion_mode": "fixed_value",
                    "value": "tool_call"
                  }
                ],
                "keep_original": true
              }
            }
          ]
        }
      ]
    }
  ]
}
```

通过这种方式，开发者可灵活处理不同组件间的消息格式差异，无需关心底层实现细节。

## "subgraph\_1:ext\_c" 的意义

在前面的示例中，`subgraph_1:ext_c` 中的 `subgraph_1` 不是表示子图本身，而是引用 `nodes` 字段中的一个元素。在子图应用中，`subgraph_1` 可视为一个命名空间。

通过这种机制，可以在图的连接中指定不同位置的扩展作为源或目标，实现跨多个图的连接。

核心概念是：图的连接本质上是扩展之间的连接。通过这种机制，可以在 `nodes` 字段中指定不同类型的元素作为连接的源或目标，实现跨多个子图甚至多个图的连接。

## 跨多个图的连接

除了子图外，还可以通过类似机制在一个图中指定与其他图的连接。

### 与同一 TEN 应用中的预定义图连接

示例：如何在一个图中连接到同一 TEN 应用中的预定义图：

```json
{
  "nodes": [
    {
      // 定义名为 ext_a 的扩展
      "type": "extension",
      "name": "ext_a",
      "addon": "extension_a"
    },
    {
      // 定义名为 ext_b 的扩展
      "type": "extension",
      "name": "ext_b",
      "addon": "extension_b"
    },
    {
      // 引用其他图, 图 ID 是 default
      "type": "graph",
      "graph_id": "default"
    }
  ],
  "connections": [
    {
      "extension": "ext_a",
      "cmd": [
        {
          "name": "B",
          "dest": [
            {
              // 第一个目标是 ext_b
              "extension": "ext_b"
            },
            {
              // 第二个目标是另一张图中的 ext_d
              "extension": "default:ext_d"
            }
          ]
        }
      ]
    },
    {
      // 另一张图中的 ext_c 将 cmd H 传输给 ext_a
      "extension": "default:ext_c",
      "cmd": [
        {
          "name": "H",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    },
    {
      // 另一张图将 cmd X 传输给 ext_a, 但这需要另一张图 expose 出 X 这个 cmd_out 接口,
      // 不然会被当成是错误情况. 这个错误情况如果可以被静态检查出来, 则会在静态检查的时候报
      // 错, 或者在运行时报错.
      "graph": "default",
      "cmd": [
        {
          "name": "X",
          "dest": [
            {
              "extension": "ext_a"
            }
          ]
        }
      ]
    }
  ]
}
```

### graph node 的种类

1. Subgraph

   ```json
   {
     "type": "subgraph",
     "name": "subgraph_1", // namespace 的意义
     "import_uri": "http://a.b.c.d/subgraph.json"
   }
   ```

2. Local graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": true
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": false
   }
   ```

3. Remote graph

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": true,
     "app": "msgpack:://127.0.0.1:8002/"
   }
   ```

   ```json
   {
     "type": "graph",
     "graph_name": "default",
     "singlton": false,
     "app": "msgpack:://127.0.0.1:8002/"
   }
   ```


# 消息转换 (/cn/docs/ten_framework/0.11.60/graph/msg_conversion)

## 消息转换

TEN Framework 提供了强大的消息转换功能，允许您在消息传递过程中对消息内容进行转换和修改。通过配置转换规则，您可以实现消息字段的重命名、值的替换、以及复杂的数据结构转换。

### 基本配置

消息转换通过 `msg_conversion` 配置项进行设置：

```json
{
  "msg_conversion": {
    "type": "per_property",
    "keep_original": false,
    "rules": [
      {
        "path": "user.name",
        "conversion_mode": "fixed_value",
        "value": "default_user"
      },
      {
        "path": "data.items[0].id",
        "conversion_mode": "from_original",
        "original_path": "original_id"
      }
    ]
  }
}
```

### 配置参数说明

* **type**: 转换类型，目前支持 `per_property`（按属性转换）
* **keep\_original**: 是否保留原始消息内容，设为 `false` 时只保留转换后的字段
* **rules**: 转换规则数组，定义具体的转换逻辑

### 转换规则

每个转换规则包含以下字段：

* **path**: 目标字段路径，指定要设置的消息字段位置
* **conversion\_mode**: 转换模式
  * `fixed_value`: 设置固定值
  * `from_original`: 从原始消息的指定路径复制值
* **value**: 当转换模式为 `fixed_value` 时使用的固定值
* **original\_path**: 当转换模式为 `from_original` 时，指定原始消息中的源字段路径

### 路径表达式

消息转换中的路径表达式支持多种访问方式：

* **对象属性访问**: `user.profile.name` - 访问嵌套对象的属性
* **数组索引访问**: `items[0].name` - 访问数组指定索引的元素
* **混合路径访问**: `user.tags[1].value` - 结合对象和数组访问

### 使用场景

消息转换功能适用于以下场景：

* **字段重命名**: 将消息中的字段名称转换为目标系统要求的格式
* **数据格式转换**: 调整消息的数据结构以适配不同的处理模块
* **默认值设置**: 为缺失的字段提供默认值
* **数据清洗**: 过滤或转换不符合要求的数据内容


# Go (/cn/docs/ten_framework/0.11.60/language_binding/go)

让我们使用 TEN framework 构建一个 Go 应用程序。

## 创建 TEN 应用

首先，我们将通过集成几个预构建的 TEN 包来创建一个基本的 TEN Go 应用。请按照以下步骤操作：

```shell title="终端"
tman install app default_app_go
cd default_app_go

tman install protocol msgpack
tman install extension_group default_extension_group
```

### 安装默认的 TEN 扩展

接下来，安装一个用 Go 编写的默认 TEN 扩展：

```shell title="终端"
tman install extension default_extension_go
```

### 声明预构建的自动启动图

现在，我们将修改 TEN 应用的 `property.json` 文件，以包含图的声明。这将确保默认扩展在 TEN 应用启动时自动启动。

```json title="property.json"
"predefined_graphs": [
  {
    "name": "default",
    "auto_start": true,
    "nodes": [
      {
        "type": "extension_group",
        "name": "default_extension_group",
        "addon": "default_extension_group"
      },
      {
        "type": "extension",
        "name": "default_extension_go",
        "addon": "default_extension_go",
        "extension_group": "default_extension_group"
      }
    ]
  }
]
```

## 构建应用

与标准的 Go 项目不同，TEN Go 应用使用 CGo，因此您需要在构建之前设置某些环境变量。TEN runtime Go 绑定系统包中已经提供了一个构建脚本，因此您可以使用一个命令构建应用：

```shell title="终端"
go run ten_packages/system/ten_runtime_go/tools/build/main.go
```

编译后的二进制文件 `main` 将在 `/bin` 文件夹中生成。

## 启动应用

由于需要设置一些环境变量，建议使用提供的脚本来启动应用：

```shell title="终端"
./bin/start
```

## 调试

如果您使用 Visual Studio Code (VSCode) 作为开发环境，则可以使用以下配置来调试 Go/C 代码。

### 调试 Go 代码

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Go app (launch)",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}",
      "cwd": "${workspaceFolder}",
      "output": "${workspaceFolder}/bin/tmp",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

### 调试 C 代码

```json title=".json"
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "C (launch)",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/bin/main",
      "cwd": "${workspaceFolder}",
      "env": {
        "LD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "DYLD_LIBRARY_PATH": "${workspaceFolder}/lib",
        "CGO_LDFLAGS": "-L${workspaceFolder}/lib -lten_runtime_go -Wl,-rpath,@loader_path/lib"
      }
    }
  ]
}
```

## CGO

### 生成的代码

当 Go 与 C 接口时，`cgo` 工具至关重要。当从 Go 调用 C 函数时，`cgo` 会将 Go 源文件转换为多个输出文件，包括 Go 和 C 源文件。然后，使用 gcc 或 clang 等编译器编译这些生成的 C 源文件。

使用以下命令生成这些 C/Go 源文件：

```bash title="终端"
mkdir out
go tool cgo -objdir out
```

示例：

```bash title="终端"
go tool cgo -objdir out cmd.go error.go
```

生成的文件包括：

```bash title="终端"
├── _cgo_export.c
├── _cgo_export.h
├── _cgo_flags
├── _cgo_gotypes.go
├── _cgo_main.c
├── _cgo_.o
├── cmd.cgo1.go
├── cmd.cgo2.c
├── error.cgo1.go
└── error.cgo2.c
```

* **`_cgo_export.h`** 是 cgo 上下文中 Go 和 C 之间互操作的关键。它包含可以从 C 访问的 Go 函数的必要声明。

  此文件是根据使用 `//export` 指令显式导出的 Go 函数自动生成的。它还包括与这些函数中使用的 Go 类型对应的 C 类型的定义，确保两种语言之间的兼容性。

  类型定义示例：

```c title=".h"
  typedef signed char GoInt8;
  typedef unsigned char GoUint8;
  typedef short GoInt16;
  typedef unsigned short GoUint16;
  typedef int GoInt32;
  typedef unsigned int GoUint32;
  typedef long long GoInt64;
  typedef unsigned long long GoUint64;
  typedef GoInt64 GoInt;
  typedef GoUint64 GoUint;
  typedef size_t GoUintptr;
  typedef float GoFloat32;
  typedef double GoFloat64;
```

* **`_cgo_gotypes.go`** 包含 C 中定义并在 Go 中使用的相应 Go 类型。

  例如，如果您在头文件 `common.h` 中定义了一个结构体 `ten_go_error_t` 并在 Go 中使用 `C.ten_go_error_t`，则在 `_cgo_gotypes.go` 中会有一个相应的 Go 类型：

```go title=".go"
  package ten

  type _Ctype_ten_go_status_t = _Ctype_struct_ten_go_status_t

  type _Ctype_struct_ten_go_status_t struct {
      err_no   _Ctype_int
      msg_size _Ctype_uint8_t
      err_msg  [256]_Ctype_char
      _        [3]byte
  }
```

* **`cmd.cgo1.go`** 是 `cgo` 生成的与原始 Go 源文件 `cmd.go` 对应的文件。它将对 C 函数和类型的直接调用替换为对 `cgo` 提供的生成的 Go 函数和类型的调用。

  示例：

```go title=".go"
  package ten

  func NewCmd(cmdName string) (Cmd, error) {
      // ...
      cStatus := func() _Ctype_struct_ten_go_status_t {
          var _cgo0 _cgo_unsafe.Pointer = unsafe.Pointer(unsafe.StringData(cmdName))
          var _cgo1 _Ctype_int = _Ctype_int(len(cmdName))
          _cgoBase2 := &bridge
          _cgo2 := _cgoBase2
          _cgoCheckPointer(_cgoBase2, 0 == 0)
          return _Cfunc_ten_go_cmd_create_custom_cmd(_cgo0, _cgo1, _cgo2)
      }()
      // ...
  }
```

* **`cmd.cgo2.c`** 是从 Go 调用的原始 C 函数的包装器。

  示例：

```c title=".c"
  CGO_NO_SANITIZE_THREAD void _cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd(void *v) {
      struct {
          const void* p0;
          int p1;
          char __pad12[4];
          ten_go_msg_t** p2;
          ten_go_error_t r;
      } __attribute__((__packed__, __gcc_struct__)) *_cgo_a = v;
      char *_cgo_stktop = _cgo_topofstack();
      __typeof__(_cgo_a->r) _cgo_r;
      _cgo_tsan_acquire();
      _cgo_r = ten_go_cmd_create_cmd(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2);
      _cgo_tsan_release();
      _cgo_a = (void*)((char*)_cgo_a + (_cgo_topofstack() - _cgo_stktop));
      _cgo_a->r = _cgo_r;
      _cgo_msan_write(&_cgo_a->r, sizeof(_cgo_a->r));
  }
```

因此，从 Go 调用 `C.ten_go_cmd_create_cmd()` 的调用顺序为：

```text
_Cfunc_ten_go_cmd_create_custom_cmd (自动生成的 Go)
           |
           V
   _cgo_runtime_cgocall (Go)
           |
           V
      entrysyscall     // 将 Go 堆栈切换到 C 堆栈
           |
           V
_cgo_cb1b98e39356_Cfunc_ten_go_cmd_create_custom_cmd (自动生成的 C)
           |
           V
   ten_go_cmd_create_cmd (C)
           |
           V
      exitsyscall
```

### 不完整类型

如前所述，`cgo` 工具会根据通过 `import "C"` 导入的 C 头文件在 `_cgo_gotypes.go` 中生成相应的 Go 类型。如果结构体在 C 头文件中是不透明的，则 `cgo` 工具会生成一个不完整类型。

不透明 C 结构体的示例：

```c title=".h"
typedef struct ten_go_msg_t ten_go_msg_t;
```

`cgo` 工具将在 Go 中生成一个不完整类型：

```go title=".go"
type _Ctype_ten_go_msg_t = _Ctype_struct_ten_go_msg_t
type _Ctype_struct_ten_go_msg_t _cgopackage.Incomplete
```

#### 如果在 Go 中使用不完整类型会发生什么？

* **无法分配不完整类型。**

  由于 `sys.NotInHeap` 无法在 Go 堆上分配，因此 `new` 或 `make` 等操作将不起作用。尝试在 Go 中创建不透明结构体的新实例将导致编译器错误：

```go title=".go"
  msg := new(C.ten_go_msg_t) // 错误：无法在 Go 中分配；它是不完整的（或不可分配的）
```

* **无法将指向不完整类型的指针直接传递给 C。**

  如果您的 C 函数具有指向不透明结构体的指针作为参数，则根据 cgo 规则，将 Go 指针直接传递给此不完整类型的 C 函数将不起作用。Go 编译器将要求指针“固定”，以确保它符合 Go 的垃圾收集器 (GC) 约束。

### 使用 `C.uintptr_t` 而不是指向不透明结构体的指针的规则

* **优点：**

  * Go 中的 `C.uintptr_t` 和 `uintptr` 是足够大的整数，可以保存 C 指针，从而避免从 Go 传递到 C 时进行内存分配或转换。

* **限制：**

  * `uintptr` 是一个整数，而不是指针。它表示没有指针运算的地址的位模式。
  * `uintptr` 无法在 Go 中解引用。将其转换为 `unsafe.Pointer` 可能会导致 Go 的 GC 出现问题。
  * 由于 `uintptr` 是一个整数，因此无法将 `nil` 或 `NULL` 传递给 C。使用 `0` 而不是 `nil` 来表示空地址。


# Manifest (/cn/docs/ten_framework/0.11.60/metadata_system/manifest)

## 什么是 Manifest 文件

Manifest 文件是 TEN Framework 中用于存储软件包元数据的重要配置文件。它定义了扩展（Extension）或应用（App）的基本信息、依赖关系、API 接口、脚本配置等关键信息。

## Manifest 文件的作用

* **包标识**：定义包的类型、名称和版本信息
* **依赖管理**：声明包所需的依赖项及其版本要求
* **API 定义**：描述包对外提供的数据接口和命令接口
* **打包配置**：指定需要包含在发布版本中的文件
* **脚本管理**：定义构建、测试等自动化脚本

## Extension Manifest 示例

以下是一个 AI 聊天扩展的完整 Manifest 示例：

```json
{
  "type": "extension",
  "name": "openai_chatgpt_python",
  "version": "0.1.0",
  "tags": ["ai", "llm", "python"],
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_python",
      "version": "^0.8.0"
    }
  ],
  "package": {
    "include": [
      "manifest.json",
      "property.json",
      "*.py",
      "README.md"
    ]
  },
  "api": {
    "property": {
      "properties": {
        "api_key": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "max_tokens": {
          "type": "int64"
        },
        "temperature": {
          "type": "float64"
        }
      }
    },
    "data_in": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "data_out": [
      {
        "name": "text_data",
        "property": {
          "properties": {
            "text": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_in": [
      {
        "name": "flush"
      },
      {
        "name": "tool_register",
        "property": {
          "properties": {
            "tool": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "description": {
                  "type": "string"
                },
                "parameters": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                }
              },
              "required": ["name", "description", "parameters"]
            }
          }
        },
        "result": {
          "property": {
            "properties": {
              "response": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "tool_call",
        "property": {
          "properties": {
            "name": {
              "type": "string"
            },
            "args": {
              "type": "string"
            }
          },
          "required": ["name"]
        }
      }
    ]
  },
  "scripts": {
    "test": "python -m pytest tests/",
    "build": "python setup.py build"
  }
}
```

## App Manifest 示例

以下是一个语音助手应用的 Manifest 示例：

```json
{
  "type": "app",
  "name": "voice_assistant_app",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime_go",
      "version": "^0.8.0"
    }
  ],
  "api": {
    "property": {
      "properties": {
        "app_id": {
          "type": "string"
        },
        "log_level": {
          "type": "uint8"
        },
        "server_config": {
          "type": "object",
          "properties": {
            "host": {
              "type": "string"
            },
            "port": {
              "type": "int32"
            },
            "ssl_enabled": {
              "type": "bool"
            }
          },
          "required": ["host", "port"]
        }
      }
    }
  }
}
```

## 主要字段说明

### 基本信息

* **type**：包的类型，可以是 `extension` 或 `app`
* **name**：包的唯一标识名称
* **version**：遵循语义化版本规范的版本号
* **tags**：用于分类和搜索的标签数组

### 依赖配置

* **dependencies**：依赖项列表，每个依赖包含类型、名称和版本要求

### 打包配置

* **package.include**：指定打包时需要包含的文件和目录

### API 配置

* **api.property**：定义扩展的配置属性
* **api.data\_in/data\_out**：定义数据输入和输出接口
* **api.cmd\_in/cmd\_out**：定义命令输入和输出接口

### 脚本配置

* **scripts**：定义各种自动化脚本，如测试、构建等

## 最佳实践

1. **版本管理**：使用语义化版本控制，确保版本号的一致性
2. **依赖声明**：明确声明所有必需的依赖项及其版本范围
3. **API 设计**：清晰定义输入输出接口，便于其他组件集成
4. **文档完整**：确保 Manifest 中的配置与实际实现保持一致


# 元数据系统 (/cn/docs/ten_framework/0.11.60/metadata_system/metadata_system)

TEN framework 在各种类型的包中采用一致的元数据系统。

## 元数据类型

在 TEN 元数据系统中，有两种主要类型的元数据：

### 1. **Manifest（清单）**

* **位置：** 存储在相关 TEN 包的根目录中，文件名为 `manifest.json`。
* **内容：**

  * **包名称：** TEN 包的名称。
  * **包版本：** TEN 包的版本，遵循语义版本控制。
  * **API Schema：** 定义与包的属性和输入/输出消息相关的模式。

  > ⚠️ **注意：** `manifest.json` 中的 API schema *不是* JSON schema。

### 2. **Property（属性）**

* **位置：** 通常存储在 TEN 包根目录的 `property.json` 文件中。
* **目的：** `property.json` 文件存储初始属性值，这些属性值在运行时是可读写的。这意味着可以在 TEN runtime 运行时修改属性。

## 清单文件 (Manifest File)

`manifest.json` 文件充当 TEN 包的蓝图，定义其元数据、属性以及它处理的输入/输出消息。

### `manifest.json` 示例

```json
{
  "type": "app",
  "name": "default_app_cpp",
  "version": "1.0.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "1.0.0"
    }
  ],
  "api": {
    "property": {
      "exampleInt8": {
        "type": "int8"
      },
      "exampleString": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "cmd_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        },
        "result": {
          "property": {
            "detail": {
              "type": "string"
            },
            "aaa": {
              "type": "int8"
            },
            "bbb": {
              "type": "string"
            }
          }
        }
      }
    ],
    "cmd_out": [],
    "data_in": [
      {
        "name": "data_foo",
        "property": {
          "foo": {
            "type": "int8"
          },
          "bar": {
            "type": "string"
          }
        }
      }
    ],
    "data_out": [],
    "video_frame_in": [],
    "video_frame_out": [],
    "audio_frame_in": [],
    "audio_frame_out": []
  }
}
```

### 清单中 TEN 模式的目的

`manifest.json` 中的 TEN 模式为 TEN runtime 提供了关于 extension 的外部 API 的元数据，包括属性和消息的类型信息。这有助于运行平台在操作期间正确处理 extension 的属性和消息。

### 何时使用 TEN 模式

1. **属性验证：** 当 TEN runtime 获取/设置 TEN 包或消息的属性时。
2. **数据转换：** 使用 `from_json` API 将 JSON 文档转换为 TEN 包或消息属性时。
3. **兼容性检查：** 当 TEN 管理器根据 TEN 模式检查消息是否可以从源 extension 输出到目标 extension 时。

## 属性管理

在 TEN framework 中，有两种类型的属性：

1. **消息属性：** 这些属性与框架中 extension 之间交换的消息相关联。消息属性定义消息中携带的特定数据或参数，例如命令参数、数据负载或与音频/视频帧相关的元数据。
2. **TEN 包属性：** 这些属性与 TEN 包本身（例如 extension）相关联。TEN 包属性定义特定于包的配置或设置。例如，extension 可能具有配置其行为的属性，例如运行时设置、初始化参数或其他配置数据。

这两种类型的属性都在 TEN framework 中进行管理，但服务于不同的目的：一种侧重于组件之间的通信（消息属性），另一种侧重于组件本身的配置和操作（TEN 包属性）。

![属性系统](https://ten-framework-assets.s3.amazonaws.com/doc-assets/property_system.png)

### 定义 TEN 包属性

`property.json` 文件定义 TEN 包的属性。以下是一个示例：

```json
{
  "prop_1_name": 0,
  "prop_2_name": "prop_2_value",
  "prop_3_name": ["hello", "prop_3_sub_value"],
  "prop_4_name": {
    "prop_4_1_name": 1,
    "prop_4_2_name": "prop_4_sub_value"
  }
}
```

> ⚠️ **注意：** `property.json` 文件中的每个属性名称必须是唯一的。

### 属性的 TEN 模式

您可以在 `manifest.json` 文件中为属性定义 TEN 模式，使 TEN runtime 能够更有效地处理这些属性。如果属性没有相应的 TEN 模式，则运行平台将使用默认的 JSON 处理方法（例如，将所有 JSON 数字视为 `float64`）。如果提供了 TEN 模式，则运行平台将使用它来验证和相应地处理属性。

| 属性 | TEN 模式 | 效果                                          |
| -- | ------ | ------------------------------------------- |
| 是  | 是      | TEN runtime 根据 TEN 模式验证属性值（例如，检查类型）。        |
| 是  | 否      | TEN runtime 使用默认处理，将所有 JSON 数字视为 `float64`。 |

### 从 Extension 访问属性

TEN runtime 为 extension 提供了访问各种属性的 API。

### 从 Extension 访问 App 的属性

> ⚠️ **注意：** 本节所描述的行为还未被实现，是未来的 roadmap。

TEN framework 提供了强大的属性访问机制，让 extension 能够读取和修改 app 的属性。通过 get/set property API，您可以访问所有属性，包括 app 的属性。当 API 参数以 `app:` 前缀开头时，系统会识别这是对 app 属性的操作请求。

各语言实现的 API 设计考量：

* **Go**：提供不带 callback 的同步版本即可，简洁高效。
* **Node.js**：实现为不带 callback 的 async function，符合 Node.js 的异步编程模型。
* **Python**：理想情况下应提供带 callback 和不带 callback 的版本。目前可先实现不带 callback 的同步版本，但在访问 `app:` 属性时会使用 event wait，可能影响性能。
* **Async Python**：实现为不带 callback 的 async function，与 Python 异步模型一致。
* **C++**：应提供带 callback 和不带 callback 的版本。目前可先实现不带 callback 的同步版本，但在访问 `app:` 属性时会使用 event wait，可能影响性能。

这种设计使 app 属性的增删改查通过统一的 get/set\_property API 实现，保持了接口的一致性，且不会影响 message 属性的相关 API。

关于 `ten:` 命名空间字段的处理，采用以下规则：

1. `get/set_property_to/from_json(nullptr)` 操作不处理 `ten:` 字段
2. 要访问 `ten:` 字段，必须明确指定如 `get/set_property("ten:xxx")` 的完整路径
3. 当使用 `get/set_property_to/from_json()` 时，只有明确指定 `"ten:xxx"` 形式的 key 时才会处理 `ten:` 字段

这种设计既保持了 API 的简洁性，又避免了为特定需求创建额外命令的复杂性。

### 在 `start_graph` 命令中指定属性值

与 TEN 包相关的属性值可以在 `start_graph` 命令中指定。TEN runtime 根据 TEN 模式（如果已定义）处理这些属性，并将它们存储在相应的 TEN 包实例中。

#### `start_graph` 命令中属性规范的示例

```json
{
  "nodes": [
    {
      "type": "extension_group",
      "name": "foo_extension_group",
      "addon": "foo_extension_group_addon"
    },
    {
      "type": "extension",
      "name": "bar_extension",
      "extension_group": "foo_extension_group",
      "property": {
        "prop_1_name": 0,
        "prop_2_name": "prop_2_value",
        "prop_3_name": ["hello", "prop_3_sub_value"],
        "prop_4_name": {
          "prop_4_1_name": 1,
          "prop_4_2_name": "prop_4_sub_value"
        }
      }
    }
  ]
}
```


# Property (/cn/docs/ten_framework/0.11.60/metadata_system/property)

## Property 文件概述

Property 文件是 TEN Framework 中用来存储扩展（extension）属性值的配置文件。它定义了扩展的运行时参数和配置选项，是扩展正常运行的重要组成部分。

## Property 文件格式

Property 文件采用 JSON 格式，包含扩展运行所需的各种配置参数，如 API 密钥、模型配置、运行参数等。

### 基础属性示例

以下是一个简单的 Property 文件示例：

```json
{
  "api_key": "your-api-key-here",
  "max_tokens": 1000,
  "temperature": 0.7,
  "model_config": {
    "name": "gpt-4",
    "version": "2024-01-01"
  },
  "allowed_languages": ["en", "zh", "es"]
}
```

### AI 模型配置示例

针对 AI 模型扩展的 Property 文件配置示例：

```json
{
  "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxx",
  "model": "gpt-4",
  "max_tokens": 4096,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "prompt": "You are a helpful AI assistant.",
  "greeting": "Hello! How can I help you today?",
  "max_memory_length": 10,
  "vendor": "openai"
}
```

## 图配置

Property 文件还可以包含图配置（Graph Configuration），定义扩展之间的连接关系和数据流。

### 预定义图配置示例

```json
{
  "ten": {
    "predefined_graphs": [
      {
        "name": "default",
        "nodes": [
          {
            "type": "extension",
            "name": "my_extension",
            "addon": "my_extension_addon",
            "property": {
              "setting1": "value1",
              "setting2": 42
            }
          }
        ],
        "connections": [
          {
            "extension": "source_extension",
            "cmd": [
              {
                "name": "process_cmd",
                "dest": [
                  {
                    "extension": "target_extension"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}
```

## 使用说明

* Property 文件必须是有效的 JSON 格式
* 文件中的属性值会在扩展启动时被加载
* 可以通过修改 Property 文件来调整扩展的运行行为
* 图配置部分定义了扩展间的连接和数据流向


# 插件系统 (/cn/docs/ten_framework/0.11.60/software_packages/addon_system)

TEN framework 是一个软件模块系统，其中每个软件模块都由 **插件 (addon)** 的概念定义。本质上，TEN framework 中的每个软件模块都是一个插件。例如，用户可以定义以下类型的 TEN framework 插件：

1. TEN **扩展 (extension)** 插件
2. TEN **扩展组 (extension group)** 插件
3. TEN **协议 (protocol)** 插件

## 加载插件所需的信息

要将插件加载到 TEN runtime，需要三个信息：

1. **插件名称**

   插件名称在整个 TEN 应用中必须全局唯一。将多个同名插件加载到 TEN runtime 是未定义行为，可能会导致第二个插件实例无法正确加载。

2. **插件文件夹位置**

   在 TEN framework 中，加载插件时，用户必须告知 TEN runtime 插件目录的所在位置。此目录将作为插件的 `base_dir`。TEN runtime 将使用此基本目录加载与插件相关的其他文件，例如插件的 `manifest.json` 和 `property.json` 文件。

   基本上，在设计 TEN framework 的插件注册 API 时，其目标是自动检测插件所在的目录，因此用户无需显式指定 `base_dir`。但是，在某些 TEN framework 无法自动检测 `base_dir` 的特殊情况下，用户将需要为 TEN runtime 显式指定插件所在的文件夹。

3. **插件实例本身**

## 将插件注册到 TEN runtime

有两种方法可以将插件注册到 TEN runtime ：

1. 显式调用 API 以将插件的 `base_dir` 注册到 TEN runtime。

   **注意：** 目前，只有独立测试框架提供此类 API。

2. 将插件放置在 TEN 应用文件夹树中的指定位置。

通过将插件放置在指定位置，TEN 应用将在启动时自动加载这些插件。目前，此指定位置位于 TEN 应用的 `ten_packages/` 目录下。如果插件以此格式放置，它们将在 TEN 应用启动时自动加载。

```text
.
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    └── protocol/
        └── <protocol_a>/
```


# 应用 (/cn/docs/ten_framework/0.11.60/software_packages/app)

应用程序是 TEN 世界中的容器，包含组和扩展。
应用程序可以作为放置在 Docker 容器内的进程（主要在云中使用）。
应用程序也可以作为嵌入在现有应用程序内的线程（在客户端或服务器上）。
⇒ TEN 可以在现有项目上执行本地化修改（小幅增强），而无需根据 TEN 重写整个项目。

![将 TEN 作为进程或线程运行](https://ten-framework-assets.s3.amazonaws.com/doc-assets/run_ten_as_process_or_thread.png)

![应用程序运行的不同模式](https://ten-framework-assets.s3.amazonaws.com/doc-assets/different_mode_of_app_runnings.png)

## TEN 应用程序文件夹结构

在 TEN framework 中，开发人员可以使用各种编程语言创建 TEN 软件包。无论使用哪种语言，TEN 应用程序的整体文件夹结构都保持一致，确保不同实现之间的一致性。

![TEN 软件包文件夹结构](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_package_folder_structure.png)


# 扩展 (/cn/docs/ten_framework/0.11.60/software_packages/extension)

## 生命周期

扩展的生命周期分为以下阶段：

1. `on_configure`
2. `on_init`
3. `on_start`
4. `on_stop`
5. `on_deinit`

在所有这些阶段，扩展都可以发送消息，如果发送的消息是命令类型，则扩展还可以接收其发送的命令的结果。换句话说，不存在扩展**不能**发送消息或接收其自身发送的命令结果的阶段。因此，在所有这些阶段，扩展都可以向其他扩展发送消息，这有助于在所有阶段维持扩展之间的依赖关系。

每个生命周期阶段都对应一个回调函数，并且有一个相应的 `on_xxx_done()` 函数来标记该生命周期阶段的结束。

| 生命周期回调         | 生命周期结束函数            |
| -------------- | ------------------- |
| `on_configure` | `on_configure_done` |
| `on_init`      | `on_init_done`      |
| `on_start`     | `on_start_done`     |
| `on_stop`      | `on_stop_done`      |
| `on_deinit`    | `on_deinit_done`    |

### 阶段转换图

```mermaid
graph TD;
  on_configure((on_configure));
  on_init((on_init));
  on_start((on_start));
  on_stop((on_stop));
  on_deinit((on_deinit));

  on_configure-->on_configure_done
  on_configure_done-->on_init;
  on_configure-->on_stop;
  on_init-->on_init_done;
  on_init_done-->on_start;
  on_init-->on_stop;
  on_start-->on_stop;
  on_start-->on_start_done;
  on_stop-->on_stop_done;
  on_stop_done-->on_deinit;
  on_deinit-->on_deinit_done;
```

### on\_configure

用于设置扩展属性的初始值。这允许在其他生命周期阶段中使用 `ten_env` 的 `get_property` API 来检索在 `on_configure()` 阶段设置的属性。调用 `on_configure_done()` 函数以标记 `on_configure` 阶段的结束。

```c++
  void on_configure(ten::ten_env_t &ten_env) override {
    // 设置扩展属性的初始值。
    ten_env.init_property_from_json(
      R"({
           "ten": {
             "uri": "msgpack://127.0.0.1:8001/",
             "log_level": 2
           }
         })", nullptr);
    ten_env.on_configure_done();
  }
```

### on\_init

用于初始化扩展。由于在 `on_init_done()` 之前扩展尚未准备就绪，TEN runtime 会令发送到扩展的所有消息排队，直到扩展准备就绪。准备就绪后，会将排队的消息传输给扩展。重要的是，如果结果来自扩展自身发送的命令，则*不受*此限制；TEN runtime 会将命令的结果直接传递回扩展。这是因为发送命令是扩展的自我启动操作，因此它必须准备好接收并处理其自身命令的结果。

### on\_start

此阶段标志着运行时开始向扩展发送消息。`on_start_done` 函数没有特殊用途，但为了与其他生命周期阶段保持一致并降低学习复杂性，TEN framework 包含了它。通常，`on_start_done` 在 `on_start` 回调函数的末尾调用。TEN runtime 将在收到其 `on_start_done()` 后开始将来自其他扩展的消息传递到此扩展。

```c++
  void on_start(ten::ten_env_t &ten_env) override {
    // 一些操作。
    ten_env.on_start_done();
  }
```

### on\_stop

在许多情况下需要停止扩展，例如当包含扩展的应用程序或引擎终止时。当扩展即将停止时，TEN runtime 会使用此 `on_stop` 回调通知扩展它已进入 `on_stop` 生命周期阶段。然后，扩展可以执行任何必要的终止操作。

### on\_deinit

在扩展调用 `on_stop_done()` 后，它将进入 `on_deinit` 阶段。在此阶段，由于扩展内的资源可能不再完全可用，因此 TEN runtime 不会将来自其他扩展的任何消息传递到此扩展。

## 不同生命周期阶段的扩展之间的关系

基本上，没有固有的关系。每个扩展都独立运行，在其自身的生命周期阶段之间切换。扩展是彼此独立的，它们之间的任何依赖关系都必须由扩展本身显式实现。TEN runtime 不进行任何假设或保证。例如，如果扩展 A 需要等待扩展 B 完成其初始化才能完成自身的初始化，则扩展 A 可以在其 `on_init` 期间向扩展 B 发送命令。一旦扩展 B 完成初始化并收到命令，它可以回复结果，当扩展 A 收到结果时，它可以调用 `on_init_done`。

```mermaid
sequenceDiagram
  participant A as 扩展 A
  participant B as 扩展 B

  Note over A,B: on_init
  activate A
  activate B
  deactivate B
  Note over B: on_init_done

  Note over B: on_start
  activate B
  deactivate B
  Note over B: on_start_done

  A->>B: cmd
  B-->>A: result

  deactivate A
  Note over A: on_init_done
```

基本上，在所有其他生命周期阶段，如果想要实现不同扩展的顺序，可以使用相同的方法来实现。

## 与 TEN runtime 的接口

扩展主要通过三个接口与 TEN runtime 交互：

1. **生命周期回调**

   * 这些包括像 `on_init`、`on_deinit`、`on_start` 和 `on_stop` 这样的回调。

2. **接收消息的回调**

   * 这些包括诸如 `on_cmd`、`on_data`、`on_audio_frame` 和 `on_video_frame` 之类的回调，用于处理传入扩展的消息。

3. **发送消息的函数**

   * 这些包括诸如 `send_cmd`、`send_data`、`send_audio_frame` 和 `send_video_frame` 之类的函数，用于将消息发送出扩展。

## 生命周期回调函数

扩展生命周期的不同阶段及其与消息处理的联系如下：

* **on\_init \~ on\_init\_done**: 处理扩展自身的初始化。在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。
* **on\_start \~ on\_start\_done**: 在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。由于属性在 `on_configure` 期间初始化，因此您可以执行依赖于这些属性设置的操作。但是，由于这仍然是初始化阶段的一部分，扩展将不会收到其他扩展主动发起的消息，避免了各种检查的需要。允许主动发送消息。
* **After on\_start\_done \~ on\_stop\_done**: 在此阶段，扩展可以正常发送和接收所有类型的消息及其结果。
* **on\_deinit \~ on\_deinit\_done**: 处理扩展的反初始化。与 `on_init` 阶段类似，在此阶段，扩展可以发送消息并接收其发送的命令的结果，但不能接收其他扩展主动发送的消息。

## 使用不同语言实现扩展

在 TEN framework 中，可以使用各种语言（例如 C++、Go 和 Python）来实现扩展。开发者可以使用相同的概念方法以不同的语言实现扩展。学习如何使用一种语言开发扩展后，相对容易使用其他语言进行开发。

## 扩展中的异步消息处理

<figure><img src="../../..https://ten-framework-assets.s3.amazonaws.com/doc-assets/asynchronous_message_processing.png" alt=""><figcaption><p>异步消息处理</p></figcaption></figure>

扩展以异步方式处理消息。当 TEN runtime 通过 `on_cmd`、`on_data`、`on_audio_frame` 或 `on_video_frame` 等回调函数将消息传递给扩展时，并不要求扩展立即在回调函数内处理该消息。相反，扩展可以将消息委托给其他线程、进程，甚至其他机器进行处理。这可以充分利用多核和分布式计算资源。

处理完成后，可以通过 `send_cmd`、`send_data`、`send_audio_frame` 或 `send_video_frame` 等回调函数将结果发送回 TEN runtime。整个过程是异步的，这意味着扩展无需在 `on_cmd`、`on_data`、`on_audio_frame` 或 `on_video_frame` 回调函数返回之前就发送处理后的结果。结果只有在真正准备就绪时才可以使用适当的发送函数进行传输。


# 概述 (/cn/docs/ten_framework/0.11.60/software_packages/overview)

一个基本的 TEN framework 软件包的目录结构如下所示。

```text
.
├── bin/
├── src/
├── manifest.json
├── manifest-lock.json
├── property.json
└── ten_packages/
    ├── extension/
    │   ├── <extension_foo>/
    │   └── <extension_bar>/
    ├── extension_group/
    │   └── <extension_group_x>/
    ├── protocol/
    │   └── <protocol_a>/
    └── system/
        ├── <system_package_1>/
        └── <system_package_2>/
```

根据编程语言的不同，可能还会存在其他必需的文件。例如，在 C++ 项目中，由于构建系统是 `ten_gn`，因此会存在一个 `BUILD.gn` 文件。在 Go 项目中，会存在诸如 `go.mod` 和 `go.sum` 之类的文件。在 Python 项目中，你可能会找到 `requirements.txt` 文件。

## TEN framework 软件包类型

| 类型                    | 描述              |
| --------------------- | --------------- |
| 应用 (App)              | 包含一个 TEN 应用。    |
| 扩展组 (Extension group) | 包含一个 TEN 扩展组。   |
| 扩展 (Extension)        | 包含一个 TEN 扩展。    |
| 协议 (Protocol)         | 包含一个 TEN 协议。    |
| 系统 (System)           | 包含一个 TEN 系统软件包。 |

## TEN framework 软件包种类

有两种类型的软件包：

* 发布包 (Release package)
* 开发包 (Development package)

这两种软件包之间的关系如下：

* 发布包可以通过构建过程从开发包生成。
* 如果我们将与源代码相关的内容添加到发布包中，它就会变成一个开发包。
* 如果我们从开发包中删除与源代码相关的内容，它就会变成一个发布包。

简而言之，开发包和发布包的目录结构非常相似，区别在于开发包包含与源代码相关的内容，而发布包则不包含。

### 开发包

开发包的主要目的是开发 TEN 软件包。例如，如果您想创建一个新的 TEN 扩展，您可以修改现有的开发包来实现此目的。这是因为开发包包含所有与源代码相关的内容，可以从中创建新的 TEN 软件包。

### 发布包

发布包的主要目的是创建和运行图。由于不需要构建任务，因此可以使用发布包来执行这些操作。但是，开发包也可以用于这些任务。本质上，开发包的应用范围比发布包更广泛。


# Python 异步扩展 (/cn/docs/ten_framework/0.11.60/software_packages/python_async_extension)

**Python 异步扩展**是一个使用 Python 的 `asyncio` 框架的模块化组件。如果您想将使用 `asyncio` 的现有 Python 代码包装到 TEN 扩展中，那么使用 Python 异步扩展是最简单、最方便的选择。

## 示例：默认 Python 异步扩展

以下是 Python 异步扩展的结构方式：

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    async def on_configure(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_init(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_deinit(self, ten_env: AsyncTenEnv) -> None:
        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

    async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
        cmd_json = cmd.to_json()
        ten_env.log_debug(f"DefaultAsyncExtension on_cmd: {cmd_json}")

        # 模拟异步操作，例如网络、文件 I/O。
        await asyncio.sleep(0.5)

        # 向其他扩展发送新命令并等待结果。结果将返回给原始发送者。
        new_cmd = Cmd.create("hello")
        cmd_result = await ten_env.send_cmd(new_cmd)
        ten_env.return_result(cmd_result, cmd)
```

每种方法都使用 `await asyncio.sleep()` 模拟延迟。

## 常见问题

### 用于事件处理的异步循环

* 创建一个队列：使用 `asyncio.Queue`。
* 创建一个用于事件处理的异步任务。

这是示例代码：

```python
import asyncio
from ten_runtime import AsyncExtension, AsyncTenEnv

class DefaultAsyncExtension(AsyncExtension):
    queue = asyncio.Queue()
    loop:asyncio.AbstractEventLoop = None

    async def on_start(self, ten_env: AsyncTenEnv) -> None:
        self.loop = asyncio.get_event_loop()

        self.loop.create_task(self._consume())

    async def on_stop(self, ten_env: AsyncTenEnv) -> None:
        self.queue.put(None)

    async def _consume(self) -> None:
        while True
            try:
                value = await self.queue.get()
                if value is None:
                    self.ten_env.log_info("async loop exit")
                    break

                # 用于处理从队列检索的值的代码。

            except Exception as e:
                self.ten_env.log_error(f"Failed to handle {e}")
```

## 结论

TEN 的 Python 异步扩展提供了一种以异步方式处理长时间运行任务的强大方法。通过集成 Python 的 `asyncio` 框架，扩展能够确保诸如网络调用或文件处理之类的操作是高效且非阻塞的。这使得 TEN 成为构建具有异步功能的可扩展模块化应用程序的绝佳选择。


# 标准接口 (/cn/docs/ten_framework/0.11.60/standard_extension/standard_interface)

在 TEN framework 中，我们鼓励开发者创建自己的扩展（Extension）并将其发布到 TEN 扩展市场，供其他开发者使用。为了让开发者能够轻松地选择和替换扩展，我们需要定义标准化的接口规范，这样开发者就可以根据这些标准接口快速找到符合需求的扩展。

简而言之，标准接口是一种语法糖，它让 TEN 工具（如 TEN Manager）能够更方便地协助开发者进行扩展的选择和替换等操作。

TEN 的接口定义类似于 TypeScript 的 `.d.ts` 文件，让 TypeScript 编译器能够进行类型检查，但在实际运行时，这些接口定义并不会被使用。因此，标准接口设计带来的功能和代码增强，主要体现在 TEN Manager 这类开发工具上，而不是运行时的 TEN Runtime 上。例如，TEN Manager 需要提供相关的 RESTful API 给 tman designer Frontend，让用户可以根据标准接口定义，通过 UI 界面进行扩展的选择和替换。

## 什么是标准接口

在 TEN framework 中，每个扩展的接口都包含以下几个核心组成部分：

1. **Properties（属性）** - 扩展的配置参数
2. **Command（命令）**
   * **Input command** - 输入命令
   * **Output command** - 输出命令
3. **Data（数据）**
   * **Input data** - 输入数据
   * **Output data** - 输出数据
4. **Audio frame（音频帧）**
   * **Input audio frame** - 输入音频帧
   * **Output audio frame** - 输出音频帧
5. **Video frame（视频帧）**
   * **Input video frame** - 输入视频帧
   * **Output video frame** - 输出视频帧

### 标准接口的意义和用途

标准接口就是将这些组成部分进行规范化定义，形成统一的接口标准。开发者可以参考这些接口标准来实现自己的扩展，同时扩展也可以声明自己支持特定的标准接口，这样开发工具就能够根据扩展声明的标准接口提供更好的开发支持，使用户能够根据标准接口来选择或替换所需的扩展。

在标准接口中，Command/Data/Audio frame/Video frame 用于规范扩展之间的交互，实现扩展的互换性和兼容性。而 Properties 则用于规范扩展的配置参数，实现与 tman designer 的配置兼容性。

需要注意的是，规范扩展之间的交互并不是接口本身的功能，这是 Command/Data/Audio frame/Video frame 的功能。接口只是一个用于聚合的语法糖，帮助开发者快速指定扩展的接口定义。

### 接口的检查与报错

在扩展的 Command/Data/Audio frame/Video frame 定义中，如果存在两个同名的消息，且它们的 schema 定义相同，这不算错误，但会被视为重复定义并被忽略。这个规则在标准接口中同样适用，目的是让不同的标准接口可以各自定义共性功能，形成自包含的标准接口定义。当一个扩展实现这些具有相同定义的标准接口时，不会因此产生错误。这个特性在标准接口中非常有用，因为标准接口的定义可以被引用，所以当一个扩展实现多个标准接口时，如果这些标准接口中（或其引用的其他标准接口中）有相同的 message 和相同的 schema，也不会因此产生错误。

由于 TEN framework 采用运行与开发分离的架构（类似于 JavaScript 与 TypeScript 的 .d.ts 文件的关系），标准接口的检查与报错主要发生在开发工具阶段，而不是运行时的 TEN Runtime 上。因此，TEN Runtime 在启动图（start graph）时，不会因为接口错误而报错。

在 tman designer 这个开发工具中，如果扩展的接口定义不完整，会显示某些功能暂时无法在该扩展上使用的提示。例如，在使用某个扩展替代另一个扩展的功能时，如果接口定义不完整，会提示该功能暂时无法在这个扩展上使用，需要等待接口定义完整后才能使用。

另外，由于接口定义可以是一个被引用的独立文件，因此在 TEN Manager 的 publish/install 操作中，接口定义不完整并不算错误。

### 接口的兼容规则

当要将实现接口 A 的扩展替换为实现接口 B 的扩展时，需要接口 A 兼容于接口 B 才能进行替换。接口之间的兼容规则有两种模式：

**严格模式：**

* 接口 A 的所有消息都必须出现在接口 B 中，且接口 A 中消息的 schema 必须兼容于接口 B 中对应消息的 schema。

**宽松模式：**

* 只有存在连线（即有 connection）的接口 A 中的消息必须出现在接口 B 中，且接口 A 中这些消息的 schema 必须兼容于接口 B 中同名消息的 schema。

严格模式和宽松模式供用户选择，适用于不同的场景。例如，如果确定扩展实现的接口 A 中的某个消息 X 一定不会被使用，那么因为消息 X 的不兼容或不存在而导致无法替换为想要的另一个扩展可能过于严格。

## 标准接口的实现方式

TEN 扩展支持的 API 定义在其 `manifest.json` 文件中。标准接口的实现就是将定义好的 API 规范写入独立的 JSON 文件，然后在 `manifest.json` 中引用这个文件。

```json
{
  "type": "extension",
  "name": "foo_extension",
  "version": "0.1.0",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.1"
    }
  ],
  "api": {
    "interface": [
      {
        "import_uri": "https://xxx/asr_interface.json"  // <= 引用标准接口定义文件
      },
      {
        "import_uri": "https://xxx/tts_interface.json"  // <= 引用标准接口定义文件
      }
    ],
    "property": {
      "bar": {
        "type": "string"
      }
    },
    "cmd_in": [
      {
        "name": "query_vector",
        "property": {
          "collection_name": {
            "type": "string"
          },
          "top_k": {
            "type": "int64"
          },
          "embedding": {
            "type": "array",
            "items": {
              "type": "float64"
            }
          }
        },
        "required": [
          "collection_name",
          "top_k",
          "embedding"
        ],
        "result": {
          "property": {
            "response": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "content": {
                    "type": "string"
                  },
                  "score": {
                    "type": "float64"
                  }
                }
              }
            }
          }
        }
      }
    ],
    "cmd_out": [
      {
        "name": "publish",
        "property": {
          "message": {
            "type": "buf"
          }
        }
      },
      {
        "name": "set_presence_state"
      }
    ],
    "data_in": [
      {
        "name": "data"
      }
    ]
  }
}
```

### 标准接口定义文件示例

当 TEN 系统（包括 TEN Runtime、TEN Manager 等）解析 `manifest.json` 时，会根据 `import_uri` 下载对应的 JSON 文件，并将其内容合并到 `manifest.json` 中。被引用的标准接口文件内容示例如下，可以看到内容与 manifest.json 中的 `api` 字段相同，只是被独立成一个文件以便引用。并且这个独立的标准接口文件也可以再引用其他的标准接口文件。

```json
{
  "interface": [
    {
      "import_uri": "https://xxx/stt_interface.json"// <= 引用其他的标准接口定义文件
    }
  ],
  "property": {
    "bar": {
      "type": "string"
    }
  },
  "cmd_in": [
    {
      "name": "query_vector",
      "property": {
        "collection_name": {
          "type": "string"
        },
        "top_k": {
          "type": "int64"
        },
        "embedding": {
          "type": "array",
          "items": {
            "type": "float64"
          }
        }
      },
      "required": [
        "collection_name",
        "top_k",
        "embedding"
      ],
      "result": {
        "property": {
          "response": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string"
                },
                "score": {
                  "type": "float64"
                }
              }
            }
          }
        }
      }
    }
  ],
  "cmd_out": [
    {
      "name": "publish",
      "property": {
        "message": {
          "type": "buf"
        }
      }
    },
    {
      "name": "set_presence_state"
    }
  ],
  "data_in": [
    {
      "name": "data"
    }
  ]
}
```

### 接口合并规则

系统会将 `import_uri` 指向的 JSON 文件内容合并到 `manifest.json` 的 `api` 字段中，具体包括：

* `property`
* `cmd_in`、`cmd_out`
* `data_in`、`data_out`
* `audio_frame_in`、`audio_frame_out`
* `video_frame_in`、`video_frame_out`

**重要规则：**

1. **冲突检测**：如果合并过程中发现同名的同类字段（比如同名的 `cmd_in` 或同名的 `property`），如果 schema 相同则不算错误，否则系统会报错。
2. **递归合并**：合并过程是递归进行的，如果检测到循环引用，系统会报错。
3. **多接口支持**：一个扩展可以支持多个标准接口，在 `manifest.json` 中定义多个 `interface` 字段即可。
4. **重复引用检测**：如果不同的 `interface` 字段引用了相同的 `import_uri`，系统会报错。

## 标准接口的存储方式

标准接口文件支持以下几种存储方式：

<div className="fd-steps">
  <div className="fd-step">
    ### TEN 系统包（TEN System Package）

    **概述**：这种方式是将标准接口定义文件打包成 TEN 系统包，然后在 `manifest.json` 中引用。同时也可以在 manifest.json 的 dependencies 字段中依赖这个系统包，这样在安装扩展时就会一并安装所依赖的标准接口系统包，从而提供最佳的开发体验。通过这种方法，可以利用 TEN 包的版本机制来实现标准接口的版本管理，并且可以利用 TEN 包的依赖机制来实现标准接口的自动安装。

    使用 TEN 系统包，除了标准接口的 JSON 定义文件外，还可以包含相关的资源，例如模型文件、公共工具函数库、配套工具等。因此，当所有这些资源出现不兼容的变更时，都可以通过 TEN 系统包的版本号变更来更好地体现这种变化。例如，系统包内 JSON 文件的存放位置或者文件名称的变化，也算是该系统包的破坏性变更，因此应该升级该系统包的主版本号。

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "0.1.0"
        }
      ],
      "api": {
        "interface": [
          {
            "import_uri": "../../../ten_packages/system/foo_interface/interface.json"
          }
        ]
      }
    }
    ```

    **使用 TEN 系统包的优势：**

    1. **易于分发**：系统包可以上传到 TEN 扩展市场，用户可以直接下载使用
    2. **资源丰富**：包中不仅包含接口定义文件，还可以包含相关资源，如：
       * 模型文件
       * 公共工具函数库
       * 配套工具等
    3. **依赖管理**：系统包可以作为扩展的依赖项，当扩展被下载时，相关的系统包也会自动下载
    4. **版本管理**：不仅限于标准接口协议本身，该系统包内的任何资源都是该系统包版本管理的一部分

    需要注意的是，TEN 系统包的版本号并不仅仅代表标准接口的版本号，而是代表整个系统包的版本号。因此，如果系统包内的资源有破坏性变更，就应该升级系统包的主版本号。例如，系统包内 JSON 文件的存放位置或者文件名称的变化，即使没有改变 JSON 文件的内容，也算是该系统包的破坏性变更，因为对使用者来说，使用方式需要改变，所以应该升级该系统包的主版本号。

    因此，不能仅仅通过系统包的版本号来判断标准接口的兼容性。标准接口的兼容性还是需要通过接口的兼容规则来判断。

    #### 支持多版本共存

    > **注意**：这一节所描述的功能，基本上不会实现，只是描述如果之后要做的话，可能的实现方法以及需要考虑的点。

    **概述**：在某些场景下，一个 TEN 应用可能需要同时使用同一标准接口的不同版本。为了支持这种需求，TEN 允许在同一个应用内安装同一个系统包的多个版本。

    **系统包配置：**

    首先，在系统包的 `manifest.json` 中声明支持多版本共存：

    ```json
    {
      "type": "system",
      "name": "foo_interface",
      "version": "0.1.0",
      "install_policy": {
        "coinstallable": "true"
      }
    }
    ```

    **扩展依赖配置：**

    在依赖该系统包的扩展的 `manifest.json` 中，也需要声明使用多版本共存：

    ```json
    {
      "dependencies": [
        {
          "type": "system",
          "name": "foo_interface",
          "version": "=0.1.0", // 注意：这里只能是 equal 的语法，不能用 >= 之类的语法
          "install_policy": {
            "coinstallable": "true"
          }
        }
      ],
    }
    ```

    并且在整个依赖树的构建过程中，如果一个系统包的 coinstallable 为 true，则需要所有依赖该系统包的地方都声明是 coinstallable，否则系统会报错。

    **文件系统路径：**

    启用多版本共存后，系统包在文件系统中的路径会变为：

    ```text
    ten_packages/system/<hash_value>_foo_extension_0_1_0/
    ```

    > **注意**：未来可能会引入映射机制，让用户在使用时不需要明确指定 hash\_value，而是通过某种映射机制自动找到对应的系统包。但这种机制如果要好用，还需要修改 TEN 所支持的语言中的模块加载机制，使得开发者可以不用在该语言中的模块加载机制中指定 hash\_value，而是通过某种映射机制自动找到对应的系统包。这种深入语言内部的改动也是这种单包多版本共存的机制基本上不会被实现的原因之一。
    >
    > **注意**：如果真的需要在一个 TEN 应用内使用多个版本的一个标准接口，可以使用下述的几种方式来实现。如果一个标准接口有配套的公用资源，例如模型文件、公共工具函数库、配套工具等，那么这些资源很多无法做到在一个 TEN 应用内的多版本共存，这也是这种单包多版本共存的机制基本上不会被实现的原因之一。
  </div>

  <div className="fd-step">
    ### 网络地址

    **概述**：将标准接口定义文件部署到网络服务器上，通过 HTTP/HTTPS 地址引用：

    ```json
    {
      "interface": [
        {
          "import_uri": "http://xxx/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    > **注意**：TEN Manager 在处理这种引用方式时，可以将文件下载到自己的缓存中，然后通过某种映射机制，让 TEN Manager 能够找到定义。

    **适用场景：**

    * **轻量级接口**：没有配套的公共资源的接口定义
    * **版本灵活性**：需要在一个 `manifest.json` 中引用同一标准接口的不同版本
  </div>

  <div className="fd-step">
    ### 本地路径

    **概述**：引用本地文件系统中的标准接口定义文件：

    **绝对路径：**

    ```json
    {
      "interface": [
        {
          "import_uri": "file://c:/asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **相对路径：**

    ```json
    {
      "interface": [
        {
          "import_uri": "../asr_1.0_interface/interface.json"
        }
      ]
    }
    ```

    **适用场景：**

    与网络地址方式类似：

    * **轻量级接口**：适合没有配套的公共资源的接口定义
    * **版本灵活性**：需要在一个 `manifest.json` 中引用同一标准接口的不同版本
  </div>
</div>

## 如何从云商店中筛选想要的扩展

标准接口与云商店的关系体现在将标准接口包装成 TEN 系统包。有以下两种方式可以搜索出符合特定条件的扩展：

1. **依赖搜索**：搜索依赖某个系统包的扩展。由于云商店中扩展的元数据包含了依赖信息，所以云商店可以据此筛选出依赖某个系统包的所有扩展。

2. **标签搜索**：通过 TEN package 的 tag 来快速搜索出符合某些条件的扩展，选择安装后，再进行替换操作。如果是 TEN 官方认证过的扩展，甚至可以打上 `ten:` 开头的标签来表明这是一个官方认证过的扩展，确实符合所声明的标准接口定义。


# 图检查 (/cn/docs/ten_framework/0.11.60/ten_manager/check_graph)

`tman` 提供了 `check graph` 命令，用于验证预定义图或 `start_graph` 命令的正确性。要查看使用细节，请使用以下命令：

```shell
$ tman check graph -h
检查预定义图或 start_graph 命令的图内容是否正确。有关更详细的用法，请运行“graph -h”

用法：tman check graph [选项] --app <APP>

选项：
      --app <APP>
          图中定义的应用程序的绝对路径。默认情况下，将从列表中的第一个应用程序读取预定义图。
      --predefined-graph-name <PREDEFINED_GRAPH_NAME>
          指定要检查的预定义图的名称，否则将检查所有预定义图。
      --graph <GRAPH>
          指定要检查的“start_graph”命令的 JSON 字符串。如果未指定，则将检查第一个应用程序中的预定义图。
  -h, --help
          打印帮助信息
```

`check graph` 命令旨在处理可能跨越多个应用程序的图，因此不需要从特定应用程序的根目录运行。相反，它可以从任何目录执行，并使用 `--app` 参数指定图中涉及的应用程序的文件夹。

## 使用示例

* **检查 `property.json` 中的所有预定义图：**

  ```shell
  tman check graph --app /home/TEN-Agent/agents
  ```

* **检查特定的预定义图：**

  ```shell
  tman check graph --predefined-graph-name va.openai.azure --app /home/TEN-Agent/agents
  ```

* **检查 `start_graph` 命令：**

  ```shell
  tman check graph --graph '{
    "type": "start_graph",
    "seq_id": "55",
    "nodes": [
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_2__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      },
      {
        "type": "extension",
        "name": "test_extension",
        "addon": "basic_hello_world_1__test_extension",
        "extension_group": "test_extension_group",
        "app": "msgpack://127.0.0.1:8001/"
      }
    ]
  }' --app /home/TEN-Agent/agents
  ```

## 前提条件

* **预定义图定义要求：** 如果指定了预定义图的名称，则将从第一个 `--app` 参数指定的应用程序的 `property.json` 文件中提取其定义。
* **包安装要求：** 在运行 `check graph` 命令之前，必须使用 `tman install` 命令安装应用程序依赖的所有扩展。这是必需的，因为验证过程需要有关图中每个扩展的信息，例如其 `manifest.json` 文件中定义的 API。
* **唯一应用程序 URI 要求：** 在多应用程序图中，每个应用程序的 `property.json` 必须定义唯一的 `ten::uri`。此外，`uri` 值不能设置为 `"localhost"`。

## 验证规则

<div className="fd-steps">
  <div className="fd-step">
    ### 节点 (Nodes) 的存在性

    任何图定义中都必须存在 `nodes` 数组。如果缺少该数组，将会抛出错误。

    * **示例（未定义节点）：**

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": []
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        No extension node is defined in graph.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 节点的唯一性

    `nodes` 数组中的每个节点表示一个应用程序组中由指定插件创建的特定扩展实例。因此，每个扩展实例都应该由一个单独的节点唯一表示。必须通过 `app`、`extension_group` 和 `name` 的组合来唯一标识一个节点。不允许出现同一扩展实例的多个条目。

    * **示例（重复节点）：**

      ```shell
      tman check graph --graph '{
        "type": "start_graph",
        "seq_id": "55",
        "nodes": [
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_2__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          },
          {
            "type": "extension",
            "name": "test_extension",
            "addon": "basic_hello_world_1__test_extension",
            "extension_group": "test_extension_group",
            "app": "msgpack://127.0.0.1:8001/"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        Duplicated extension was found in nodes[1], addon: basic_hello_world_1__test_extension, name: test_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在通信链路中使用的扩展应在 nodes 中定义

    `connections` 字段中引用的所有扩展实例（无论是作为源还是目标）都必须在 `nodes` 字段中显式定义。任何未在 `nodes` 数组中定义的实例都会导致验证错误。

    * **示例（未定义源扩展）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0] is not defined in nodes, extension_group: producer, extension: some_extension.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

    * **示例（未定义目标扩展）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                },
                {
                  "type": "extension",
                  "name": "some_extension_1",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ],
              "connections": [
                {
                  "extension": "some_extension",
                  "cmd": [
                    {
                      "name": "hello",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        }
                      ]
                    },
                    {
                      "name": "world",
                      "dest": [
                        {
                          "extension": "some_extension_1"
                        },
                        {
                          "extension": "consumer"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The extension declared in connections[0].cmd[1] is not defined in nodes extension_group: some_group, extension: consumer.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 `nodes` 中声明的插件必须安装在应用中

    * **示例（`property.json` 中的 `ten::uri` 与 `nodes` 中的 `app` 字段不相等）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ],
          "uri": "http://localhost:8001"
        }
      }
      ```

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```

      问题在于，应用程序中的所有包都将存储在以应用程序的 `uri` 为键的映射中，并且图中的每个节点都将通过 `app` 字段（默认为 `localhost`）检索。节点中的 `app`（即 `localhost`）与应用程序的 `uri`（即 `http://localhost:8001`）不匹配。

    * **示例（由于未执行 `tman install` 而导致 `ten_packages` 目录不存在）：**

      假设一个 TEN 应用程序的 `property.json` 文件的内容如下。

      ```json
      {
        "ten": {
          "predefined_graphs": [
            {
              "name": "default",
              "auto_start": false,
              "nodes": [
                {
                  "type": "extension",
                  "name": "some_extension",
                  "addon": "default_extension_go",
                  "extension_group": "some_group"
                }
              ]
            }
          ]
        }
      }
      ```

      并且 `ten_packages` 目录**不存在**。

      ```shell
      tman check graph --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        The following packages are declared in nodes but not installed: [("localhost", Extension, "default_extension_go")].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 通信链路中，从一个扩展发送的消息应在同一部分中定义

    * **示例：**

      假设所有包都已安装。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "default_extension_go",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          },
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello_2",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        extension 'some_extension' is defined in connection[0] and connection[1], merge them into one section.

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 在 通信链路中，从一个扩展发出的消息在每个类型中应具有唯一的名称

    * **示例：**

      假设所有包都已安装。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              },
              {
                "name": "hello",
                "dest": [
                  {
                    "extension": "some_extension"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        - connection[0]:
         - Merge the following cmd into one section:
            'hello' is defined in flow[0] and flow[1].

      All is done.
      ❌  Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 通信链路中声明的消息应兼容

    根据扩展的 `manifest.json` 文件中的模式定义，将检查在 `connections` 中的每个消息流中声明的消息的模式是否兼容。消息兼容的规则如下。

    * 如果在源扩展和目标扩展中都找不到消息模式，则消息是兼容的。

    * 如果仅在其中一个扩展中找到消息模式，则该消息不兼容。

    * 只有满足以下条件，消息才是兼容的：

      * 属性类型兼容。
        * 如果属性是一个对象，则源模式和目标模式中的字段都必须具有兼容的类型。
        * 如果在目标模式中定义了 `required` 关键字，则源模式中必须存在一个 `required` 关键字，该关键字是目标中 `required` 的超集。

    * **示例：**

      假设所有包都已安装。

      `addon_a` 中的 `manifest.json` 的内容如下。

      ```json
      {
        "type": "extension",
        "name": "addon_a",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_out": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "string"
                }
              }
            }
          ]
        }
      }
      ```

      并且，`addon_b` 中的 `manifest.json` 的内容如下。

      ```json
      {
        "type": "extension",
        "name": "addon_b",
        "version": "0.1.0",
        "dependencies": [
          {
            "type": "system",
            "name": "ten_runtime_go",
            "version": "0.1.0"
          }
        ],
        "package": {
          "include": ["**"]
        },
        "api": {
          "cmd_in": [
            {
              "name": "cmd_1",
              "property": {
                "foo": {
                  "type": "int8"
                }
              }
            }
          ]
        }
      }
      ```

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      Checking graph[0]... ❌. Details:
        - connections[0]:
          - cmd[0]:  Schema incompatible to [extension_group: some_group, extension: another_ext], { .foo: type is incompatible, source is [string], but target is [int8] }

      All is done.
      ❌ Error: 1/1 graphs failed.
      ```
  </div>

  <div className="fd-step">
    ### 节点中的 `app` 必须明确

    每个节点中的 `app` 字段必须满足以下规则。

    * `app` 字段必须等于相应 TEN 应用程序的 `ten::uri`。

    * 要么所有节点都应该声明 `app`，要么都不应该声明。

    * `app` 字段不能为 `localhost`。

    * `app` 字段不能为空字符串。

    * **示例（某些节点指定了 `app` 字段）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        Either all nodes should have 'app' declared, or none should, but not a mix of both.
      ```

    * **示例（所有节点均未指定 `app`，但某些源扩展在通信链路中指定了 `app`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
         {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].the 'app' should not be declared, as not any node has declared it
      ```

    * **示例（所有节点均未指定 `app`，但某些目标扩展在通信链路中指定了 `app`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext",
                    "app": "http://localhost:8000"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: The graph json string is invalid

      Caused by:
        connections[0].cmd[0].dest[0]: the 'app' should not be declared, as not any node has declared it
      ```

    * **示例 (节点中的 `app` 字段与应用程序的 `ten::uri` 不相等)：**

      与 [规则 4](#id-4.-the-addons-declared-in-the-nodes-must-be-installed-in-the-app) 相同。

    * **示例（单应用图中 `app` 字段为 `localhost`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, and the graph seems to be a single-app graph, just remove the 'app' field
      ```

    * **示例（多应用图中 `app` 字段为 `localhost`）：**

      使用以下命令检查图。

      ```shell
      tman check graph --graph '{
        "nodes": [
          {
            "type": "extension",
            "name": "some_extension",
            "addon": "addon_a",
            "extension_group": "some_group",
            "app": "http://localhost:8000"
          },
          {
            "type": "extension",
            "name": "another_ext",
            "addon": "addon_b",
            "extension_group": "some_group",
            "app": "localhost"
          }
        ],
        "connections": [
          {
            "extension": "some_extension",
            "app": "http://localhost:8000",
            "cmd": [
              {
                "name": "cmd_1",
                "dest": [
                  {
                    "extension": "another_ext"
                  }
                ]
              }
            ]
          }
        ]
      }' --app /home/TEN-Agent/agents
      ```

      **输出：**

      ```text
      ❌  Error: Failed to parse graph string, nodes[1]: 'localhost' is not allowed in graph definition, change the content of 'app' field to be consistent with 'ten::uri'
      ```
  </div>
</div>


# 概述 (/cn/docs/ten_framework/0.11.60/ten_manager/overview)

在 TEN 生态系统中，几乎所有事物都是组件。例如，一个 TEN 应用是一种组件，一个 TEN 扩展是另一种组件，而一个 TEN 协议又是另一种组件。为了帮助 TEN 开发者管理与这些组件相关的所有操作，我们提供了一个至关重要的工具，称为 TEN 管理器。

<Callout type="info">
  `tman`

   是 TEN 管理器 (TEN Manager) 的缩写。
</Callout>

TEN 管理器包含一个 TEN 软件包管理器，旨在简化 TEN 软件包的管理。此软件包管理器处理诸如确定哪些依赖包是必需的、获取这些依赖包以及安装它们等任务。简而言之，TEN 管理器简化了从头开始创建新的 TEN 软件包的过程，并使从 TEN 云商店安装 TEN 软件包更加容易。


# 调试 (/cn/docs/ten_framework/0.11.60/testing_and_debugging/debugging)

## 适用于 TEN framework 的开发人员

如果您使用 VS Code 开发 TEN framework，则 TEN framework 源代码树中有一个 `launch.json` 文件，其中提供了默认的调试目标。

## 适用于 TEN framework 的用户

如果您使用 VS Code 开发基于 TEN framework 的自己的应用程序，您可以将配置添加到您的 `.vscode/launch.json` 文件，以便您可以使用断点和变量检查来调试您的应用程序，无论您使用的是哪种编程语言（C++/Golang/Python）。

### 在 C++ 应用程序中调试

如果应用程序是用 C++ 编写的，这意味着扩展可以使用 C++ 或 Python 编写。

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

* **使用 lldb 调试 C++ 代码**

  ```json
  {
    "name": "app (C++) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    }
  }
  ```

* **使用 gdb 调试 C++ 代码**

  ```json
  {
    "name": "app (C++) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### 使用 debugpy 调试 Python 代码

您可以使用 `debugpy` 调试 Python 代码。 但是，由于程序不是直接通过 Python 解释器启动的，因此 Python 代码由嵌入的 Python 解释器执行。 因此，您只能将调试器附加到正在运行的进程。

首先，您需要在 Python 环境中安装 `debugpy`：

```shell
pip install debugpy
```

然后，您需要使用设置的环境变量 `TEN_ENABLE_PYTHON_DEBUG` 和 `TEN_PYTHON_DEBUG_PORT` 启动应用程序。 例如：

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

如果环境变量 `TEN_ENABLE_PYTHON_DEBUG` 设置为 `true`，则应用程序将阻塞，直到附加调试器。 如果设置了环境变量 `TEN_PYTHON_DEBUG_PORT`，则调试器服务器将侦听指定端口上的传入连接。

然后，您可以将调试器附加到正在运行的进程。 以下是一个配置示例：

**使用 debugpy 调试 Python 代码**

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### 同时调试 C++ 和 Python 代码

如果您想在 VSCode 中一键启动同时调试 C++ 和 Python 代码，您可以执行以下操作：

1. 定义一个任务，在附加调试器之前延迟几秒钟：

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. 将以下配置添加到 `launch.json` 文件：

   ```json
   "configurations": [
        {
            "name": "app (C++) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch", // "launch" or "attach"
            "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
            "cwd": "${workspaceFolder}/",
            "env": {
                "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
                "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (C++) (debugpy, attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "justMyCode": false,
            "preLaunchTask": "delay 3 seconds"
        }
   ],
   "compounds": [
        {
            "name": "app (C++) (lldb, launch) and app (C++) (debugpy, attach)",
            "configurations": [
                "app (C++) (lldb, launch with debugpy)",
                "app (C++) (debugpy, attach with delay)"
            ]
        }
   ]
   ```

然后，您可以使用复合配置 `app (C++) (lldb, launch) and app (C++) (debugpy, attach)` 启动调试 C++ 和 Python 代码。

### 在 Go 应用程序中调试

如果应用程序是用 Go 编写的，这意味着扩展可以使用 Go 或 C++ 或 Python 编写。

#### 使用 delve 调试 Go 代码

您可以使用以下配置在您的 Go 扩展中进行调试：

```json
{
  "name": "app (golang) (go, launch)",
  "type": "go",
  "request": "launch",
  "mode": "exec",
  "cwd": "${workspaceFolder}/",
  "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
  "env": {
    "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
    "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // macOS
    "TEN_APP_BASE_DIR": "${workspaceFolder}"
  }
}
```

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

* **使用 lldb 调试 C++ 代码**

  ```json
  {
    "name": "app (Go) (lldb, launch)",
    "type": "lldb",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "env": {
      "LD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib", // linux
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/ten_packages/system/xxx/lib" // macOS
    },
    "initCommands": ["process handle SIGURG --stop false --pass true"]
  }
  ```

* **使用 gdb 调试 C++ 代码**

  ```json
  {
    "name": "app (Go) (gdb, launch)",
    "type": "cppdbg",
    "request": "launch", // "launch" or "attach"
    "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
    "cwd": "${workspaceFolder}/",
    "MIMode": "gdb",
    "environment": [
      {
        // linux
        "name": "LD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      },
      {
        // macOS
        "name": "DYLD_LIBRARY_PATH",
        "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
      }
    ]
  }
  ```

#### 使用 debugpy 调试 Python 代码

请参阅上面的部分，了解如何使用 debugpy 调试 Python 代码。 使用设置的环境变量 `TEN_ENABLE_PYTHON_DEBUG` 和 `TEN_PYTHON_DEBUG_PORT` 启动应用程序。

```shell
TEN_ENABLE_PYTHON_DEBUG=true TEN_PYTHON_DEBUG_PORT=5678 ./bin/worker
```

然后，您可以将调试器附加到正在运行的进程。 以下是一个配置示例：

```json
{
  "name": "app (C++) (debugpy, attach)",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "justMyCode": false
}
```

#### 同时调试 C++、Go 和 Python 代码

如果您想在 VSCode 中一键启动同时调试 C++、Go 和 Python 代码，您可以执行以下操作：

1. 定义一个任务，在附加调试器之前延迟几秒钟：

   ```json
   {
     "label": "delay 3 seconds",
     "type": "shell",
     "command": "sleep 3",
     "windows": {
       "command": "ping 127.0.0.1 -n 3 > nul"
     },
     "group": "none"
   }
   ```

2. 将以下配置添加到 `launch.json` 文件：

   ```json
   "configurations": [
        {
            "name": "app (golang) (debugpy, remote attach with delay)",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "preLaunchTask": "delay 3 seconds",
            "justMyCode": false
        },
        {
            "name": "app (golang) (go, launch with debugpy)",
            "type": "go",
            "request": "launch",
            "mode": "exec",
            "cwd": "${workspaceFolder}/",
            "program": "${workspaceFolder}/bin/worker",
            "env": {
                "TEN_APP_BASE_DIR": "${workspaceFolder}",
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            }
        },
        {
            "name": "app (golang) (lldb, launch with debugpy)",
            "type": "lldb",
            "request": "launch",
            "program": "${workspaceFolder}/bin/worker",
            "cwd": "${workspaceFolder}/",
            "env": {
                "TEN_ENABLE_PYTHON_DEBUG": "true",
                "TEN_PYTHON_DEBUG_PORT": "5678"
            },
            "initCommands": [
                "process handle SIGURG --stop false --pass true"
            ]
        },
   ],
   "compounds": [
        {
            "name": "Mixed Go/Python",
            "configurations": [
                "app (golang) (go, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ],
        },
        {
            "name": "Mixed Go/Python/C++ (lldb)",
            "configurations": [
                "app (golang) (lldb, launch with debugpy)",
                "app (golang) (debugpy, remote attach with delay)"
            ]
        }
   ]
   ```

如果您想同时调试 C++ 和 Python 代码，您可以使用复合配置 `Mixed Go/Python/C++ (lldb)`。 使用此配置，您可以同时对 C++、Go 和 Python 代码执行断点调试。 但是，变量检查仅支持 C++ 和 Python 代码，而不支持 Go 代码。

如果您想同时调试 Go 和 Python 代码，您可以使用复合配置 `Mixed Go/Python`。 使用此配置，您可以同时对 Go 和 Python 代码执行断点调试，并检查 Go 和 Python 代码的变量。

### 在 Python 应用程序中调试

如果应用程序是用 Python 编写的，这意味着扩展可以使用 Python 或 C++ 编写。

#### 使用 lldb 或 gdb 调试 C++ 代码

您可以使用以下配置来调试 C++ 代码：

```json
{
  "name": "app (Python) (cpp, launch)",
  "type": "cppdbg", //
  "request": "launch", // "launch" or "attach"
  "program": "/usr/bin/python3", // Python 解释器路径
  "args": ["main.py"],
  "cwd": "${workspaceFolder}/",
  "environment": [
    {
      "name": "PYTHONPATH",
      "value": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface"
    },
    {
      "name": "TEN_APP_BASE_DIR",
      "value": "${workspaceFolder}/"
    }
  ],
  "MIMode": "gdb" // "gdb" 或 "lldb"
}
```

#### 使用 debugpy 调试 Python 代码

请参阅上面的部分，了解如何使用 debugpy 调试 Python 代码。 您可以使用以下配置来调试 Python 代码：

```json
{
  "name": "app (Python) (debugpy, launch)",
  "type": "debugpy",
  "python": "/usr/bin/python3",
  "request": "launch",
  "program": "${workspaceFolder}/main.py",
  "console": "integratedTerminal",
  "cwd": "${workspaceFolder}/",
  "env": {
    "PYTHONPATH": "${workspaceFolder}/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/ten_packages/system/ten_runtime_python/interface",
    "TEN_APP_BASE_DIR": "${workspaceFolder}/",
    "TEN_ENABLE_PYTHON_DEBUG": "true"
  }
}
```

#### 同时调试 C++ 和 Python 代码

如果您想同时启动 C++ 和 Python 代码的调试，您可以使用配置 `app (Python) (debugpy, launch)` 启动应用程序，并使用以下配置将调试器附加到正在运行的进程：

```json
{
  "name": "app (Python) (cpp, attach)",
  "type": "cppdbg",
  "request": "attach", // "launch" 或 "attach"
  "program": "${workspaceFolder}/bin/worker", // 可执行文件的路径
  "cwd": "${workspaceFolder}/",
  "MIMode": "gdb",
  "environment": [
    {
      "name": "LD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    },
    {
      "name": "DYLD_LIBRARY_PATH",
      "value": "${workspaceFolder}/ten_packages/system/xxx/lib"
    }
  ]
}
```


# 面向 TEN framework 开发者 (/cn/docs/ten_framework/0.11.60/testing_and_debugging/for_developers_of_ten_framework)

TEN framework 包括三种类型的测试套件：

* 单元测试
* 冒烟测试
* 集成测试

TEN framework 使用 `gtest` 作为单元测试和冒烟测试的测试框架，使用 `pytest` 作为集成测试的测试框架。

## 单元测试

单元测试的源目录是 `tests/ten_runtime/unit`。如果您需要添加新的单元测试用例，请将它们放在 `tests/ten_runtime/unit directory` 中。

对于 C++ 项目，在成功编译后，您可以导航到输出目录（例如，`out/linux/x64/tests/standalone/`，具体取决于目标操作系统和 CPU）并执行以下命令：

```shell
./ten_runtime_unit_test
```

## 冒烟测试

冒烟测试的源目录是 `tests/ten_runtime/smoke`。如果您需要添加新的冒烟测试用例，请将它们放在 `tests/ten_runtime/smoke` 目录中。

导航到输出目录（例如，`out/linux/x64/tests/standalone`，具体取决于目标操作系统和 CPU）并执行以下命令：

```shell
./ten_runtime_smoke_test
```

## 循环测试

您可以使用以下命令执行多轮测试。

仅运行单元测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;
```

仅运行冒烟测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

同时运行单元测试和冒烟测试：

```shell
failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;
```

## 集成测试

集成测试是 TEN framework 的黑盒测试。集成测试的源目录是 `tests/ten_runtime/integration`。这些测试用于验证实际的执行场景。

集成测试的文件结构如下：

```text
tests/ten_runtime/integration/
  ├── test_1/
  │    ├── test_case.py
  │    └── ...
  ├── test_2/
  │    ├── test_case.py
  │    └── ...
  └── ...
```

要执行集成测试，请使用以下命令：

```shell
cd path/to/out/
pytest tests/ten_runtime/integration/
```

## 测试技巧

### Linux

在 Linux 上，您可以使用以下技术来减慢 TEN 的执行速度，这有助于触发与时序相关的错误。

```shell
sudo apt install util-linux
cpulimit -f -l 50 -- taskset 0x3 ...
```

* `taskset 0x1`: 仅使用 1 个 CPU 核心来运行程序。
* `taskset 0x3`: 仅使用 2 个 CPU 核心来运行程序。


# 面向 TEN framework 用户 (/cn/docs/ten_framework/0.11.60/testing_and_debugging/for_users_of_ten_framework)

## 扩展的独立测试

TEN framework 提供了一种独立的扩展测试机制。该机制允许开发者在不依赖其他 TEN 概念（例如其他 TEN 扩展、TEN 图或 TEN 应用）的情况下测试扩展。当开发者需要在不运行整个 TEN 应用的情况下测试扩展的行为时，这种独立的扩展测试机制尤其有用。

TEN 扩展的独立测试框架遵循两个关键原则：

1. **与所用语言的任何原生测试框架兼容。**

   例如，如果使用 C++ 开发 TEN 扩展，则 Google gtest/gmock 测试框架可以与 TEN 扩展独立测试框架一起使用，以实现 C++ TEN 扩展的独立测试。

2. **用户无需修改被测扩展中的任何代码。**

   可以使用此独立测试框架对运行时使用的完全相同的扩展代码进行全面测试。

3. **以不同语言开发的扩展的独立测试流程遵循相同的设计原则和使用方法。**

   这确保一旦您学习了一种语言的独立测试概念和方法，就可以快速掌握其他语言的 TEN 扩展的独立测试概念和方法。

对于用户来说，TEN 扩展独立测试框架引入了两个主要概念：

1. **`extension_tester`**
2. **`ten_env_tester`**

`extension_tester` 的作用类似于一个测试驱动程序，负责设置和执行整个测试过程。另一方面，`ten_env_tester` 的行为类似于典型的 TEN 扩展的 `ten_env` 实例，使用户能够从 `extension_tester` 的回调中调用独立测试框架中的功能，例如向被测扩展发送消息和从被测扩展接收消息。

从 `extension_tester` 和 `ten_env_tester` 的 API 设计中，您可以看到它们与 **TEN 扩展** 和 `ten_env` 非常相似，这是有意为之的。主要目的是让熟悉扩展开发的用户能够快速适应独立测试框架，然后为扩展本身开发独立的测试用例。

然而，出于测试目的，不可避免地会有专门用于测试的 API 和功能。为了防止这些在实际运行时不需要的、专用于测试的功能和 API 污染运行平台 API 集，独立测试框架被设计为不直接使用或扩展 **extension** 和 `ten_env` 的 API 集。相反，它引入了独立于测试的类型和 API 集。这种分离确保了实际运行时使用的类型和 API 集与测试期间使用的类型和 API 集互不干扰，避免了任何负面影响。

## 独立测试框架内部机制

在内部，TEN 扩展独立测试框架会隐式启动一个测试应用，加载要测试的扩展插件（我们称之为扩展 A），并初始化一个包含要测试的扩展 A 和另一个专门用于测试其他扩展的扩展（我们称之为扩展 B）的图。扩展 A 的所有输入和输出消息都被重定向到扩展 B，允许用户在测试过程中自定义输入和输出，并完成实际的测试工作。

```mermaid
sequenceDiagram
  participant tester as Extension Tester
  participant testing as Extension Testing
  participant tested as Extension Tested

  tester->>testing: send messages
  testing->>tested: send messages

  tested-->>testing: send results
  testing-->>tester: send results
```

基本上，您可以将此隐藏在测试框架中的测试扩展视为被测试扩展和测试器之间的代理。它充当 TEN 图中的代理扩展，使用 TEN 环境的语言促进被测扩展和测试器之间的消息交换。

## 基本测试流程

基本测试流程和逻辑如下：

1. 创建一个扩展测试器来管理整个独立测试过程。
2. 向独立测试框架提供包含待测扩展的文件夹信息。
3. 设置测试模式，例如用于测试单个扩展的模式。
4. 开始测试。

## C++

以下是使用 Google gtest 进行 TEN 扩展独立测试的示例：

```c++
class extension_tester_basic : public ten::extension_tester_t {
 public:
  void on_start(ten::ten_env_tester_t &ten_env) override {
    auto new_cmd = ten::cmd_t::create("hello_world");
    ten_env.send_cmd(std::move(new_cmd),
                     [](ten::ten_env_tester_t &ten_env,
                        std::unique_ptr<ten::cmd_result_t> result) {
                       if (result->get_status_code() == TEN_STATUS_CODE_OK) {
                         ten_env.stop_test();
                       }
                     });
  }
};

TEST(Test, Basic) {
  // 1. Create an extension tester to manage the entire standalone testing process.
  auto *tester = new extension_tester_basic();

  // 2. Set a testing mode, such as a mode for testing a single extension.
  tester->set_test_mode_single("default_extension_cpp");

  // 3. Start the testing.
  tester->run();

  delete tester;
}
```

## Golang

TODO: 待补充。

## Python

```python
class ExtensionTesterBasic(ExtensionTester):
    def check_hello(self, ten_env: TenEnvTester, result: CmdResult):
        statusCode = result.get_status_code()
        print("receive hello_world, status:" + str(statusCode))

        if statusCode == StatusCode.OK:
            ten_env.stop_test()

    def on_start(self, ten_env: TenEnvTester) -> None:
        new_cmd = Cmd.create("hello_world")

        print("send hello_world")
        ten_env.send_cmd(
            new_cmd,
            lambda ten_env, result: self.check_hello(ten_env, result),
        )

        print("tester on_start_done")
        ten_env.on_start_done()


def test_basic():
    # 1. Create an extension tester to manage the entire standalone testing process.
    tester = ExtensionTesterBasic()

    # 2. Set a testing mode, such as a mode for testing a single extension.
    tester.set_test_mode_single("default_extension_python")

    # 3. Start the testing.
    tester.run()
```


# 如何使用日志进行调试 (/cn/docs/ten_framework/0.11.60/testing_and_debugging/how_to_debug_with_logs)

我们将介绍如何查看日志，现在它们已移植到 stdout，因此查看它们非常容易。

例如，如果 TEN Agent 在 localhost:3000 上运行，您可能会在日志中看到类似于以下的输出：

```bash
...
[GIN] 2024/09/29 - 09:29:42 | 200 |     634.458µs |    192.168.65.1 | POST     "/ping"
[agora_5bnxv1] 09-29 09:29:42.895  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ How about you?], text_recv_ts: 1727602181374572, read_bytes: 37200, sent_frames: 117, first_frame_latency: 306ms, finish_latency: 428ms, synthesis_first_byte_latency: 300ms, synthesis_finish_latency: 332ms
[agora_5bnxv1] 09-29 09:29:43.350  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Anything exciting happening?], text_recv_ts: 1727602181383864, read_bytes: 63600, sent_frames: 199, first_frame_latency: 300ms, finish_latency: 452ms, synthesis_first_byte_latency: 295ms, synthesis_finish_latency: 332ms
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_5bnxv1 requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
2024/09/29 09:29:45 INFO handlerPing end worker="&{ChannelName:agora_5bnxv1 HttpServerPort:10024 LogFile:/tmp/astra/app-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.log Log2Stdout:true PropertyJsonFile:/tmp/astra/property-9dc076369ea0cc310a3026a2c4c4cacd-1727602125678030593.json Pid:7455 QuitTimeoutSeconds:60 CreateTs:1727602125 UpdateTs:1727602185}" requestId=691f0b48-12f3-4cdf-b1ff-f1923149b31c service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |    1.038208ms |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:45 INFO handlerPing start channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
2024/09/29 09:29:45 ERROR handlerPing channel not existed channelName=agora_7j2nlb requestId=f8491f0c-324d-4b4d-bbf1-73f4008620e5 service=HTTP_SERVER
[GIN] 2024/09/29 - 09:29:45 | 200 |     442.375µs |    192.168.65.1 | POST     "/ping"
2024/09/29 09:29:48 INFO handlerPing start channelName=agora_5bnxv1 requestId=6fb07d85-
...
```

一旦您与 TEN Agent 交互，您应该会看到如下条目：

```bash title="Bash"
[agora_5bnxv1] 09-29 09:35:25.049  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ and the World Cup in Russia.], text_recv_ts: 1727602522313774, read_bytes: 64000, sent_frames: 200, first_frame_latency: 418ms, finish_latency: 568ms, synthesis_first_byte_latency: 413ms, synthesis_finish_latency: 458ms
[agora_5bnxv1] 09-29 09:35:25.570  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ In tech,], text_recv_ts: 1727602522347918, read_bytes: 29200, sent_frames: 92, first_frame_latency: 427ms, finish_latency: 515ms, synthesis_first_byte_latency: 419ms, synthesis_finish_latency: 460ms
[agora_5bnxv1] 09-29 09:35:26.502  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ we saw some big advancements like the rise of AI and electric cars becoming more mainstream.], text_recv_ts: 1727602522461866, read_bytes: 178800, sent_frames: 559, first_frame_latency: 510ms, finish_latency: 928ms, synthesis_first_byte_latency: 503ms, synthesis_finish_latency: 800ms
[agora_5bnxv1] 09-29 09:35:27.029  7463  7505 I AZURE_TTS_EXTENSION SpeechText@tts.cc:182 task finished for text: [ Personally,], text_recv_ts: 1727602522468652, read_bytes: 34400, sent_frames: 108, first_frame_latency: 442ms, finish_latency: 524ms, synthesis_first_byte_latency: 436ms, synthesis_finish_latency: 463ms
```

当您看到这样的日志时，这意味着系统工作正常并且正在记录您说的每句话。


# Doc Window (/cn/docs/ten_framework/0.11.60/tman_designer/doc_window)

TEN 设计师内建文档窗口，可以查看 TEN Framework 的观念跟术语. 例如在 App 下拉选单中选择 `About App` 即可打开文档窗口并显示 App 的说明.

![TMAN Designer Doc Window](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_doc_window.png)


# 概述 (/cn/docs/ten_framework/0.11.60/tman_designer/overview)

TEN Framework 具有高度灵活性，提供了丰富的功能，包括 TEN 云商店、软件包管理、多语言支持和多平台支持等。然而，这些功能可能需要较长的学习时间和较高的技术门槛，才能熟练操作 TEN Framework 完成 AI Agent 的开发和定制化。为便于开发者使用，TEN Framework 提供了一个名为 TEN Manager 的命令行工具，使开发者能够简便地修改和定制基于 TEN Framework 构建的 AI Agent。

由于命令行工具不如图形用户界面友好，因此 TEN Manager 除了命令行界面外，还内置了基于 Web UI 的可视化开发界面 — TMAN Designer，提供丰富的功能帮助开发者开发和调试 TEN Apps 及 TEN Extension。这意味着开发者只需获取 TEN Manager 这一个工具，就能同时拥有命令行和图形用户界面的完整 TEN Framework 开发环境。

您可以通过以下链接获取 TMAN Designer：

```text
https://github.com/TEN-framework/ten-framework/releases
```

TMAN Designer 支持以下操作系统及架构，请根据您的平台获取对应版本：

* Windows
  * x86\_64
* Linux
  * x86\_64
* MacOS
  * x86\_64
  * arm64

下载并解压后，使用以下命令启动 TMAN Designer：

```shell
tman designer
```

您可以在任何 TEN App 的根目录下执行此命令，也可以在任意位置执行。如果在 TEN App 的根目录下执行，TMAN Designer 会默认将该 TEN App 载入。您也可以在 TMAN Designer 中手动载入任何 TEN App。因此，在特定 TEN App 根目录下启动 TMAN Designer 并非必要操作，但可以简化初始手动载入 TEN App 的步骤。

启动后将看到如下信息，表示 TMAN Designer 默认在端口 49483 上启动：

```text
🏆  Starting server at http://0.0.0.0:49483
```

您可以使用以下 URL 与 TMAN Designer 交互：

```text
http://127.0.0.1:49483/
```

> 注意：如果您的 TMAN Designer 运行在远程机器上，请使用远程机器的 IP 地址访问 TMAN Designer。

## 概览

使用浏览器打开 TMAN Designer 页面后，会看到如下界面：

![TMAN Designer 概观](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_overview.png)

在 TMAN Designer 右上方可以看到几个功能按钮：

* 最右侧按钮显示当前 TMAN Designer 的版本号，并自动检测是否有新版本发布。如有更新，会显示一个向上的箭头，点击后将打开新版本的下载页面。
* 左侧第一个按钮是 [TEN Agent](https://agent.theten.ai/) 按钮，点击后将打开 TEN Agent 页面。TEN Agent 是使用 TEN Framework 构建的完整 AI Agent 实现，TMAN Designer 的主要目标之一是让开发者能够轻松地通过 TMAN Designer 开发、调试和定制 TEN Agent。
* 左侧第二个按钮是 [TEN Framework](https://github.com/TEN-framework/ten-framework) 的 GitHub 页面，点击后将打开 TEN Framework 的 GitHub 仓库。如需了解更多关于 TEN Framework 的信息、源代码或提交问题，可以访问此页面。
* 最左侧有两个按钮，分别用于切换明暗模式和选择语言。

在 TMAN Designer 的左上方有几个下拉菜单，开发者可以通过这些菜单使用 TMAN Designer 的各项功能。

## 处理 TEN App

### 载入已有的 TEN App

通过 App 菜单的 Load App 按钮，可以载入已有的 TEN App。点击按钮后，会看到如下界面：

![载入已有的 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_load_app.png)

通过此文件浏览对话框，开发者可以指定已有 TEN App 的根目录来载入应用。

### 管理已载入的 TEN App

通过 App 菜单的 Manage Loaded App(s) 按钮，可以管理已载入的 TEN App。点击按钮后，会看到如下 Apps Manager 界面：

![管理已经载入的 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_manage_loaded_apps.png)

开发者可以通过此对话框：

* 卸载已载入的 TEN App
* 重新载入指定的 TEN App
* 重新载入所有已载入的 TEN App
* 安装指定 App 的所有依赖
* 运行指定 App 的功能

### 安装指定 App 的所有依赖

开发者可以在 TEN App 中安装各种 TEN 软件包，一个 TEN App 所依赖的所有 TEN 软件包都会记录在 `manifest.json` 文件中。以下是一个 `manifest.json` 文件的示例：

```json
{
  "type": "app",
  "name": "aaa",
  "version": "0.8.18",
  "dependencies": [
    {
      "type": "system",
      "name": "ten_runtime",
      "version": "0.8.18"
    },
    {
      "type": "extension",
      "name": "ffmpeg_muxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_demuxer",
      "version": "^0.1.0"
    },
    {
      "type": "extension",
      "name": "ffmpeg_client",
      "version": "^0.1.0"
    },
    {
      "type": "protocol",
      "name": "msgpack",
      "version": "^0.8.18"
    }
  ],
  "scripts": {
    "start": "python3 tools/run_script.py start",
    "build": "python3 tools/run_script.py build"
  }
}
```

从上述配置可以看出，该 TEN App 依赖以下几个 TEN 软件包：

* `ten_runtime`：TEN Framework 的系统软件包
* `ffmpeg_demuxer`：用于解码音频和视频的 TEN Extension 软件包
* `ffmpeg_muxer`：用于编码音频和视频的 TEN Extension 软件包
* `ffmpeg_client`：用于使用另外两个 ffmpeg extension 软件包处理音频和视频的 TEN Extension 软件包
* `msgpack`：用于使用 MessagePack 协议进行消息序列化和反序列化的 TEN Protocol 软件包

实际上，在完成 TEN App 所有依赖的安装后，系统中的 TEN 软件包文件结构如下：

```text
├── ten_packages
│   ├── extension
│   │   ├── ffmpeg_client
│   │   ├── ffmpeg_demuxer
│   │   └── ffmpeg_muxer
│   ├── protocol
│   │   └── msgpack
│   └── system
│       ├── ffmpeg
│       └── ten_runtime
```

值得注意的是，尽管没有在 `manifest.json` 中显式声明，系统中还多了一个 `ffmpeg` 系统软件包。这是因为 `ffmpeg_demuxer` 和 `ffmpeg_muxer` 依赖了 `ffmpeg` 这个系统软件包，TEN Manager 会自动识别并安装这些传递依赖。

通过 TMAN Designer 的 Apps Manager 界面，开发者可以对指定的 App 执行 Install All 操作，轻松完成上述所有依赖的安装过程。点击按钮后，会看到如下界面，显示安装进度：

![安装指定 App 的所有依赖](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_install_all.png)

### 从云商店安装 TEN Extension

通过 Graph 菜单的 Open Extension Store 按钮，可以打开 TEN 云商店，并从云商店安装 TEN Extension 到当前的 TEN App。点击按钮后，会看到如下界面：

![TEN 云商店](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_store.png)

在 TEN 云商店中，开发者可以搜索所需的 TEN Extension，并点击 Install 按钮安装到当前的 TEN App。安装完成后，可以在当前 TEN App 中查看已安装的 TEN Extension。

### 运行 TEN App

在 Apps Manager 中，可以对指定的 App 执行预设的运行操作。在 Apps Manager 中点击 Run 按钮后，会看到如下界面：

![运行 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_run.png)

在对话框中，开发者可以从下拉菜单中选择要执行的预设操作。通常包括以下几种操作：

* 运行 App：启动 App 并在 TMAN Designer 的消息窗口中显示 App 的输出信息。
* 编译 App：对 App 进行编译。由于 TEN Framework 支持多种编程语言，有些语言需要编译，有些则无需编译。例如，基于 C++ 开发的 TEN App 需要编译，而基于 Python 开发的 TEN App 则无需编译。

以下是一个基于 C++ 开发的 TEN App 的编译界面：

![编译 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_build.png)

编译完成后，可以继续执行 TEN App 的启动操作，系统会弹出一个窗口，实时显示 TEN App 的运行日志：

![启动 TEN app](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_app_start.png)

## 处理 TEN Graph

### 载入已有的 TEN Graph

通过 Graph 菜单的 Open Existing Graph 按钮，可以载入已有的 TEN Graph。载入成功后，会看到类似如下界面：

![TEN Graph](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_graph_overview.png)

以这个 Graph 为例，它包含 3 个 TEN Extensions，分别是：

* `ffmpeg_demuxer`
* `ffmpeg_muxer`
* `ffmpeg_client`

每个 TEN Extension 都有其对应的输入和输出。以 `ffmpeg_demuxer` 为例，它具有以下输入与输出：

* 2 个 cmd 输入
* 1 个 cmd 输出
* 1 个 audio\_frame 输出
* 1 个 video\_frame 输出

点击输入输出的数字按钮，可以打开如下对话框，用于管理输入输出的连接：

![TEN Message Input/Output Dialog](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_msg_info_dialog.png)

### 自动排布 Graph

当开发者手动调整 TEN Graph 中的 TEN Extension 位置后，可以点击 Graph 菜单中的 Auto Layout 按钮，自动排布 TEN Graph 中的 TEN Extension。

### 打开 TEN Extension 的上下文菜单

在 Graph 中，对每个 TEN Extension 右键点击，可以打开 TEN Extension 的上下文菜单：

![TEN extension context menu](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_extension_context_menu.png)

在上下文菜单中，可以打开 TEN Extension 的 `manifest.json` 文件、`property.json` 文件，也可以直接在 TEN Extension 的根目录下启动终端，方便开发者对该 TEN Extension 进行修改和定制：

![Open Manifest](https://ten-framework-assets.s3.amazonaws.com/doc-assets/ten_designer_context_menu_effect.png)


# 通过代码自定义 (/cn/docs/ten_agent_examples/customize_agent/modify-main)

使用 TMAN Designer 配置扩展，可以帮助你快速完成基础功能的修改与验证。\
但当你需要实现更复杂的功能，而这些功能并未在可视化界面中直接支持时，就需要通过编写代码来自定义实现。\
在修改代码之前，建议先熟悉并理解 `main` 扩展的结构与职责。

***

在 TEN 框架中，**连接（connections）** 用于定义视频与音频数据在不同扩展之间的传输方式。\
这种设计能够最大限度地降低不必要的复杂度，并确保系统在高负载场景下依旧保持优异性能——这也是 TEN 框架的重要优势之一。

对于 **事件与文本类数据**，由于其通常占用资源较少，更适合通过代码直接处理。\
这种方式不仅具备更高的灵活性，也能够更高效地支持复杂逻辑的实现。

为简化工作流，最新版本的默认 TEN Agent 应用在每个图（graph）中都引入了一个内置的 **“main” 扩展**。\
该扩展作为核心枢纽，负责组件间的编排与管理。

采用这一设计，你将获得以下收益：

1. **视频/音频数据** 始终通过扩展间的连接传输，从而充分利用框架的性能优化能力，而无需直接处理底层的媒体逻辑。
2. **“main” 扩展** 可以与任意扩展交互，并统一收集所需的事件与文本数据：
   * 对于 **入站事件/数据**，需要在 `property.json` 文件中注册从源扩展到 “main” 扩展的连接。
   * 对于 **出站事件/数据**，则可在运行时调用 `setDests` 方法来动态指定目标扩展，无需在 `property.json` 中预先定义连接。
3. **应用层逻辑集中化** —— 所有与业务相关的自定义逻辑均可放置于 “main” 扩展内，确保整体结构清晰。
4. **扩展保持无状态与独立性** —— 其他扩展应尽量保持轻量与通用，便于在不同项目或场景中复用。

此外，**"main" 扩展支持使用 TEN 框架所兼容的任意语言** 来编写。
例如，你可以使用 Node.js 来实现 "main" 扩展，同时保留其他扩展使用 Python 或 C++。
这一设计不仅避免了 Node.js 在媒体处理方面的性能瓶颈，也能在生产环境中更好地发挥优势。

***

## Main 扩展变种对比

根据你的具体需求，选择合适的架构方案：

| 方面         | Python Cascade                 | Python Realtime V2V            | Node.js Cascade             |
| ---------- | ------------------------------ | ------------------------------ | --------------------------- |
| **语言**     | Python                         | Python                         | TypeScript/JavaScript       |
| **适用场景**   | 传统语音智能体：ASR → LLM → TTS 管道     | 直连多模态（MLLM）：实时视觉+音频            | 高性能语音智能体：支持媒体流              |
| **事件模型**   | 基于装饰器 `@agent_event_handler()` | 基于匹配/分支的 `on_data()` 方法        | 类方法中的直接事件处理                 |
| **消息流**    | **S2C**: 接收运行时类型化事件            | **S2C**: 通过单一 `on_data` 接收所有事件 | **直接**: 消息到达时直接路由           |
| **LLM 类型** | 标准 LLM（仅文本）                    | 多模态 LLM（视觉+音频+文本）              | 标准 LLM（仅文本）                 |
| **流式支持**   | 文本流 + 句子分割                     | MLLM 流 + 模态支持                  | 流式处理 + 句子解析                 |
| **中断处理**   | 通过 `_interrupt()` 检测插话         | 基于 `ServerInterruptEvent` 事件   | 通过句子片段重置处理插话                |
| **工具支持**   | 通过 `@tool_handler()` 装饰器       | 通过匹配/分支 `FunctionCallEvent`    | 通过直接代理工具注册                  |
| **配置方式**   | 通过 Python `config.yaml`        | 通过 `config.py` 带 `greeting`    | 通过 manifest 和 property.json |
| **性能**     | 适合标准工作负载                       | 针对实时多模态优化                      | 最适合媒体密集型工作负载                |
| **复杂度**    | 中等（装饰器+事件总线）                   | 中等（匹配/分支调度）                    | 中等（直接方法路由）                  |
| **最佳选择**   | 学习 TEN 基础                      | 现代 AI：视觉+语音                    | 生产语音智能体                     |

### 快速决策指南

**选择 Python Cascade** 如果你想学习 main 扩展的工作原理，或在 Python 中构建标准语音智能体。

**选择 Python Realtime V2V** 如果你需要多模态能力（摄像头输入、视觉处理）或希望使用支持多种模态的高级 LLM。

**选择 Node.js Cascade** 如果你正在构建生产级语音智能体，并希望利用 Node.js 的性能优势，同时通过基于连接的音频流来避免媒体处理瓶颈。

***

## Main 扩展文档

### Python Cascade 模式

传统 ASR → LLM → TTS 管道，采用装饰器风格的事件处理。适合学习 TEN 基础或构建标准语音智能体。

* [Python Cascade Main](./main_python_cascade) — 完整指南，包含装饰器模式、事件路由和 LLM 集成
* 最佳选择：学习基础、标准语音智能体、纯 Python 环境

### Python Realtime V2V 模式

多模态 MLLM 集成，支持实时视觉和音频。现代 AI 能力的先进方法。

* [Python Realtime V2V Main](./main_python_realtime) — 事件路由（匹配/分支）、C2S 原语、工具处理和函数调用
* 最佳选择：视觉+语音、多模态 AI、支持视觉的现代 LLM

### Node.js Cascade 模式

使用 TypeScript/JavaScript 构建的高性能语音智能体，具有优化的媒体流处理。

* [Node.js Cascade Main](./main_nodejs_cascade) — ASR 处理、LLM 响应流、工具注册和自然中断
* 最佳选择：生产级语音智能体、性能关键应用、Node.js 团队

***

## 常见任务

### 理解事件流

* **Python Cascade**: 装饰器风格事件处理器 (`@agent_event_handler()`) - 参见 [ASR 事件处理章节](./main_python_cascade#event-routing)
* **Python Realtime V2V**: 匹配/分支调度 - 参见 [事件路由章节](./main_python_realtime#event-routing)
* **Node.js Cascade**: 直接方法处理器 - 参见 [ASR 处理章节](./main_nodejs_cascade)

### 处理工具调用

* **Python Cascade**: 工具装饰器模式
* **Python Realtime V2V**: 函数调用事件处理 - 参见 [C2S 原语](./main_python_realtime#c2s-primitives-sending-to-mllm)
* **Node.js Cascade**: 代理工具注册

### 实现中断处理

学习如何在你的模式中处理用户中断（插话）：

* **Python Cascade**: 扩展中的 `_interrupt()` 方法
* **Python Realtime V2V**: `ServerInterruptEvent` 处理 - 参见 [模式章节](./main_python_realtime#common-implementation-patterns)
* **Node.js Cascade**: 通过句子片段重置的自然插话 - 参见 [LLM 处理章节](./main_nodejs_cascade)


# NodeJS - Cascade Main (/cn/docs/ten_agent_examples/customize_agent/modify-main/main_nodejs_cascade)

import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
import { Callout } from 'fumadocs-ui/components/callout';

# Main Agent Extension for Node.js

The file [`index.ts`](./index.ts) defines the **MainControlExtension** — the entry point for Node.js agents in the TEN Framework.

This is the class that wires together **ASR results**, **LLM responses**, **tool registration**, and **interruption handling**.
If you're building your own solution on top of TEN (with Node.js), start here.

<Tabs items={['Overview', 'ASR Handling', 'LLM Handling', 'Patterns', 'Summary']}>
  <Tab>
    ## Quick File Layout

    ```
    .
    ├── index.ts          → Main extension: message routing + interruption
    ├── helper.ts         → Utilities for sending Cmd/Data, sentence parsing
    └── agent/
        ├── agent.ts      → Event bus + orchestration
        ├── events.ts     → Event classes (ASR, LLM, Tools, User)
        ├── llm_exec.ts   → LLM execution queue and handlers
        └── struct.ts     → Zod schemas for TTS/ASR/LLM messages
    ```

    ## Architecture Overview

    <img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/nine.svg" alt="Architecture" />

    ## Core Architecture

    ### MainControlExtension Class

    Extends `Extension` from `ten-runtime-nodejs`. Normalizes runtime messages into **typed events** (`ASRResultEvent`, `LLMResponseEvent`, etc.). Routes events into the `Agent` class. Manages **interruption** when users talk over the assistant.

    ### Key Properties

    ```ts
    class MainControlExtension extends Extension {
      tenEnv!: TenEnv;
      agent!: Agent;
      config!: MainControlConfig;

      joinedUserCount: number = 0;
      session_id: string = "0";
      turn_id: number = 0;
      sentenceFragment: string = "";
    }
    ```

    **`session_id` + `turn_id`** track conversation state. **`sentenceFragment`** accumulates partial output for sentence splitting.
  </Tab>

  <Tab>
    ## ASR Event Handling

    When ASR detects speech, it's turned into an `ASRResultEvent`.

    ```ts
    if (event.final || event.text.length > 2) {
      await this._interrupt(); // flush LLM/TTS if user is overlapping
    }
    if (event.final) {
      this.turn_id += 1;
      await this.agent.queueLLMInput(event.text);
    }
    await this._sendTranscript("user", event.text, event.final, Number(this.session_id));
    ```

    ### Behavior

    **Partial speech** is logged as transcript and may trigger interrupt if overlapping. **Final speech** is queued for LLM input and increments turn counter. **Overlap detection** via `_interrupt()` cancels current LLM/TTS tasks.

    <Callout type="tip">
      Adjust interrupt sensitivity in `_onASRResult` based on your use case (stricter vs. looser barge-in).
    </Callout>
  </Tab>

  <Tab>
    ## LLM Response Handling

    LLM responses are streamed sentence by sentence, then forwarded to TTS and transcript collectors.

    ```ts
    if (!event.is_final && event.type === "message") {
      const [sentences, fragment] = parseSentences(this.sentenceFragment, event.delta);
      this.sentenceFragment = fragment;
      for (const s of sentences) {
        await this._sendToTTS(s, false);
      }
    }

    await this._sendTranscript(
      "assistant", event.text, event.is_final, 100,
      event.type === "reasoning" ? "reasoning" : "text"
    );
    ```

    ### Behavior

    **Streaming** sends TTS short sentences for natural, fluent speech. **Sentence splitting** via `parseSentences()` breaks output into natural pauses. **Partial transcripts** are logged as they arrive. **Final transcripts** are marked complete with reasoning/text type.

    ### Tool Registration

    The LLM can register tools dynamically.

    ```ts
    async _onToolRegister(event: ToolRegisterEvent) {
      await this.agent.registerLLMTool(event.tool, event.source);
    }
    ```

    Define your tool schema in `events.ts` and handle execution in your `Agent` logic.

    ### Interruption Handling

    Natural barge-in is handled by `_interrupt()`.

    ```ts
    async _interrupt() {
      this.sentenceFragment = "";
      await this.agent.flushLLM();
      await sendData(this.tenEnv, "tts_flush", "tts", { flush_id: uuidv4() });
      await sendCmd(this.tenEnv, "flush", "agora_rtc");
    }
    ```

    Flushes **LLM requests** (cancels pending generations). Cancels **TTS playback** (stops audio playback). Signals **RTC to stop** streaming. This ensures the user can always cut in naturally.
  </Tab>

  <Tab>
    ## Implementation Patterns

    ### Custom ASR Flow

    Edit `_onASRResult` to filter or preprocess speech:

    ```ts
    async _onASRResult(event: ASRResultEvent) {
      // Custom preprocessing
      let text = event.text.toLowerCase().trim();

      // Filter unwanted phrases
      if (text.includes("ignore")) return;

      // Then continue with normal flow
      await this._handleASRNormally(text, event.final);
    }
    ```

    ### Custom LLM Output

    Modify `_onLLMResponse` to change text before TTS or UI:

    ```ts
    async _onLLMResponse(event: LLMResponseEvent) {
      // Transform response
      event.delta = event.delta.replace("technical_term", "simplified_term");

      // Continue normally
      await this._handleLLMNormally(event);
    }
    ```

    ### Add a Tool

    1. Extend `ToolRegisterEvent` in `events.ts`
    2. Register in `Agent` via `registerLLMTool()`
    3. Handle execution in tool handler

    ### Tweak Responsiveness

    Adjust `_interrupt()` for stricter or looser barge-in:

    ```ts
    // Stricter (require more speech)
    if (event.text.length > 5) {
      await this._interrupt();
    }

    // Looser (interrupt on any sound)
    if (event.text.length > 0) {
      await this._interrupt();
    }
    ```

    <Callout type="warning">
      Node.js extension changes require a build step. Run `task build` to rebuild all Node.js extensions.
    </Callout>
  </Tab>

  <Tab>
    ## Supporting Components

    **`agent.ts`** manages event queues for ASR & LLM, plus handler registration. **`llm_exec.ts`** provides async queue that sends requests to LLMs and handles streaming responses, aborts, and tool calls. **`events.ts`** defines event classes like `ASRResultEvent`, `LLMResponseEvent`, `ToolRegisterEvent`. **`helper.ts`** provides `sendCmd`, `sendData`, and `parseSentences` for splitting output into natural phrases.

    ## Architecture Summary

    `index.ts` (`MainControlExtension`) is the **entry point** and **router**. It normalizes runtime messages into **typed events** and controls **interruption**. Supporting files (`agent/`, `helper.ts`) provide queues, event models, and execution helpers. The Cascade pattern flows: **ASR → LLM → TTS** with natural interruption handling.

    By following this structure, you can **quickly extend or replace** parts of the flow to build your own Node.js-based conversational agent on TEN.
  </Tab>
</Tabs>


# Python - Cascade Main (/cn/docs/ten_agent_examples/customize_agent/modify-main/main_python_cascade)

# Building on the Main Extension

The file [`extension.py`](./extension.py) is the **control center** of the agent.
If you want to build your own solution on top of the TEN Framework, this is the file you should understand first.

It shows how runtime messages (ASR, LLM, tools, user events) are captured, normalized, and redirected into the agent workflow.
By following its patterns, you can easily **extend** or **replace** parts of the pipeline with your own logic.

***

## Quick File Layout

```
main_python/
├── extension.py      → Main message router (start here!)
└── agent/
    ├── agent.py      → Event bus and orchestration
    ├── events.py     → Typed event definitions (ASR, LLM, Tools, User)
    ├── llm_exec.py   → Manages LLM requests and responses
    └── decorators.py → Event binding helpers
```

You mostly need to know **`extension.py`**. The other files provide typed events and execution helpers.

***

## Architecture Overview

<img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/nine.svg" alt="Architecture" />

***

## How `extension.py` Works

<div className="fd-steps">
  <div className="fd-step">
    ### Event Routing

    The extension listens to runtime messages and turns them into **typed events**.
    For example, an ASR result from ASR extension becomes an `ASRResultEvent` object that the agent can use.

    ```python
    @agent_event_handler(ASRResultEvent)
    async def _on_asr_result(self, event: ASRResultEvent):
        if not event.text:
            return
        if event.final or len(event.text) > 2:
            await self._interrupt()
        if event.final:
            self.turn_id += 1
            await self.agent.queue_llm_input(event.text)
        await self._send_transcript("user", event.text, event.final, int(self.session_id))
    ```

    Here’s what happens:

    * **Partial speech** → streamed to transcript/logging
    * **Final speech** → sent into the LLM queue
    * **Overlap detected** → `_interrupt()` flushes ongoing TTS/LLM so the user can speak naturally

    ***
  </div>

  <div className="fd-step">
    ### Handling LLM Responses

    When the LLM responds, the extension splits sentences and forwards them to TTS while also recording transcripts.

    ```python
    @agent_event_handler(LLMResponseEvent)
    async def _on_llm_response(self, event: LLMResponseEvent):
        if not event.is_final and event.type == "message":
            sentences, self.sentence_fragment = parse_sentences(
                self.sentence_fragment, event.delta
            )
            for s in sentences:
                await self._send_to_tts(s, False)

        await self._send_transcript(
            "assistant", event.text, event.is_final, 100,
            data_type=("reasoning" if event.type == "reasoning" else "text"),
        )
    ```

    Key points:

    * **Streaming**: You can send partial outputs sentence by sentence to TTS for natural speech.
    * **Final output**: Marked and sent to the transcript collector.
    * **Reasoning traces**: Can be separated if you want to show them differently in your UI.

    ***
  </div>

  <div className="fd-step">
    ### Tool Registration

    The extension lets the LLM register tools dynamically.

    ```python
    @agent_event_handler(ToolRegisterEvent)
    async def _on_tool_register(self, event: ToolRegisterEvent):
        await self.agent.register_llm_tool(event.tool, event.source)
    ```

    To add your own tool, define its metadata in `events.py`, then handle it in your solution.
    This makes it easy to plug in **APIs, databases, or custom functions** that the LLM can call.

    ***
  </div>

  <div className="fd-step">
    ### Interruption

    Natural conversation needs interruption (barge-in).
    If the user speaks while the assistant is still generating, `_interrupt()` flushes LLM and TTS:

    ```python
    async def _interrupt(self):
        self.sentence_fragment = ""
        await self.agent.flush_llm()
        await _send_data(
            self.ten_env, "tts_flush", "tts", {"flush_id": str(uuid.uuid4())}
        )
        await _send_cmd(self.ten_env, "flush", "agora_rtc")
    ```

    This ensures the assistant doesn’t “talk over” the user.

    ***
  </div>
</div>

## How to Extend for Your Own Solution

* **Custom ASR behavior**: Edit `_on_asr_result` to filter text, add punctuation, or preprocess before sending to LLM.
* **Custom LLM logic**: Change `_on_llm_response` to transform text before TTS, or enrich transcripts with metadata.
* **Add tools**: Use `ToolRegisterEvent` to expose your own APIs or functions.
* **Custom interruption policy**: Tweak `_interrupt()` to make the agent more/less tolerant of overlapping speech.

## Summary

* **`extension.py` is the heart of the agent**: it routes runtime messages into typed events and applies core conversation logic.
* By modifying or extending these handlers, you can **quickly build your own conversational AI solution** on top of TEN.
* Everything else (`agent.py`, `llm_exec.py`, `events.py`) supports this routing, but the patterns in `extension.py` are what you’ll reuse the most.


# Python - Realtime V2V Main (/cn/docs/ten_agent_examples/customize_agent/modify-main/main_python_realtime)

import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
import { Callout } from 'fumadocs-ui/components/callout';

# Realtime V2V / MLLM Agent (`extension.py`)

The file [`extension.py`](./extension.py) is the **entry point** of the realtime voice-to-voice / multimodal-LLM app.
It consumes **Server→Client (S2C)** events (transcripts, tool calls, interrupts, user join/leave), normalizes them into typed events, and routes them to the agent.
It also provides **Client→Server (C2S)** primitives to set context, send messages, return tool results, and trigger responses.

<Tabs items={['Overview', 'Event Routing', 'C2S Primitives', 'Patterns', 'Summary']}>
  <Tab>
    ## Quick File Layout

    ```
    .
    ├── extension.py      → Main extension: routing + context + trigger + interrupt
    ├── config.py         → Runtime config (e.g., greeting)
    ├── helper.py         → Cmd/Data send helpers
    └── agent/
        ├── agent.py      → Event queue, function call handling
        └── events.py     → Typed events (UserJoined, Transcripts, FunctionCall, Interrupt, etc.)
    ```

    <Callout type="warning">
      **Audio data** is not handled here. RTC audio is sent **directly to MLLM via TEN graph connections**.
      `extension.py` only reacts to the **S2C transcript events** generated by the MLLM server.
    </Callout>

    ### Architecture Overview

    <img src="https://ten-framework-assets.s3.amazonaws.com/blog/main-control/ten.svg" alt="Architecture" />
  </Tab>

  <Tab>
    ## Event Routing

    Unlike decorator-based handlers, this extension uses a **single `on_data` method with a `match/case`** to dispatch events.

    ```python
    async def on_data(self, ten_env: AsyncTenEnv, data: Data):
        event = parse_event(data)  # converted to a typed AgentEvent

        match event:
            case UserJoinedEvent():
                self._rtc_user_count += 1
                await self._greeting_if_ready()

            case UserLeftEvent():
                self._rtc_user_count -= 1

            case ToolRegisterEvent():
                await self.agent.register_tool(event.tool, event.source)

            case FunctionCallEvent():
                await self.agent.call_tool(event.call_id, event.function_name, event.arguments)

            case InputTranscriptEvent():
                self.current_metadata = {"session_id": event.metadata.get("session_id", "100")}
                self.session_ready = True
                await self._greeting_if_ready()

            case OutputTranscriptEvent():
                await self._send_transcript("assistant", event.text, event.is_final, event.stream_id)

            case ServerInterruptEvent():
                await self._interrupt()

            case _:
                self.ten_env.log_warn(f"[MainControlExtension] Unhandled event: {event}")
    ```

    **S2C events handled:** `UserJoinedEvent` / `UserLeftEvent` track users. `InputTranscriptEvent` captures user speech text from RTC audio. `OutputTranscriptEvent` sends assistant response. `ToolRegisterEvent` registers new tools. `FunctionCallEvent` handles LLM tool requests. `ServerInterruptEvent` stops ongoing output.
  </Tab>

  <Tab>
    ## C2S Primitives (Sending to MLLM)

    `extension.py` provides simple methods for sending instructions to the MLLM server:

    <div className="fd-steps">
      <div className="fd-step">
        ### Set context

        ```python
        async def _set_context_messages(self, messages: list[MLLMClientMessageItem]):
            await _send_data(
                self.ten_env,
                DATA_MLLM_IN_SET_MESSAGE_CONTEXT,
                "v2v",
                MLLMClientSetMessageContext(messages=messages).model_dump(),
            )
        ```
      </div>

      <div className="fd-step">
        ### Send a message

        ```python
        async def _send_message_item(self, message: MLLMClientMessageItem):
            await _send_data(
                self.ten_env,
                DATA_MLLM_IN_SEND_MESSAGE_ITEM,
                "v2v",
                MLLMClientSendMessageItem(item=message).model_dump(),
            )
        ```
      </div>

      <div className="fd-step">
        ### Trigger a response

        ```python
        async def _send_create_response(self):
            await _send_data(
                self.ten_env,
                DATA_MLLM_IN_CREATE_RESPONSE,
                "v2v",
                MLLMClientCreateResponse().model_dump(),
            )
        ```
      </div>

      <div className="fd-step">
        ### Send function output

        (Sent from `agent.py` after handling a `FunctionCallEvent`)

        ```python
        await _send_data(
            self.ten_env,
            DATA_MLLM_IN_FUNCTION_CALL_OUTPUT,
            "v2v",
            MLLMClientFunctionCallOutput(
                output=result,
                call_id=call_id,
            ).model_dump(),
        )
        ```
      </div>
    </div>
  </Tab>

  <Tab>
    ## Common Implementation Patterns

    ### Greeting Recipe

    The greeting is handled in `_greeting_if_ready()` and triggered when the first user joins and session is ready:

    ```python
    async def _greeting_if_ready(self):
        if self._rtc_user_count == 1 and self.config.greeting and self.session_ready:
            await self._send_message_item(
                MLLMClientMessageItem(
                    role="user",
                    content=f"say {self.config.greeting} to me",
                )
            )
            await self._send_create_response()
    ```

    This ensures the assistant greets the user automatically.

    ### Function Call Handling

    **S2C**: `FunctionCallEvent` means the model requests a tool. **Agent**: executes the tool logic. **C2S**: return result with `DATA_MLLM_IN_FUNCTION_CALL_OUTPUT` and the same `call_id`. The model may continue its response afterwards.

    ### Interruption

    ```python
    async def _interrupt(self):
        await _send_cmd(self.ten_env, "flush", "agora_rtc")
    ```

    Triggered on `ServerInterruptEvent`. Stops RTC playback/streaming.

    ### Supporting Files

    **`agent.py`** queues events, executes tools, sends function outputs. **`events.py`** defines typed events: `UserJoinedEvent`, `UserLeftEvent`, `InputTranscriptEvent`, `OutputTranscriptEvent`, `ToolRegisterEvent`, `FunctionCallEvent`, `ServerInterruptEvent`. **`helper.py`** provides wrappers for `_send_cmd`, `_send_data`. **`config.py`** holds config like `greeting`. **Graph (`property.json`)** wires RTC audio → MLLM.
  </Tab>

  <Tab>
    ## Events/API Summary

    | Direction | Event / Channel                | Purpose                         |
    | --------- | ------------------------------ | ------------------------------- |
    | **S2C**   | `UserJoinedEvent`              | Track users; trigger greeting   |
    | **S2C**   | `UserLeftEvent`                | Track users leaving             |
    | **S2C**   | `InputTranscriptEvent`         | User speech text from RTC audio |
    | **S2C**   | `OutputTranscriptEvent`        | Assistant response text         |
    | **S2C**   | `ToolRegisterEvent`            | Register new tool               |
    | **S2C**   | `FunctionCallEvent`            | Model requests tool             |
    | **S2C**   | `ServerInterruptEvent`         | Stop ongoing output             |
    | **C2S**   | `_set_context_messages([...])` | Provide system/dev/user context |
    | **C2S**   | `_send_message_item(...)`      | Send user/developer message     |
    | **C2S**   | `_send_create_response()`      | Trigger assistant response      |
    | **C2S**   | Function output (from agent)   | Return tool results             |

    <Callout type="tip">
      For detailed event documentation and parameters, see the [API Reference](/docs/ten_agent_examples/api-reference/events).
    </Callout>
  </Tab>
</Tabs>


# Configure Extension Nodes (/cn/docs/ten_agent_examples/customize_agent/tman-designer/configure_modules)

Extensions (or modules) are the core building blocks of a voice agent. They define the capabilities and functionalities of the agent. With TMAN Designer, you can easily add, remove, or replace extensions of same kind to tailor the agent to your specific needs.

You can follow [this guide]("/docs/ten-agent/customize_agent/tman-designer/") to learn to launch TMAN Designer if you haven't done so already.

## Replace Extension Node

Replacing can be very useful when you want to replace a node with another node of same protocol, as it will keep the original connections while changing the underlying implementation (e.g. to a different vendor model).

To replace a module or extension in TMAN Designer, follow these steps:

1. Right click on the node you would like to replace, in the context menu, select "Replace Node With..."
2. In the popup window, select the extension implementation you would like to use with "Addon Name" text input field.
3. Click "Replace Node" to confirm the replacement.

<Callout title="Note">
  The replacing extension will be using the properties defined in `property.json` by default.
</Callout>

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/three.png)

## Add Extension Node

You can add an extension node to a graph with following steps:

1. Right click on an empty area in the canvas, in the context menu, select "Add Node"
2. In the popup window, select the extension implementation you would like to use with "Addon Name" text input field, and give the node a name with "Node Name" text input field. The node name needs to be unique in a graph.
3. Click "Add Node" to confirm the addition.

<Callout title="Note">
  The added extension will be using the properties defined in `property.json` by default.
</Callout>

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/six.png)

## Remove Extension Node

You can remove an extension you no longer need from graph by following these steps:

1. Right click on the node you want to remove, in the context menu, select "Delete"
2. Confirm the removal in the popup window.

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/four.png)

## Supported Extension Categories

Some key categories of extensions are widely adopted in voice ai agent. In TEN-Agent these extensions are built using certain shared protocols so that it's easy to switch.

Following is a list of supported extension categories:

### Audio Speech Recognition (ASR)

* aliyun\_asr
* aliyun\_asr\_bigmodel\_python
* aws\_asr\_python
* azure\_asr\_python
* bytedance\_asr
* deepgram\_asr\_python
* gladia\_asr\_python
* google\_asr\_python
* openai\_asr\_python
* soniox\_asr\_python
* speechmatics\_asr\_python
* tencent\_asr\_python
* xfyun\_asr\_bigmodel\_python
* xfyun\_asr\_dialect\_python
* xfyun\_asr\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Large Language Model (LLM)

* openai\_llm2\_python
* dify\_llm2\_python
* coze\_llm2\_python

<Callout title="Note">
  Most LLM models are compatible with openai api schema. You can switch vendors by changing base\_uri / model to vendor you want.
</Callout>

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Text to Speech (TTS)

* azure\_tts\_python
* bytedance\_tts\_duplex
* cartesia\_tts2
* elevenlabs\_tts2\_python
* fish\_audio\_tts\_python
* google\_tts\_python
* humeai\_tts\_python
* minimax\_tts\_websocket\_python
* openai\_tts2\_python
* polly\_tts
* tencent\_tts\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Multimodal Realtime V2V Model (MLLM)

* azure\_mllm\_python
* glm\_mllm\_python
* gemini\_mllm\_python
* openai\_mllm\_python
* stepfun\_mllm\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>

### Tools (TOOLS)

* weatherapi\_tool\_python
* vision\_analyze\_tool\_python
* mcp\_client\_python

<Callout title="Note">
  The list above is subject to change as new extensions are added or existing ones are updated.
</Callout>


# Configure Extension Properties (/cn/docs/ten_agent_examples/customize_agent/tman-designer/configure_properties)

Every extension has a set of configurable properties. The configurable properties can be checked via the extension's `manifest.json` file.
The properties can be used to customize the behavior of the extension during runtime.

You can follow [this guide](/docs/ten-agent/customize_agent/tman-designer/) to learn to launch TMAN Designer if you haven't done so already.

## Configure Node Properties

1. Right click on a node you want to configure, in the context menu, select "Update Properties"
2. In the popup window, you may see a set of properties displayed as a form for you to edit.
3. Make the necessary changes and click "Save Change" to apply the updates.

![Properties Dialog](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/properties_dialog.png)

## Dynamic Fields

"Dynamic fields" are essentially a way to define an object without strictly specifying all of its possible keys upfront. Instead of enforcing a fixed schema, the object can accept and carry through any extra properties provided at runtime.

The main purpose is **flexibility and extensibility**:

* You can design an object with only the core or required fields defined.
* Any additional fields can be passed in dynamically, without changing the original object definition.
* When the object is handed off to an external SDK or API, those dynamic fields are transparently forwarded along with the known fields.

This means you don’t have to constantly update your object definition whenever new attributes are needed by the external system—you can just attach them on the fly.

To set a dynamic field, you can follow these steps:

1. Right click on a node you want to configure, in the context menu, select "Update Properties"
2. In the popup window, click on top tab to switch to "Dynamic Fields"
3. Choose the object key route where you want to add the dynamic field, set the name of the field, choose the data type and default value.
4. Confirm "Add Field" to finish adding the dynamic field.
5. Back to "Node Properties", you should see the newly added dynamic field in the "View Form Values" preview.

![Dynamic Fields](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/dynamic_fields.png)


# Use TMAN Designer (/cn/docs/ten_agent_examples/customize_agent/tman-designer)

TEN Manager includes a web UI-based visual development interface called TMAN Designer, which eases the process of creating and customizing voice agents. You can refer to [TEN Framework docs](/docs/ten_framework/tman_designer/overview) for more details about TMAN Designer.

This section provides various examples of how to use TMAN Designer to create and customize voice agents for different use cases.

<Callout title="Deprecated use of playground">
  In earlier versions of TEN Agent, we supported to use playground to customize voice agents. This has been deprecated. For latest version of TEN Agent, please use TMAN Designer to customize your voice agents.
</Callout>

## Launching TMAN Designer

1. Make sure `task run` is running and TEN Manager server is started. If you haven't started it yet, you can follow the instructions in [Get Started](/docs/ten_agent/getting_started) to set it up.
2. Open the designer UI at [http://localhost:49483](http://localhost:49483) to configure your agent.
3. You should see the TMAN Designer interface now, the graphs of your current loaded TEN App should be displayed in the canvas.

![image](https://ten-framework-assets.s3.amazonaws.com/blog/main-control/five.png)


# 故事讲述者 (/cn/docs/ten_agent_examples/playground/use-cases/run_story_telling)

本指南将帮助您在 TEN-Agent Playground 中使用故事大王用例。

## STT + TTS + LLM

### 前提条件

* 确保您已运行 TEN-Agent Playground。如果没有，请按照[运行 Playground](../../demo) 指南启动 Playground。
* 您需要准备以下信息：
  * STT 信息，可以使用任何支持的 STT。[Deepgram](https://deepgram.com/) 相对容易注册和上手。
  * TTS 信息，可以使用任何支持的 TTS。[Fish.Audio](https://fish.audio/) 相对容易注册和上手。
  * LLM 信息，对于此用例，仅支持 [OpenAI](https://openai.com) 或与 OpenAI API 兼容的模型。
  * RTC 信息，目前仅支持 Agora RTC。您可以在 [Agora](https://www.agora.io/) 注册您的帐户。我们假设您在配置 `.env` 文件时已准备好您的 App ID 和 App Certificate。

### 步骤

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择图表类型 `story_teller`。
3. 单击“模块选择器”以打开模块选择。
4. 如果您首选的 STT/TTS 模块默认未选中，您可以从下拉列表中选择模块。请注意，您需要使用 API 密钥等正确信息配置模块。
5. “LLM”模块已预先配置为选择“OpenAI ChatGPT”，请勿更改它。
6. 单击“保存更改”以将模块应用于图表。
7. 单击图表选择右侧的按钮以打开属性配置。您将看到可以为选定的大型语言模型配置的属性列表。
8. 使用您准备的信息配置属性。
9. 单击“保存更改”以将属性应用于大型语言模型。
10. 如果您看到成功提示，则表示属性已成功应用于大型语言模型。
11. 您已全部设置完毕！现在，您可以单击“连接”按钮开始与语音助手对话。请注意，您需要等待几秒钟才能初始化智能体。

### 使用 Azure STT

Azure STT 集成在 RTC 扩展模块中。因此，如果您想使用 Azure STT，您需要选择 `story_teller_integrated_stt` 图表类型。

### 绑定工具

故事大王用例预先配置为使用 `openai_image_generate_tool`，因此通常您无需进行任何更改。

## 实时 V2V

### 前提条件

* 确保您已运行 TEN-Agent Playground。如果没有，请按照[运行 Playground](../../demo) 指南启动 Playground。
* 您需要准备以下信息：
  * 实时 API 密钥
* RTC 信息，目前仅支持 Agora RTC。您可以在 [Agora](https://www.agora.io/) 注册您的帐户。我们假设您在配置 `.env` 文件时已准备好您的 App ID 和 App Certificate。

### 步骤

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择图表类型 `story_teller_realtime`。
3. 单击“模块选择器”以打开模块选择。
4. “V2V”模块已预先配置为选择“OpenAI Realtime”。如果需要，您可以从下拉列表中选择其他 V2V 模块。请注意，您需要将 `prompt` 属性从“OpenAI Realtime”模块复制到新模块，因为切换时模块属性将重置为默认值。
5. 如果您更改了 V2V 模块，请单击“保存更改”以将模块应用于图表；如果您未更改 V2V 模块，则可以跳过此步骤。
6. 单击图表选择右侧的按钮以打开属性配置。您将看到可以为选定的 V2V 模块配置的属性列表。
7. 使用您准备的信息配置“实时 API 密钥”属性。如果您在前几步中更改了 V2V 模块，请务必将 `prompt` 属性从“OpenAI Realtime”模块复制到新模块。
8. 单击“保存更改”以将属性应用于 V2V 模块。
9. 如果您看到成功提示，则表示该属性已成功应用于 V2V 模块。
10. 您已全部设置完毕！现在，您可以单击“连接”按钮开始与语音助手对话。请注意，您需要等待几秒钟才能初始化智能体。

### 绑定工具

故事大王实时用例预先配置为使用 `openai_image_generate_tool`，因此通常您无需进行任何更改。


# 语音助手 (/cn/docs/ten_agent_examples/playground/use-cases/voice-assistant)

本指南将帮助您在 TEN-Agent Playground 中使用大语言模型运行语音助手。

## STT + TTS + LLM

### 前提条件

* 确保您已运行 TEN-Agent Playground。如果没有，请按照[运行 Playground](../../demo) 指南启动 Playground。
* 您需要准备以下信息：
  * STT 信息，可以使用任何支持的 STT。[Deepgram](https://deepgram.com/) 相对容易注册和上手。
  * TTS 信息，可以使用任何支持的 TTS。[Fish.Audio](https://fish.audio/) 相对容易注册和上手。
  * LLM 信息，可以使用任何支持的 LLM。建议使用 [OpenAI](https://openai.com)。
  * RTC 信息，目前仅支持 Agora RTC。您可以在 [Agora](https://www.agora.io/) 注册您的帐户。我们假设您在配置 `.env` 文件时已准备好您的 App ID 和 App Certificate。

### 步骤

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择图表类型 `voice_assistant`。
3. 单击“模块选择器”以打开模块选择。
4. 如果您首选的 STT/TTS 模块默认未选中，您可以从下拉列表中选择模块。请注意，您需要使用 API 密钥等正确信息配置模块。
5. 从“LLM”模块中，单击下拉列表并选择您首选的大型语言模型。
6. 单击“保存更改”以将模块应用于图表。
7. 单击图表选择右侧的按钮以打开属性配置。您将看到可以为选定的大型语言模型配置的属性列表。
8. 使用您准备的信息配置属性。
9. 单击“保存更改”以将属性应用于大型语言模型。
10. 如果您看到成功提示，则表示属性已成功应用于大型语言模型。
11. 您已全部设置完毕！现在，您可以单击“连接”按钮开始与语音助手对话。请注意，您需要等待几秒钟才能初始化智能体。

### 使用 Azure STT

Azure STT 集成在 RTC 扩展模块中。因此，如果您想使用 Azure STT，您需要选择 `voice_assistant_integrated_stt` 图表类型。

### 将天气工具绑定到您的 LLM

您可以在 TEN-Agent Playground 中将天气工具绑定到您的 LLM 模块。
建议使用以下 OpenAI LLM。

1. 当您的智能体运行时。打开模块选择器。
2. 单击 LLM 模块右侧的按钮以打开工具选择。
3. 从弹出列表中选择“天气工具”。
4. 单击“保存更改”以将工具应用于 LLM 模块。
5. 如果您看到成功提示，则表示该工具已成功应用于 LLM 模块。
6. 您已全部设置完毕！现在，您可以通过与智能体对话来询问天气。

## 实时对话 V2V

### 前提条件

* 确保您已运行 TEN-Agent Playground。如果没有，请按照[运行 Playground](../../demo) 指南启动 Playground。
* 您需要准备以下信息：
  * 实时 API 密钥
* RTC 信息，目前仅支持 Agora RTC。您可以在 [Agora](https://www.agora.io/) 注册您的帐户。我们假设您在配置 `.env` 文件时已准备好您的 App ID 和 App Certificate。

### 步骤

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择图表类型 `voice_assistant_realtime`。
3. 单击“模块选择器”以打开模块选择。
4. 从下拉列表中选择您首选的 V2V 模块。
5. 单击“保存更改”以将模块应用于图表。
6. 单击图表选择右侧的按钮以打开属性配置。您将看到可以为选定的 V2V 模块配置的属性列表。
7. 使用您准备的信息配置“实时 API 密钥”属性。
8. 单击“保存更改”以将属性应用于 V2V 模块。
9. 如果您看到成功提示，则表示该属性已成功应用于 V2V 模块。
10. 您已全部设置完毕！现在，您可以单击“连接”按钮开始与语音助手对话。请注意，您需要等待几秒钟才能初始化智能体。

### 将天气工具绑定到您的 V2V

您可以在 TEN-Agent Playground 中将天气工具绑定到您的 V2V 模块。

1. 当您的智能体运行时。打开模块选择器。
2. 单击 V2V 模块右侧的按钮以打开工具选择。
3. 从弹出列表中选择“天气工具”。
4. 单击“保存更改”以将工具应用于 V2V 模块。
5. 如果您看到成功提示，则表示该工具已成功应用于 V2V 模块。
6. 您已全部设置完毕！现在，您可以通过与智能体对话来询问天气。


# Coze 聊天机器人 (/cn/docs/ten_agent_examples/playground/use-cases/voice-assistant/run_coze)

在本教程中，我们将向您展示如何在 TEN-Agent Playground 中让 Coze 机器人说话。

## 前提条件

* 确保您已运行 TEN-Agent Playground。如果没有，请按照[运行 Playground](../../demo) 指南启动 Playground。
* 您需要准备以下信息：
  * Coze 信息：
    * Coze Bot ID
    * Coze Token
    * Coze Base URL（仅当您需要访问非全局环境时才需要）
  * STT 信息，可以使用任何支持的 STT。[Deepgram](https://deepgram.com/) 相对容易注册和上手。
  * TTS 信息，可以使用任何支持的 TTS。[Fish.Audio](https://fish.audio/) 相对容易注册和上手。
  * RTC 信息，目前仅支持 Agora RTC。您可以在 [Agora](https://www.agora.io/) 注册您的帐户。我们假设您在配置 `.env` 文件时已准备好您的 App ID 和 App Certificate。

## 步骤

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择图表类型 `voice_assistant`。
3. 单击“模块选择器”以打开模块选择。
4. 如果您首选的 STT/TTS 模块默认未选中，您可以从下拉列表中选择模块。请注意，您需要使用 API 密钥等正确信息配置模块。
5. 从“LLM”模块中，单击下拉列表并选择“Coze 聊天机器人”。
6. 单击“保存更改”以将模块应用于图表。
7. 单击图表选择右侧的按钮以打开属性配置。您将看到可以为“Coze 聊天机器人”模块配置的属性列表。
8. 使用您准备的信息配置“Coze Bot ID”、“Coze Token”和“Coze Base URL”属性。
9. 单击“保存更改”以将属性应用于“Coze 聊天机器人”模块。
10. 如果您看到成功提示，则表示该属性已成功应用于“Coze 聊天机器人”模块。
11. 您已全部设置完毕！现在，您可以单击“连接”按钮开始与 Coze 机器人对话。请注意，您需要等待几秒钟才能初始化智能体。

## 使用 Azure STT

Azure STT 集成在 RTC 扩展模块中。因此，如果您想使用 Azure STT，您需要选择 `voice_assistant_integrated_stt` 图表类型。


# Dify 聊天机器人 (/cn/docs/ten_agent_examples/playground/use-cases/voice-assistant/run_dify)

在本教程中，我们将向您展示如何在 TEN-Agent Playground 中让 Dify 机器人说话。

## 前提条件

* 确保您已运行 TEN-Agent Playground。如果没有，请按照[运行 Playground](./getting_started) 指南启动 Playground。
* 您需要准备以下信息：
  * Dify 信息：
    * Dify API 密钥
  * STT 信息，可以使用任何支持的 STT。[Deepgram](https://deepgram.com/) 相对容易注册和上手。
  * TTS 信息，可以使用任何支持的 TTS。[Fish.Audio](https://fish.audio/) 相对容易注册和上手。
  * RTC 信息，目前仅支持 Agora RTC。您可以在 [Agora](https://www.agora.io/) 注册您的帐户。我们假设您在配置 `.env` 文件时已准备好您的 App ID 和 App Certificate。

> 您可以使用在 Dify 平台中定义的任何智能体/聊天助手。每个智能体/聊天助手都有自己的 API 密钥。

## 步骤

1. 打开 [localhost:3000](http://localhost:3000) 上的 Playground 以配置您的智能体。
2. 选择图表类型 `voice_assistant`。
3. 单击“模块选择器”以打开模块选择。
4. 如果您首选的 STT/TTS 模块默认未选中，您可以从下拉列表中选择模块。请注意，您需要使用 API 密钥等正确信息配置模块。
5. 从“LLM”模块中，单击下拉列表并选择“Dify 聊天机器人”。
6. 单击“保存更改”以将模块应用于图表。
7. 单击图表选择右侧的按钮以打开属性配置。您将看到可以为“Dify 聊天机器人”模块配置的属性列表。
8. 使用您准备的信息配置“Dify API 密钥”属性。
9. 单击“保存更改”以将属性应用于“Dify 聊天机器人”模块。
10. 如果您看到成功提示，则表示该属性已成功应用于“Dify 聊天机器人”模块。
11. 您已全部设置完毕！现在，您可以单击“连接”按钮开始与 Dify 机器人对话。请注意，您需要等待几秒钟才能初始化智能体。

## 使用 Azure STT

Azure STT 集成在 RTC 扩展模块中。因此，如果您想使用 Azure STT，您需要选择 `voice_assistant_integrated_stt` 图表类型。

## 故障排除

如果您遇到任何问题，请检查以下各项：

* 确保您具有 Dify 聊天机器人的正确 API 密钥。
* 确保在 Dify 中，您的聊天助手配置了有效的模型密钥。
* 确保您选择了并配置了正确的 STT 和 TTS 模块。
* 确保您选择了正确的图表类型。