Automate Video Calls with AI Agents

Make your meetings accessible to AI Agents 🤖

joinly.ai is a connector middleware designed to enable AI agents to join and actively participate in video calls. Through its MCP server, joinly.ai provides essential meeting tools and resources that can equip any AI agent with the skills to perform tasks and interact with you in real time during your meetings.

Want to dive right in? Jump to the Quickstart! Want to know more? Visit our website!

:sparkles: Features

• Live Interaction: Lets your agents execute tasks and respond in real-time by voice or chat within your meetings
• Conversational flow: Built-in logic that ensures natural conversations by handling interruptions and multi-speaker interactions
• Cross-platform: Join Google Meet, Zoom, and Microsoft Teams (or any available over the browser)
• Bring-your-own-LLM: Works with all LLM providers (also locally with Ollama)
• Choose-your-preferred-TTS/STT: Modular design supports multiple services - Whisper/Deepgram for STT and Kokoro/ElevenLabs/Deepgram for TTS (and more to come...)
• 100% open-source, self-hosted and privacy-first :rocket:

:video_camera: Demos

GitHub

In this demo video, joinly answers the question 'What is Joinly?' by accessing the latest news from the web. It then creates an issue in a GitHub demo repository.

Notion

In this demo video, we connect joinly to our notion via MCP and let it edit the content of a page content live in the meeting.

Any ideas what we should build next? Write us! :rocket:

:zap: Quickstart

Run joinly via Docker with a basic conversational agent client.

Create a new folder joinly or clone this repository (not mandatory for the following steps). In this directory, create a new .env file with a valid API key for the LLM provider you want to use, e.g. OpenAI:

# .env
# for OpenAI LLM
# change key and model to your desired one
JOINLY_LLM_MODEL=gpt-4o
JOINLY_LLM_PROVIDER=openai
OPENAI_API_KEY=your-openai-api-key

Pull the Docker image (~2.3GB since it packages browser and models):

docker pull ghcr.io/joinly-ai/joinly:latest

Launch your meeting in Zoom, Google Meet or Teams and let joinly join the meeting using the meeting link as <MeetingURL>. Then, run the following command from the folder where you created the .env file:

docker run --env-file .env ghcr.io/joinly-ai/joinly:latest --client <MeetingURL>

:red_circle: Having trouble getting started? Let's figure it out together on our discord!

:technologist: Run a custom client

In Quickstart, we ran the Docker Container directly as a client using --client. But we can also run it as a server and connect to it from outside the container, which allows us to connect other MCP servers. Here, we run a custom client using the joinly-client package and connect it to the joinly MCP server.

Start the joinly server in the first terminal (note, we are not using --client here and publish port 8000 on localhost only):

docker run -p 127.0.0.1:8000:8000 ghcr.io/joinly-ai/joinly:latest

While the server is running, start the example client implementation in the second terminal window to connect to it and join a meeting:

uvx joinly-client --env-file .env <MeetingUrl>

Add MCP servers to the client

Add the tools of any MCP server to the agent by providing a JSON configuration. The configuration file can contain multiple entries under "mcpServers" which will all be available as tools in the meeting (see fastmcp client docs for config syntax):

{
    "mcpServers": {
        "localServer": {
            "command": "npx",
            "args": ["-y", "package@0.1.0"]
        },
        "remoteServer": {
            "url": "http://mcp.example.com",
            "auth": "oauth"
        }
    }
}

Add for example a Tavily config for web searching, then run the client using the config file, here named config.json:

uvx joinly-client --env-file .env --mcp-config config.json <MeetingUrl>

:wrench: Configurations

Configurations can be given via env variables and/or command line args. Here is a list of common configuration options, which can be used when starting the docker container:

docker run --env-file .env -p 127.0.0.1:8000:8000 ghcr.io/joinly-ai/joinly:latest <MyOptionArgs>

Alternatively, you can pass --name, --lang, and provider settings as command line arguments using joinly-client, which will override settings of the server:

uvx joinly-client <MyOptionArgs> <MeetingUrl>

Basic Settings

In general, the docker image provides an MCP server which is started by default. But to quickly get started, we also include a client implementation that can be used via --client. Note, in this case no server is started and no other client can connect to it.

# Start directly as client; default is as server, to which a custom client can connect
--client <MeetingUrl>

# Change participant name (default: joinly)
--name "AI Assistant"

# Change language of TTS/STT (default: en)
# Note, availability depends on the TTS/STT provider
--lang de

# Change host & port of the joinly MCP server
--host 0.0.0.0 --port 8000

Providers

Text-to-Speech

# Kokoro (local) TTS (default)
--tts kokoro
--tts-arg voice=<VoiceName>  # optionally, set different voice

# ElevenLabs TTS, include ELEVENLABS_API_KEY in .env
--tts elevenlabs
--tts-arg voice_id=<VoiceID>  # optionally, set different voice

# Deepgram TTS, include DEEPGRAM_API_KEY in .env
--tts deepgram
--tts-arg model_name=<ModelName>  # optionally, set different model (voice)

Transcription

# Whisper (local) STT (default)
--stt whisper
--stt-arg model_name=<ModelName>  # optionally, set different model (default: base), for GPU support see below

# Deepgram STT, include DEEPGRAM_API_KEY in .env
--stt deepgram
--stt-arg model_name=<ModelName>  # optionally, set different model

Debugging

# Start browser with a VNC server for debugging;
# forward the port and connect to it using a VNC client
--vnc-server --vnc-server-port 5900

# Logging
-v  # or -vv, -vvv

# Help
--help

GPU Support

We provide a Docker image with CUDA GPU support for running the transcription and TTS models on a GPU. To use it, you need to have the NVIDIA Container Toolkit installed and CUDA >= 12.6. Then pull the CUDA-enabled image:

docker pull ghcr.io/joinly-ai/joinly:latest-cuda

Run as client or server with the same commands as above, but use the joinly:{version}-cuda image and set --gpus all:

# Run as server
docker run --gpus all --env-file .env -p 8000:8000 ghcr.io/joinly-ai/joinly:latest-cuda -v
# Run as client
docker run --gpus all --env-file .env ghcr.io/joinly-ai/joinly:latest-cuda -v --client <MeetingURL>

By default, the joinly image uses the Whisper model base for transcription, since it still runs reasonably fast on CPU. For cuda, it automatically defaults to distil-large-v3 for significantly better transcription quality. You can change the model by setting --stt-arg model_name=<model_name> (e.g., --stt-arg model_name=large-v3). However, only the respective default models are packaged in the docker image, so it will start to download the model weights on container start.

:test_tube: Create your own agent

You can also write your own agent and connect it to our joinly MCP server. See the code examples for the joinly-client package or the client_example.py if you want a starting point that doesn't depend on our framework.

The joinly MCP server provides following tools and resources:

Tools

• join_meeting - Join meeting with URL, participant name, and optional passcode
• leave_meeting - Leave the current meeting
• speak_text - Speak text using TTS (requires text parameter)
• send_chat_message - Send chat message (requires message parameter)
• mute_yourself - Mute microphone
• unmute_yourself - Unmute microphone
• get_chat_history - Get current meeting chat history in JSON format
• get_participants - Get current meeting participants in JSON format
• get_transcript - Get current meeting transcript in JSON format, optionally filtered by minutes
• get_video_snapshot - Get an image from the current meeting, e.g., view a current screenshare

Resources

• transcript://live - Live meeting transcript in JSON format, including timestamps and speaker information. Subscribable for real-time updates when new utterances are added.

:building_construction: Developing joinly.ai

For development we recommend using the development container, which installs all necessary dependencies. To get started, install the DevContainer Extension for Visual Studio Code, open the repository and choose Reopen in Container.

The installation can take some time, since it downloads all packages as well as models for Whisper/Kokoro and the Chromium browser. At the end, it automatically invokes the download_assets.py script. If you see errors like Missing kokoro-v1.0.onnx, run this script manually using:

uv run scripts/download_assets.py

We'd love to see what you are using it for or building with it. Showcase your work on our discord

:pencil2: Roadmap

Meeting

• Meeting chat access
• Camera in video call with status updates
• Enable screen share during video conferences
• Participant metadata and joining/leaving
• Improve browser agent capabilities

Conversation

• Speaker attribute for transcription
• Improve client memory: reduce token usage, allow persistence across meetings events
• Improve End-of-Utterance/turn-taking detection
• Human approval mechanism from inside the meeting

Integrations

• Showcase how to add agents using the A2A protocol
• Add more provider integrations (STT, TTS)
• Integrate meeting platform SDKs
• Add alternative open-source meeting provider
• Add support for Speech2Speech models

:busts_in_silhouette: Contributing

Contributions are always welcome! Feel free to open issues for bugs or submit a feature request. We'll do our best to review all contributions promptly and help merge your changes.

Please check our Roadmap and don't hesitate to reach out to us!

:memo: License

This project is licensed under the MIT License ‒ see the LICENSE file for details.

:speech_balloon: Getting help

If you have questions or feedback, or if you would like to chat with the maintainers or other community members, please use the following links:

• Join our Discord
• Explore our GitHub Discussions

examples/config_tavily.json

{
    "mcpServers": {
        "tavily": {
            "command": "npx",
            "args": ["-y", "tavily-mcp@0.2.2"]
        }
    }
}

examples/client_example.py

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "fastmcp",
#     "langchain",
#     "langchain-anthropic",
#     "langchain-mcp-adapters",
#     "langchain-ollama",
#     "langchain-openai",
#     "langgraph",
#     "py-dotenv",
#     "rich",
# ]
# ///

import asyncio
import contextlib
import datetime
import json
import logging
import os
import re

from fastmcp import Client
from fastmcp.client.transports import StreamableHttpTransport
from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage
from langchain_core.tools import tool
from langchain_mcp_adapters.tools import load_mcp_tools
from langgraph.checkpoint.memory import MemorySaver
from langgraph.prebuilt import ToolNode, create_react_agent
from mcp import ResourceUpdatedNotification, ServerNotification
from pydantic import AnyUrl, BaseModel

logger = logging.getLogger(__name__)


class TranscriptSegment(BaseModel):
    """A segment of a transcript."""

    text: str
    start: float
    end: float
    speaker: str | None = None


class Transcript(BaseModel):
    """A transcript containing multiple segments."""

    segments: list[TranscriptSegment]


def transcript_to_messages(transcript: Transcript) -> list[HumanMessage]:
    """Convert a transcript to a list of HumanMessage."""

    def _normalize_speaker(speaker: str | None) -> str:
        if speaker is None:
            return "Unknown"
        speaker = re.sub(r"\s+", "_", speaker.strip())
        return re.sub(r"[<>\|\\\/]+", "", speaker)

    return [
        HumanMessage(
            content=s.text,
            name=_normalize_speaker(s.speaker),
        )
        for s in transcript.segments
    ]


def transcript_after(transcript: Transcript, after: float) -> Transcript:
    """Get a new transcript including only segments starting after given time."""
    segments = [s for s in transcript.segments if s.start > after]
    return Transcript(segments=segments)


def log_chunk(chunk) -> None:  # noqa: ANN001
    """Log an update chunk from langgraph."""
    if "agent" in chunk:
        for m in chunk["agent"]["messages"]:
            for t in m.tool_calls or []:
                args_str = ", ".join(
                    f'{k}="{v}"' if isinstance(v, str) else f"{k}={v}"
                    for k, v in t.get("args", {}).items()
                )
                logger.info("%s: %s", t["name"], args_str)
    if "tools" in chunk:
        for m in chunk["tools"]["messages"]:
            logger.info("%s: %s", m.name, m.content)


async def run(  # noqa: C901
    mcp_url: str,
    meeting_url: str,
    model_name: str,
    model_provider: str | None = None,
    config: dict | None = None,
) -> None:
    """Simple conversational agent for a meeting.

    Args:
        mcp_url: The URL of the MCP server.
        meeting_url: The URL of the meeting to join.
        model_name: The model to use for the agent.
        model_provider: The provider for the model.
        config: Optional configuration for additional MCP servers.
    """
    transcript_url = AnyUrl("transcript://live")
    transcript_event = asyncio.Event()

    async def _message_handler(message) -> None:  # noqa: ANN001
        if (
            isinstance(message, ServerNotification)
            and isinstance(message.root, ResourceUpdatedNotification)
            and message.root.params.uri == transcript_url
        ):
            transcript_event.set()

    llm = init_chat_model(model_name, model_provider=model_provider)

    prompt = (
        f"Today is {datetime.datetime.now(tz=datetime.UTC).strftime('%d.%m.%Y')}. "
        "You are joinly, a professional and knowledgeable meeting assistant. "
        "Provide concise, valuable contributions in the meeting. "
        "You are only with one other participant in the meeting, therefore "
        "respond to all messages and questions. "
        "When you are greeted, respond politely in spoken language. "
        "Give information, answer questions, and fullfill tasks as needed. "
        "You receive real-time transcripts from the ongoing meeting. "
        "Respond interactively and use available tools to assist participants. "
        "Always finish your response with the 'finish' tool. "
        "Never directly use the 'finish' tool, always respond first and then use it. "
        "If interrupted mid-response, use 'finish'."
    )

    # optionally, set settings for joinly (requires v0.3.2)
    settings = {
        # "name": "joinly",  # noqa: ERA001
        # "language": "en",  # noqa: ERA001
        # "tts": "elevenlabs",  # noqa: ERA001
    }
    transport = StreamableHttpTransport(
        url=mcp_url, headers={"joinly-settings": json.dumps(settings)}
    )

    # separate joinly client, since fastmcp does not support notifications
    # in proxy server mode yet (v2.7.0)
    joinly_client = Client(transport, message_handler=_message_handler)
    client = Client(config) if config and config.get("mcpServers") else None

    mcp_servers = list(config.get("mcpServers", {}).keys()) if config else None
    logger.info(
        "Connecting to joinly MCP server at %s and following other MCP servers: %s",
        mcp_url,
        mcp_servers,
    )
    async with joinly_client, client or contextlib.nullcontext():
        if joinly_client.is_connected():
            logger.info("Connected to joinly MCP server")
        else:
            logger.error("Failed to connect to joinly MCP server at %s", mcp_url)
        if client and not client.is_connected():
            logger.error("Failed to connect to additional MCP servers: %s", mcp_servers)

        await joinly_client.session.subscribe_resource(transcript_url)

        @tool(return_direct=True)
        def finish() -> str:
            """Finish tool to end the turn."""
            return "Finished."

        # load tools from joinly and other MCP servers
        tools = await load_mcp_tools(joinly_client.session)
        if client:
            tools.extend(await load_mcp_tools(client.session))
        tools.append(finish)

        tool_node = ToolNode(tools, handle_tool_errors=lambda e: e)
        llm_binded = llm.bind_tools(tools, tool_choice="any")

        memory = MemorySaver()
        agent = create_react_agent(
            llm_binded, tool_node, prompt=prompt, checkpointer=memory
        )
        last_time = -1.0

        logger.info("Joining meeting at %s", meeting_url)
        await joinly_client.call_tool("join_meeting", {"meeting_url": meeting_url})
        logger.info("Joined meeting successfully")

        while True:
            await transcript_event.wait()
            transcript_full = Transcript.model_validate_json(
                (await joinly_client.read_resource(transcript_url))[0].text  # type: ignore[attr-defined]
            )
            transcript = transcript_after(transcript_full, after=last_time)
            transcript_event.clear()
            if not transcript.segments:
                logger.warning("No new segments in the transcript after update")
                continue

            last_time = transcript.segments[-1].start
            for segment in transcript.segments:
                logger.info(
                    '%s: "%s"',
                    segment.speaker if segment.speaker else "User",
                    segment.text,
                )

            try:
                async for chunk in agent.astream(
                    {"messages": transcript_to_messages(transcript)},
                    config={"configurable": {"thread_id": "1"}},
                    stream_mode="updates",
                ):
                    log_chunk(chunk)
            except Exception:
                logger.exception("Error during agent invocation")


if __name__ == "__main__":
    import argparse
    from pathlib import Path

    from dotenv import load_dotenv
    from rich.logging import RichHandler

    load_dotenv()

    logging.basicConfig(
        level=logging.WARNING,
        format="%(message)s",
        datefmt="[%X]",
        handlers=[RichHandler(rich_tracebacks=True)],
    )
    logger.setLevel(logging.INFO)

    parser = argparse.ArgumentParser(
        description=(
            "Run a conversational agent for a meeting using joinly.ai. "
            "Optionally, connect to different MCP servers."
        )
    )
    parser.add_argument("meeting_url", help="The URL of the meeting to join")
    parser.add_argument(
        "--mcp-url",
        dest="mcp_url",
        default="http://localhost:8000/mcp/",
        help="The URL of the joinly MCP server",
    )
    parser.add_argument(
        "--model-name",
        dest="model_name",
        default=os.getenv("JOINLY_MODEL_NAME", "gpt-4o"),
        help="The model to use for the agent",
    )
    parser.add_argument(
        "--model-provider",
        dest="model_provider",
        default=os.getenv("JOINLY_MODEL_PROVIDER"),
        help="The provider for the model",
    )
    parser.add_argument(
        "--config",
        dest="config",
        type=str,
        default=None,
        help=(
            "Path to a JSON configuration file for additional MCP servers. "
            "The file should contain configuration like: "
            '\'{"mcpServers": {"remote": {"url": "https://example.com/mcp"}}}\'. '
            "See https://gofastmcp.com/clients/client for more details."
        ),
    )
    args = parser.parse_args()
    config = None
    if args.config:
        try:
            with Path(args.config).open("r") as f:
                config = json.load(f)
        except Exception:
            logger.exception("Failed to load configuration file")
            args.config = None

    asyncio.run(
        run(
            mcp_url=args.mcp_url,
            meeting_url=args.meeting_url,
            model_name=args.model_name,
            model_provider=args.model_provider,
            config=config,
        )
    )

scripts/download_assets.py

import argparse
import logging
import os
import pathlib
import subprocess
import sys
import urllib.request

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def download_playwright() -> None:
    """Download Playwright browser."""
    logger.info("Downloading Playwright browser")
    playwright_cmd = ["playwright", "install", "--no-shell", "chromium"]
    subprocess.run(playwright_cmd, check=True)  # noqa: S603
    logger.info("Playwright browser downloaded successfully")


def download_whisper(model_name: str) -> None:
    """Download Whisper model."""
    from faster_whisper import WhisperModel

    logger.info("Downloading Whisper model %s", model_name)
    _ = WhisperModel(model_name)
    logger.info("Whisper model downloaded successfully")


def download_assets(
    cache_subdir: str,
    file_urls: list[str],
    description: str | None = None,
) -> pathlib.Path:
    """Download a set of assets into a cache directory.

    Args:
        cache_subdir: subdirectory under XDG_CACHE_HOME (default ~/.cache) to store
            assets.
        file_urls: list of URLs pointing to the files to download.
        description: optional description for logging (e.g., "Kokoro model and voices").

    Returns:
        Path to the cache directory containing the downloaded assets.
    """
    if description:
        logger.info("Downloading %s", description)
    cache_dir = (
        pathlib.Path(os.getenv("XDG_CACHE_HOME", "~/.cache")).expanduser()
        / cache_subdir
    )
    cache_dir.mkdir(parents=True, exist_ok=True)

    bar_len = 40  # width of the textual progress bar

    for url in file_urls:
        filename = url.rsplit("/", 1)[-1]
        dest = cache_dir / filename
        if dest.exists():
            logger.info("[cached] %s", filename)
            continue

        def _reporthook(block_num: int, block_size: int, total_size: int) -> None:
            if total_size <= 0 or not sys.stdout.isatty():
                return
            downloaded = block_num * block_size
            ratio = min(downloaded / total_size, 1.0)
            filled = int(bar_len * ratio)
            bar = "=" * filled + "-" * (bar_len - filled)
            sys.stdout.write(f"\r{filename} [{bar}] {ratio * 100:6.2f}%")  # noqa: B023
            sys.stdout.flush()
            if downloaded >= total_size:
                sys.stdout.write("\n")

        logger.info("Downloading %s", filename)
        if not url.startswith(("http:", "https:")):
            msg = f"URL must start with 'http:' or 'https:'. Got: {url}"
            raise ValueError(msg)
        urllib.request.urlretrieve(url, dest, _reporthook)  # noqa: S310
        logger.info("Saved %s to %s", filename, dest)

    if description:
        logger.info("%s downloaded successfully", description)
    return cache_dir


def download_silero_vad() -> None:
    """Download Silero VAD model."""
    file_urls = [
        "https://raw.githubusercontent.com/snakers4/silero-vad/v5.0/files/silero_vad.onnx",
        "https://raw.githubusercontent.com/snakers4/silero-vad/master/LICENSE",
    ]
    download_assets("silero", file_urls, "Silero VAD v5 ONNX model")


def download_kokoro() -> None:
    """Download Kokoro model and voices."""
    file_urls = [
        "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.onnx",
        "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/voices-v1.0.bin",
        "https://raw.githubusercontent.com/thewh1teagle/kokoro-onnx/main/LICENSE",
        "https://www.apache.org/licenses/LICENSE-2.0.txt",
    ]
    download_assets("kokoro", file_urls, "Kokoro v1.0 ONNX model and voices")


def parse_args() -> argparse.Namespace:
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(description="Download assets for the project.")
    parser.add_argument(
        "--assets",
        nargs="*",
        choices=["playwright", "whisper", "kokoro", "silero", "all"],
        default=["all"],
        help="Specify which assets to download (default: all)",
    )
    parser.add_argument(
        "--whisper-model",
        type=str,
        default="base",
        help="Whisper model to download (default: base)",
    )
    return parser.parse_args()


def main() -> None:
    """Download assets for the project."""
    args = parse_args()

    assets = args.assets

    if "playwright" in assets or "all" in assets:
        download_playwright()
    if "whisper" in assets or "all" in assets:
        download_whisper(model_name=args.whisper_model)
    if "silero" in assets or "all" in assets:
        download_silero_vad()
    if "kokoro" in assets or "all" in assets:
        download_kokoro()


if __name__ == "__main__":
    main()