Spark
Sign In
Prompt Chain

Test Google Vertex AI Models with Function Calling

Promptfoo examples testing Vertex AI function calling - one validating tool call shape, one actually executing a callback function.

Copy chain
Works with google vertex ai
Promptfoo
Maintainer?
92
Spark score
out of 100
Updated 15 days ago
Version 0.121.18
Models
gemini 2 0
Add to Favorites

Why it matters

Streamline the testing of Google Vertex AI models, specifically focusing on their function calling and tool callback capabilities. Ensure your AI integrations are robust and reliable before deployment.

Outcomes

What it gets done

01

Configure prompt chains for Google Vertex AI.

02

Implement and test function calling for AI models.

03

Utilize tool callbacks for enhanced model interaction.

04

Generate test cases for AI model behavior.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-google-vertex-tools | bash

Steps

Steps in the chain

01
Install Google Auth Library
02
Enable Vertex AI API
03
Configure Google Cloud Project
04
Set up Authentication
05
Run Basic Tool Declaration Example
06
Run Function Callback Example
07
View Results

Overview

Google Vertex Tools

This example demonstrates Vertex AI function calling both as tool-call validation without execution and as a working callback that actually runs a local function. Use it when testing whether a Vertex AI model produces correct tool calls, or when the tool call needs to actually execute against real logic.

What it does

This example demonstrates Google Vertex AI's function calling in two complementary ways: basic tool declaration, where the model returns a structured function call that's validated but not executed, and function callbacks, where a local JavaScript implementation actually runs the called function and the computed result is checked.

When to use - and when NOT to

Use it when you want to test whether a Vertex AI model produces correct tool calls, or when you need the tool call to actually execute against real logic - the callback example demonstrates a working addition function, not just a validated call shape. Do not use it if Vertex AI's function calling capability is not yet enabled in your Google Cloud project, or if you only need YAML-based config with no custom JavaScript execution.

Inputs and outputs

Requires the google-auth-library package, the Vertex AI API enabled on your project (gcloud config set project PROJECT_ID), and authentication via gcloud auth application-default login, a machine-level authorized service account, or a GOOGLE_APPLICATION_CREDENTIALS service-account JSON path. The basic example uses external tool definitions in tools.json for a weather lookup; the callback example defines tools and a local addition-function implementation inline in promptfooconfig-callback.js.

Integrations

Runs via npx promptfoo@latest init --example google-vertex-tools, then promptfoo eval -c promptfooconfig.yaml for the basic tool-declaration example, which validates the model calls the weather function with correct parameters, or promptfoo eval -c promptfooconfig-callback.js for the callback example, which actually executes the addition function and validates the computed result. View results with promptfoo view.

Who it's for

Developers building Vertex AI function-calling integrations who need both a validation-only reference config and a working example of executing a called function locally.

Source README

google-vertex-tools (Google Vertex Tools)

Example configurations for testing Google Vertex AI models with function calling and tool callbacks.

You can run this example with:

npx promptfoo@latest init --example google-vertex-tools
cd google-vertex-tools

Purpose

This example demonstrates how to use Vertex AI models with:

  • Function calling and tool declarations
  • Function callback execution with local implementations
  • Different configuration approaches (YAML vs JavaScript)

Prerequisites

  1. Install the Google Auth Library:

    npm install google-auth-library

  2. Enable the Vertex AI API in your Google Cloud project

  3. Configure your Google Cloud project:

    gcloud config set project PROJECT_ID

  4. Set up authentication using one of these methods:

    • Authenticate with your Google account:

      gcloud auth application-default login

    • Use a machine with an authorized service account

    • Use service account credentials file:

      1. Download your service account JSON

      2. Set the credentials path:

        export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json

Configurations

This example includes two different approaches:

Basic Tool Declaration (promptfooconfig.yaml)

Uses external tool definitions and validates function calls without execution:

  • promptfooconfig.yaml - YAML configuration with external tools
  • tools.json - Function definitions for weather lookup

Function Callbacks (promptfooconfig-callback.js)

Demonstrates actual function execution with local callbacks:

  • promptfooconfig-callback.js - JavaScript configuration with inline tools and callbacks
  • Includes local function implementation for adding numbers

Running the Examples

  1. Basic tool declaration example:

    promptfoo eval -c promptfooconfig.yaml

  2. Function callback example:

    promptfoo eval -c promptfooconfig-callback.js

  3. View results:

    promptfoo view

Expected Results

  • Basic example: Validates that the model correctly calls the weather function with proper parameters
  • Callback example: Actually executes the addition function and validates the computed results

Learn More

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.