Test Google Vertex AI Models with Function Calling

This promptfoo example tests Google Vertex AI models with function calling and tool callbacks. It demonstrates function calling and tool declarations with Gemini 3.8 Flash, streamed function-call arguments with callback execution against local implementations, and two different configuration approaches - YAML and JavaScript.

The basic tool-declaration config (promptfooconfig.yaml, with function definitions in tools.json) uses external tool definitions and validates that function calls are produced correctly, without actually executing them; it selects gemini-3.8-flash on the global endpoint for both the target and the text grader, and its semantic assertion uses Vertex's text-embedding-005 model - a separate, cloud-specific embedding namespace from Gemini's native embeddings. The function-callbacks config (promptfooconfig-callback.mjs) is a portable ESM configuration with inline tools and callbacks, including a local function implementation for adding numbers, and demonstrates assembling streamed function-call arguments and actually executing the resulting function. Because callbacks execute as trusted, unsandboxed local code, runs that involve untrusted models or evaluated content should be isolated.

Use it to validate Vertex AI's function-calling behavior on Gemini 3.8 Flash, or to see how to assemble streamed function-call arguments and wire them into real local function execution via callbacks. Because the callback example runs arbitrary local code untrusted-input-driven, don't run it against untrusted models or content without isolating the process first.

npx promptfoo@latest init --example google-vertex-tools
cd google-vertex-tools

Requires the google-auth-library npm package, the Vertex AI API enabled on a Google Cloud project (gcloud config set project PROJECT_ID), and authentication via gcloud auth application-default login, an authorized service account on the running machine, or a downloaded service-account credentials JSON referenced by GOOGLE_APPLICATION_CREDENTIALS. Run the basic example with promptfoo eval -c promptfooconfig.yaml (validates the model calls the weather function with correct parameters) or the callback example with promptfoo eval -c promptfooconfig-callback.mjs (validates the streamed arguments assemble correctly and the addition function computes the right result), then inspect results with promptfoo view.

Uses promptfoo's Vertex AI provider against Google Cloud's Vertex AI API, authenticated through the Google Auth Library.

Developers building on Vertex AI's function-calling and streaming tool-call APIs who want working promptfoo examples for both declarative (YAML) tool validation and executable (JavaScript) callback-driven tool use.