googlecloudplatform/dataflowtemplates/add-source-functional-integ-test
Claude Code and agent skill by googlecloudplatform/dataflowtemplates
- community
- ext:[{"type":"hook","description":"Pre/post lifecycle hooks"}]
Checking that the SKILL.md file is still in the repository.
Skill for adding functional integ test for new source by taking reference from existing MySQL integ tests.
Install
npx vskill@latest install googlecloudplatform/dataflowtemplates/add-source-functional-integ-testSemi-extensibleMentions customization, but needs skill-specific knowledge. Read the extensible skills standard
SKILL.md
The same SKILL.md was published earlier as googlecloudplatform/dataflowtemplates/add-source-functional-integ-test.
New Source Functional Integration Test Generation Skill
This skill instructs an AI Coding Agent to generate and verify integration tests for a new database source scenario (e.g. migrating from a reference source to a target source) for Dataflow Templates.
1. Goal
Given a Scenario ID, read its configuration from manifest.yaml, locate all reference resources and code, discover logical data type mappings, translate schemas/DDLs and the concrete Java test class, verify via compilation, and self-heal any compilation failures.
1.1 Initialization Check
Before you begin parsing or executing any core steps, you MUST verify the following inputs and dependencies exist:
- Scenario ID
- Manifest File Path
- Target Source Database Name
- Reference Datatype Mapping Matrix File Path
- Testing Environment Setup Path: You MUST proactively check the workspace root directory for a
.envfile namedtesting_execution.env. - Mapping Matrix Schema Validation:
You must dynamically validate the .csv mapping file against the canonical schema.
- Use the
read_url_contenttool to fetch the canonical reference:https://raw.githubusercontent.com/GoogleCloudPlatform/spanner-migration-tool/master/.agents/skills/source_research_helper/sampleOutput/mysql_datatype_mapping_matrix.csv - Parse the headers (first line) of both the fetched sample and the local file at
Reference Datatype Mapping Matrix File Path. - Verify that every column header present in the fetched sample is also present in the local provided matrix (a subset match; the local matrix may contain extra columns).
If ANY of the prompt inputs are missing, or if the testing_execution.env file does not exist, or if the provided mapping matrix is missing any canonical headers, you MUST HALT EXECUTION IMMEDIATELY. Do not attempt to guess, hallucinate paths, or proceed. Output a direct question asking for the missing inputs, missing file, or reporting the explicitly missing columns.
1.2 Prerequisites: Execution Configuration
The agent must look for a testing_execution.env file in the workspace root with the following schema:
1.3 Directory Segregation Rule
To prevent cluttering the root templates package, you MUST place all generated files into a strictly nested directory structure matching the lowercased name of the target database.
- Java Class Path:
src/test/java/com/google/cloud/teleport/v2/templates/{target_db_name_lowercase}/{TestClassName}.java - Java Package:
package com.google.cloud.teleport.v2.templates.{target_db_name_lowercase}; - SQL Resources Path:
src/test/resources/{target_db_name_lowercase}/{TestClassName}/
1.4 Global Constraint: Production Code Priority Rule
[!CRITICAL]Production Code Priority Rule (Source Over Test):This is a global constraint that applies to all phases of this skill (context discovery, type mapping, code generation, compilation, execution, and debugging).If an integration test fails during compilation or execution, you MUST first assume that the bug lies in the production template source code (e.g., dialect adapters, DML generators, schema mappers, or exception classifiers) rather than the test harness or DDL.* Systematic Investigation: You must thoroughly analyze the template's core Java engine to identify if it is failing to support the target database's dialect behavior correctly.* No Test Hacking: You are strictly forbidden from "hacking", disabling, or loosening a test assertion or DDL constraint to bypass a failure, unless the production template is proven to be behaving 100% correctly and the mismatch is purely due to database-specific padding/formatting differences.* Documentation: Every change to the production template must be documented in your final report with a root-cause explanation, and any test adaptation must be explicitly justified in the code with JavaDocs.
2. Step 1: Input Parsing & Scenario Scouting
- The user MUST provide the path to the
manifest.yamlscenario registry file in their prompt. Open that file. - Locate the scenario matching the input
Scenario ID. - Extract:
context: Fully qualified name of the reference test class (e.g.com.google.cloud.teleport.v2.templates.MySQLDataTypesIT).template: Path to the template module (e.g.v2/sourcedb-to-spanner).spanner_dialect: Target Spanner database dialect (GOOGLE_STANDARD_SQLorPOSTGRESQL). If not explicitly defined in the manifest for the scenario, default toGOOGLE_STANDARD_SQL.type(optional): Set todatatypesfor exhaustive type-mapping integration tests.type_mappings(optional): A pre-defined list of mappings. Each mapping specifies:source_type: The data type in the reference source database (e.g.,VARCHAR(21000)).spanner_type: The desired Spanner data type (e.g.,STRING(MAX)).
3. Step 2: Context Discovery (Scouting)
To avoid hallucinating helper methods or base configurations, you must gather all relevant context:
[!IMPORTANT]Strict Context Constraint: Anchor all context discovery strictly starting from the reference class file extracted in Step 1. Do NOT search for or inspect other unrelated integration tests (e.g., other PostgreSQL or MySQL tests) in the codebase unless they are direct ancestors or referenced resources of the reference class. Inspecting unrelated test files can lead to hallucinating mismatched setup methods, parent classes, or helper signatures.
- Locate Reference Class File:
- Perform a glob/grep search to find the file matching the context class name on disk.
- Scan Inheritance Chain:
- Open the reference class file and inspect its
extends <SuperClass>signature. - Recursively search for and open all parent/base classes up the inheritance chain to read all available setup and helper APIs.
- Scan Referenced Resource Files:
- In the reference class file and parent classes, look for string constants pointing to any resource files (e.g., SQL schemas, Avro files, JSON mappings, CSVs).
- Locate and read all of these referenced resource files from the repository.
4. Step 3: Logical Type Mapping Discovery
Because you are generating functional tests (e.g., sharding, foreign keys) and not exhaustively generating boundary data, you MUST translate the DDL schemas using the exact data types defined in the provided CSV Ground Truth mapping.
- Ingest the CSV Mapping File:
- Read the
.csvfile provided in your prompt. - Parse the matrix to understand how default types are mapped for the target Spanner Dialect (
GOOGLE_STANDARD_SQLvsPOSTGRESQL).
- Translate Types:
- As you scan the reference DDL schema from Step 2, identify the column data types.
- Use the ingested CSV matrix as a strict dictionary to translate those column types into the Target Source Type and the matching Target Spanner Type.
- No User Approval Required:
- Because this mapping file was vetted in a prior phase (or curated by the user), it is the absolute source of truth.
- Do NOT construct a markdown mapping table to ask the user for approval.
- Proceed immediately to Step 4 (DDL Schema Translation).
5. Step 4: DDL Schema Translation
Identify whether case sensitivity is required to match Spanner schemas, and perform quoting adjustments:
[!IMPORTANT]Case Preservation & Identifier Quoting:To ensure generated tests and schemas run successfully, you must align and preserve the exact character casing of all table names, column names, keys, and constraints between the target source database and the target Spanner database.1. Check Case Preservation Requirement:- First check if the reference schema, target Spanner schema, or target test uses any uppercase or mixed-case identifiers.- Check if the target database folds unquoted identifiers to a different case by default (e.g., some database folds unquoted names to lowercase, or another database folds to uppercase).- If the target database folds unquoted identifiers, AND there are mixed-case/uppercase names, case preservation is REQUIRED.- If all identifiers are already completely lowercase, case preservation is NOT required (and you should NOT wrap them in redundant quotes).2. Apply Quoting if Case Preservation is Required:- Wrap all identifiers (table names, column names, keys, etc.) in dialect-appropriate quotes in the generated DDL statements.- Do Not Alter Original Intent: You MUST NOT alter the original casing of the reference schema identifiers (e.g., converting mixed-case names to all-caps or all-lowercase) just to trivially bypass the database's folding rules. You MUST apply quotes to preserve the original intended mixed-casing exactly as it was provided in the reference.3. Casing Parity Between Target Source and Spanner:- In some cases (such as data type ITs), target schema table names might differ from the reference test schema.- However, whatever names are used for the target source schema, their exact character case MUST be identical in the target Spanner database schema DDL.
5.1 Target Source Schema DDL
- Translate the reference source DDL schema file(s) into native SQL statements compatible with the target source database dialect.
- The column data types in the generated SQL schema MUST use the standard equivalent target source types identified in the logical type mapping table.
- Apply the Case Preservation & Identifier Quoting check and rule to prevent the target database from folding identifier cases.
- Save the generated DDL file to the target template resources directory using the pattern:
{target_source_dialect}-schema.sql.
5.2 Target Spanner Schema DDL
- Translate the reference Spanner DDL SQL file to the target Spanner database dialect (
GOOGLE_STANDARD_SQLorPOSTGRESQL). - Align column names, table structures, and primary keys with the target source schema, ensuring the exact character case of all table and column names matches the target source schema exactly. Apply the Case Preservation & Identifier Quoting rule if targeting Spanner PostgreSQL dialect.
- Important: For Spanner
GOOGLE_STANDARD_SQLdialect, do NOT use double quotes (") or single quotes (') to quote identifiers (tables, columns, etc.) as they are not supported. Only use backticks (```) if escaping is required; otherwise, leave identifiers unquoted.
- Save the translated Spanner DDL SQL file to the target template resources directory using the pattern:
{target_source_dialect}-{spanner_dialect}-spanner-schema.sql.
6. Step 5: Infrastructure Lifecycle & Resource Manager Setup
6.1 Source Database ResourceManager lookup and check
To provision infrastructure resources for integration tests, you MUST inherit the ResourceManager architectural pattern used by the reference_test_class. If the reference test delegates setup to a method in a base class (e.g., an ITBase.java class), you MUST declare a matching method for your target database in that same base class if it doesn't already exist.
This is the start of the file. Read the full SKILL.md on GitHub.
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingv2/spanner-common/.agents/skills/add-source-functional-integ-test/SKILL.md
Popularity
- Repository stars
- 1,315
- Repository forks
- 1,119
- 7-day trend score
- 36
Stars and forks are for the whole GitHub repository, not this skill alone. Refreshed 31 Oct 2026, 18 days ago.
Extension points
hookPre/post lifecycle hooks
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/googlecloudplatform/dataflowtemplates/add-source-functional-integ-test)