googlecloudplatform/dataflowtemplates/add-source-datatype-integ-test

Claude Code and agent skill by googlecloudplatform/dataflowtemplates

Trust tier 1
  • community
  • ext:[{"type":"hook","description":"Pre/post lifecycle hooks"}]
Source check pending

Checking that the SKILL.md file is still in the repository.

Skill for adding data type integ tests for new source by provided datatype_mapping_matrix.

Install

npx vskill@latest install googlecloudplatform/dataflowtemplates/add-source-datatype-integ-test

Semi-extensibleMentions customization, but needs skill-specific knowledge. Read the extensible skills standard

SKILL.md

The same SKILL.md was published earlier as googlecloudplatform/dataflowtemplates/add-source-datatype-integ-test.

New Source DataTypes Integration Test Generation Skill

This skill instructs an AI Coding Agent to design and synthesize an exhaustive, dialect-wide integration test suite for Dataflow Templates for a new database source dialect from first principles, without being biased by a reference database test.


1. Goal

Given a Scenario ID, parse its configuration from manifest.yaml, gather structural class/helper context from a reference test (without copying its columns/assertions), read provided datatype mapping matrix and compile a comprehensive list of all native datatypes supported by the target database dialect (including recommended and alternative Spanner mappings), generate native source SQL schemas containing CREATE and INSERT statements with type-specific formats, write a complete Java integration test from scratch with dynamic assertions, compile-verify, and self-heal.

1.1 Initialization Check

Before you begin parsing or executing any core steps, you MUST verify the following inputs and dependencies exist:

  1. Scenario ID
  2. Manifest File Path
  3. Target Source Database Name
  4. Reference Datatype Mapping Matrix File Path
  5. Testing Environment Setup Path: You MUST proactively check the workspace root directory for a .env file named testing_execution.env.
  6. Mapping Matrix Schema Validation:

You must dynamically validate the .csv mapping file against the canonical schema.

  1. Use the read_url_content tool to fetch the canonical reference: https://raw.githubusercontent.com/GoogleCloudPlatform/spanner-migration-tool/master/.agents/skills/source_research_helper/sampleOutput/mysql_datatype_mapping_matrix.csv
  2. Parse the headers (first line) of both the fetched sample and the local file at Reference Datatype Mapping Matrix File Path.
  3. Verify that every column header present in the fetched sample is also present in the local provided matrix (a subset match; the local matrix may contain extra columns).

If ANY of the prompt inputs are missing, or if the testing_execution.env file does not exist, or if the provided mapping matrix is missing any canonical headers, you MUST HALT EXECUTION IMMEDIATELY. Do not attempt to guess, hallucinate paths, or proceed. Output a direct question asking for the missing inputs, missing file, or reporting the explicitly missing columns.


1.2 Prerequisites: Execution Configuration

The agent must look for a testing_execution.env file in the workspace root with the following schema:

env
REMOTE_WORKSPACE_PATH=
REMOTE_EXEC_TEMPLATE=
REMOTE_SYNC_TEMPLATE=
TEST_PROJECT=
TEST_REGION=
TEST_SPANNER_INSTANCE=
TEST_STAGE_BUCKET=
TEST_ARTIFACT_BUCKET=
TEST_HOST_IP=
TEST_PRIVATE_CONNECTIVITY=
TEST_DB_USER=
TEST_DB_PASSWORD=

1.3 Directory Segregation Rule

To prevent cluttering the root templates package, you MUST place all generated files into a strictly nested directory structure matching the lowercased name of the target database.

  • Java Class Path: src/test/java/com/google/cloud/teleport/v2/templates/{target_db_name_lowercase}/{TestClassName}.java
  • Java Package: package com.google.cloud.teleport.v2.templates.{target_db_name_lowercase};
  • SQL Resources Path: src/test/resources/{target_db_name_lowercase}/{TestClassName}/

1.4 Global Constraint: Production Code Priority Rule

[!CRITICAL]
Production Code Priority Rule (Source Over Test):
This is a global constraint that applies to all phases of this skill (context discovery, type mapping, code generation, compilation, execution, and debugging).
If an integration test fails during compilation or execution, you MUST first assume that the bug lies in the production template source code (e.g., dialect adapters, DML generators, schema mappers, or exception classifiers) rather than the test harness or DDL.
* Systematic Investigation: You must thoroughly analyze the template's core Java engine to identify if it is failing to support the target database's dialect behavior correctly.
* No Test Hacking: You are strictly forbidden from "hacking", disabling, or loosening a test assertion or DDL constraint to bypass a failure, unless the production template is proven to be behaving 100% correctly and the mismatch is purely due to database-specific padding/formatting differences.
* Documentation: Every change to the production template must be documented in your final report with a root-cause explanation, and any test adaptation must be explicitly justified in the code with JavaDocs.

2. Step 1: Input Parsing & Scenario Scouting

  1. The user MUST provide the path to the manifest.yaml scenario registry file in their prompt. Open that file.
  2. Locate the scenario matching the input Scenario ID.
  3. Extract:
  • context: Fully qualified name of the reference test class (e.g., com.google.cloud.teleport.v2.templates.<ReferenceSourceDialect>DataTypesIT).
  • template: Path to the template module (e.g., v2/<template_path>).
  • spanner_dialect: Target Spanner database dialect (GOOGLE_STANDARD_SQL or POSTGRESQL). If not explicitly defined, default to GOOGLE_STANDARD_SQL.
  • Check if a baseline data type mapping file has been supplied in the prompt/context (e.g., mapping.json or mapping.yaml).
  1. Determine the Migration Direction:
  • Inspect the template path and the context class inheritance chain (scouted in Step 2):
  • If the reference class extends a forward migration base class (e.g., <SourceDbToSpannerITBaseClass>) or the template path contains a forward template name (e.g., sourcedb-to-spanner or datastream-to-spanner): Forward Migration (Source Database -> Spanner).
  • If the reference class extends a reverse migration base class (e.g., <SpannerToSourceDbITBaseClass>) or the template path contains a reverse template name (e.g., spanner-to-sourcedb): Reverse Migration (Spanner -> Target Database).

3. Step 2: Context Scouting (Structural Analysis Only)

To build a compilable integration test, you must analyze the structural patterns of the template test suite.

[!IMPORTANT]
Strict Scouting Constraint:
Open the reference class, its parent classes, and configuration base classes strictly to discover:
- The class inheritance chain (e.g., extends <BaseITClass>).
- Setup helper APIs (e.g., setUp<SourceDialect>ResourceManager(), setUpSpannerResourceManager()).
- The pipeline launching structure (e.g., launchDataflowJob(...), pipelineOperator()).
You MUST NOT copy the list of tables, columns, data types, test values, or assertions defined in the reference class. The new test suite must be built independently from first principles.
  1. Locate Reference Class File:
  • Perform a glob/grep search to find the file matching the context class name on disk.
  1. Scan Inheritance Chain:
  • Open the reference class file and inspect its extends <SuperClass> signature.
  • Recursively search for and open all parent/base classes up the inheritance chain to read all available setup and helper APIs.

4. Step 3: Ingesting the Ground Truth Mapping

Instead of researching the database dialect from scratch, this skill assumes a complete, human-approved data type mapping matrix has been provided in CSV format (the "Ground Truth"). Your task is to ingest this mapping and use it to strictly drive all subsequent schema and code generation.

[!IMPORTANT]
Mandatory Input Rule:
You MUST receive the exact path to the initial mapping .csv file from the user or Orchestrator in your prompt.
You MUST NOT perform independent web research, nor should you invent types that are not listed in this CSV file. Treat this file as the absolute, exhaustive dictionary of what must be tested.
  1. Ingest the Mapping File:
  • Read the provided mapping .csv file.
  • Parse the matrix into memory.
  • Use the "Source Database Type / Alias" column to identify the native source database type.
  1. Identify Dialect Validations:
  • Check the spanner_dialect determined in Step 1 (GOOGLE_STANDARD_SQL vs POSTGRESQL).
  • Fetch the mapping strictly from the corresponding Spanner Dialect column in the CSV (e.g., "Spanner GoogleSQL Default Datatype", "Spanner GoogleSQL Alternative Datatypes", "Spanner PostgreSQL Default Datatype", "Spanner PostgreSQL Alternative Datatypes").
  1. Identify Validation Targets:
  • Extract every row that is intended for Default or Alternative mapping (Scenarios A and B).
  • Primary Keys (Scenario C): You MUST explicitly read the "Is Source Datatype Supported as PK?" column in the matrix. If supported (Yes or similar), you MUST apply the specific mapped type provided in the "Spanner GoogleSQL Default Datatype If Column is PK" or "Spanner PostgreSQL Default Datatype If Column is PK" column depending on the target dialect.
  • Unsupported/Complex Types (Scenario D): Extract every row marked as Unsupported or Complex. To identify rows for Scenario D, filter for any rows where "Datastream Support Status" is marked as unsupported (or equivalent phrasing like 'No' or 'Unsupported').
  1. No User Approval Required:
  • Because this mapping file was vetted in a prior phase, do NOT ask the user for approval. Proceed immediately to Step 4.

5. Step 4: DDL Schema Synthesis (Tested Scenarios)

For every discovered datatype and every identified type alias, you must generate schemas covering the following scenarios:

  • Scenario A: Default Type Migration: Regular column using recommended/default Spanner mapping.
  • Naming Pattern: Table Name: {type_or_alias}_table, Target Column: {type_or_alias}_col.
  • Scenario B: Alternative Type Migration: Regular column using an alternative Spanner mapping.
  • Naming Pattern: Table Name: {type_or_alias}_to_{clean_target_type}_table, Target Column: {type_or_alias}_to_{clean_target_type}_col.
  • Scenario C: Primary Key Mapping: Testing the data type or alias as part of the primary key constraint.
  • Naming Pattern: Table Name: {type_or_alias}_pk_table, Target Column: {type_or_alias}_pk_col as the primary key.
  • Feasibility & Column Override: You MUST only generate Scenario C if the matrix explicitly marks Is the datatype supported as a PK in the source? as Yes. Check the If Column is a PK column for the target dialect, as you might need to use a different Spanner type (e.g., BYTES or restricted STRING) when the key is indexed.
  • SMT Range Splitter Constraint: You MUST exclude all floating-point and decimal/numeric types (e.g. FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL with fractional scales) from Scenario C (is_pk_feasible = False). During range queries, SMT's range splitter casts partition boundaries to Long integers, truncating the fractional bounds and causing extreme boundary float rows to be silently dropped during migration.
  • Scenario D: Unsupported/Complex Types: Tables containing unsupported spatial, collection, or complex types to verify graceful migration and row-count consistency without pipeline failures.
  • Scenario E: Array/Collection Types: Tables explicitly testing arrays, lists, or collections of all supported scalar base types.

This is the start of the file. Read the full SKILL.md on GitHub.

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
1,315
Repository forks
1,119
7-day trend score
36

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Extension points

  • hookPre/post lifecycle hooks

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/googlecloudplatform/dataflowtemplates/add-source-datatype-integ-test/badge)](https://verified-skill.com/skills/googlecloudplatform/dataflowtemplates/add-source-datatype-integ-test)

Something wrong with this skill or its page? Tell us.