googlecloudplatform/dataflowtemplates/add-source-datastream-to-spanner
Claude Code and agent skill by googlecloudplatform/dataflowtemplates
- community
Checking that the SKILL.md file is still in the repository.
Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template. Details scope boundaries, prerequisites, connector implementations, registry registrations, unit testing, and smoke testing guidelines.
Install
npx vskill@latest install googlecloudplatform/dataflowtemplates/add-source-datastream-to-spannerSKILL.md
The same SKILL.md was published earlier as googlecloudplatform/dataflowtemplates/add-source-datastream-to-spanner.
Skill: Implement Source in Datastream to Spanner Template
Overview
This skill provides a step-by-step procedure for adding support for a new database source connector to the v2/datastream-to-spanner template. It details prerequisites, scope boundaries, type mapping requirements, connector implementation, registry registration, unit testing, and smoke testing procedures.
Prerequisites
[!CRITICAL]MANDATORY PREREQUISITE GATE:Before running code searches, inspecting files, or executing unit tests, you MUST verify if the user provided the inputs below.If any required input is missing, STOP IMMEDIATELY and ask the user for clarification before proceeding further.
- Datastream Source Compatibility Check: Verify that Google Cloud Datastream natively supports the requested source database type by referencing Datastream Supported Sources.
- > [!CRITICAL]
- > If the requested source database is NOT supported by Datastream, STOP IMMEDIATELY and inform the user that Datastream does not support CDC streaming for this database source. Do NOT proceed with inspection, implementation, or testing.
- Datatype Mapping File: Ask the user to provide or point to the Datatype Mapping Matrix for the new source database, defining the mappings to Spanner GoogleSQL and PostgreSQL dialects.
- Test Setup Details: Ask the user for the test environment details required for live smoke testing, including:
- Source Database Instance details (host/instance name, database name, and credentials/connection method).
- Target Spanner Instance & Database details.
- Datastream stream / connection profile setup or GCP project information.
Architectural Boundaries & Code Scope
All implementation for a new source connector MUST be strictly confined to:
- Source Connector Package:
v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/<source_type>/
- Implementation of
IDsToSpSourceConnector(<Source>DsToSpSourceConnector.java) - Source-specific change event processing classes in package
<source_type>(implemented similar to other existing source connectors in the codebase).
- Connector Registry:
DatastreamToSpannerSourceConnectorRegistry.java
- Dynamic connector registration via
register(new <Source>DsToSpSourceConnector()).
- Shared Core Registries & Configs:
v2/spanner-common
- Registries and configuration files in
spanner-common(updated similar to existing sources).
Datatype Mapping Matrix Requirements
Consult the Datatype Mapping Matrix provided in the prerequisites to verify correct datatype conversion to Spanner GoogleSQL and PostgreSQL dialects. Ensure proper alignment for character, numeric, temporal, binary, boolean, JSON and any other datatypes as specified in the mapping file.
Step-by-Step Implementation Workflow
Step 0: Mandatory Prerequisite Gate
- Verify Datastream Support: Check if the requested source database type is supported by Datastream per Datastream Sources. If unsupported, STOP IMMEDIATELY and notify the user.
- Verify Inputs: Inspect the request for the required inputs. If the Datatype Mapping Matrix and Test Setup Details are not provided in the user request, DO NOT run any inspection or execution tools. Stop and ask the user for the missing details first.
Step 1: Implement <Source>DsToSpSourceConnector
Implement IDsToSpSourceConnector in com.google.cloud.teleport.v2.templates.source.<source_type>:
- Define CDC metadata key constants required by the source.
- Implement any other methods in this class to work for GoogleSQL and PostgreSQL Spanner dialects similar to other supported sources.
Step 2: Implement Source Processing Classes
Implement remaining source-specific change event context and sequence classes in package com.google.cloud.teleport.v2.templates.source.<source_type> similar to other existing source connectors. Skip the types that are not supported by Datastream. These are mentioned in the type mapping skill.
Step 3: Register Source in Registries & Config Files
- Register connector in
DatastreamToSpannerSourceConnectorRegistry.java:
``java register(new <Source>DsToSpSourceConnector()); ``
- Update corresponding registries and converter files in
v2/spanner-commonsimilar to existing sources.
Step 4: Unit Testing & Verification
Execute Maven unit tests for the template:
All unit tests must pass with BUILD SUCCESS.
Step 5: Mandatory Live Smoke Testing & Verification
[!CRITICAL]MANDATORY EXECUTION REQUIREMENT:Immediately after unit tests pass, you MUST AUTOMATICALLY PROCEED to execute live end-to-end smoke testing using the test environment details provided in the prerequisites.Do NOT stop, pause, or declare completion after unit testing without running the live smoke tests.
[!IMPORTANT]WORKER MACHINE TYPE DIRECTIVE & FALLBACK RULE:When submitting the Dataflow job for live smoke testing, you MUST explicitly specify a worker machine type of the correct size with at least 4 vCPUs.If--worker-machine-type(orWORKER_MACHINE_TYPE) is not explicitly specified by the user or.env.testing, you MUST ALWAYS default to `--worker-machine-type=n2-standard-4`.Omitting this parameter or submittingnull/blank will cause Dataflow job launch validation to fail with a machine specification policy violation.
- Environment Setup:
- Connect to the live source database instance and target Spanner instance configured in the test setup.
- Perform CRUD Operations:
- INSERT: Execute
INSERTstatements covering supported datatypes in the source database. - UPDATE: Update existing rows in the source database.
- DELETE: Delete test rows from the source database.
- Verify Replication Flow:
- Verify that CDC events flow through Datastream to Cloud Storage and Dataflow.
- Query Cloud Spanner tables to confirm that
INSERT,UPDATE, andDELETEoperations are accurately reflected in Spanner.
- Retry Loop on Failure:
- If any CRUD operation fails to replicate or produces data discrepancies in Spanner, inspect error logs, modify the connector and converter code, rebuild, and re-test until all operations pass cleanly.
Security
- Security scanPassed, 90 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingv2/datastream-to-spanner/.agents/skills/add-source-datastream-to-spanner/SKILL.md
Popularity
- Repository stars
- 1,315
- Repository forks
- 1,119
- 7-day trend score
- 36
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/googlecloudplatform/dataflowtemplates/add-source-datastream-to-spanner)