mr-tbot/bot-code-mods/auto-rewrite

/auto-rewrite

Find code in this repo that came from somewhere else, prove it either way, and give the user real options for clearing it. Two questions per suspect region, in order:

1. Is it actually copied? — evidence, not vibes.
2. If so, what is the cheapest lawful remedy? — usually not a rewrite.

Not legal advice, and say so once. You are doing engineering triage: locating code, establishing provenance, and costing remedies. A lawyer decides what it means. Name doctrines and cases by name only — never invent a page cite, a docket number, or a section number. A fabricated citation destroys the credibility of a report whose entire value is credibility.

Step 0 — Scope And Trigger

Ask what prompted this, because it sets the depth and the deadline:

Then write down the scope: which paths are the user's own work, which are declared third-party (vendor/, third_party/, Pods/, node_modules/, external/), which are generated, which are test fixtures. Exclusions get stated in the report — a silent exclusion reads as a clean bill of health for code nobody looked at.

Declared third-party code is out of scope here and in scope for auto-license-check. The two skills split on the question: this one asks "did we copy it", that one asks "may we use it".

Pass 1 — Internal Duplication Sweep

Cheap, local, no network. Finds copy-paste inside the repo and gives you the hot files to investigate externally.

npx --yes jscpd@5 . \
  --min-tokens 50 --min-lines 5 \
  --format typescript,kotlin,swift,java,python,go,rust \
  --ignore "**/node_modules/**,**/build/**,**/vendor/**,**/*.min.*,**/generated/**" \
  --reporters console,json --output ./.audit/jscpd --blame

--format and --ignore are not optional — pointed at a repo root unfiltered, jscpd lexes logs, Markdown and JSON and reports them as duplication. --blame attaches authorship, which is the part you actually want. jscpd finds Type-1 and Type-2 clones (exact, and renamed identifiers); a copy that was edited afterwards slips through.

Cross-check with a second engine — agreement between a token-based and a line-based detector is a real signal:

pmd cpd --minimum-tokens 100 --language kotlin --dir src \
  --ignore-identifiers --ignore-literals --format xml --report-file .audit/cpd.xml

PMD CPD takes one `--language` per run and defaults to java, so a polyglot repo needs one invocation per language and a wrong flag yields a confident-looking zero. Exit code 4 means duplications were found, not that the run failed.

For "did our tree come from that tree", use the reference-directory shape:

pipx run --spec copydetect copydetect -t ./our-src -r ./upstream-src -b ./boilerplate -O .audit/copydetect.html

Do not use MOSS. It uploads your source to a third-party server, is licensed for non-commercial educational use, and publishes matches at an unlisted public URL. Dolos (npm i -g @dodona/dolos) does the same winnowing algorithm locally. CodeQL has no clone detection — those queries were removed in 2.8.3 and only ever worked on the retired LGTM service; any instruction to run them is citing dead functionality.

Pass 2 — External Provenance

Does this code exist upstream? Run cheapest-first.

Software Heritage — the exact-match oracle. ~350M archived origins including dead forges. The lookup key is sha1_git, not sha1 — the same file returns opposite answers under the two hashes:

H=$(git hash-object suspect.c)
curl -s -X POST -H 'Content-Type: application/json' \
  -d "[\"swh:1:cnt:$H\"]" https://archive.softwareheritage.org/api/1/known/

Batch up to 1000 SWHIDs per POST (1200 returns HTTP 413); anonymous limit is 120 requests per hour per IP. known: true on a file the user claims to have written is a strong copy signal. Two hard limits: it is exact-match only — one reformat, one renamed variable, one stripped header and it returns false — and it will not tell you where the file came from, because /api/1/provenance/whereis/ requires allowlisted permission and returns 401 to everyone else. Pull the archived bytes to diff with /api/1/content/sha1_git:<h>/raw/.

ClearlyDefined — turns "looks like lodash" into "IS lodash 4.17.21 `package/_apply.js`". It publishes a per-file hash manifest for published packages:

S=$(sha256sum suspect.js | cut -d' ' -f1)
curl -4 -s 'https://api.clearlydefined.io/definitions/npm/npmjs/-/lodash/4.17.21' \
  | jq --arg s "$S" '.files[] | select(.hashes.sha256==$s)'

The -4 is load-bearing — the host's IPv6 address black-holes and the request hangs to timeout. Coverage is published packages only; code lifted straight out of a git repo has no definition.

Sourcegraph — best free cross-forge full-text and regex search:

curl -sG --max-time 60 --data-urlencode 'q=context:global "DISTINCTIVE LITERAL" count:100 fork:yes archived:yes' \
  --data 'v=V3' --data 't=literal' https://sourcegraph.com/.api/search/stream

fork:yes archived:yes is mandatory for this job — forks and archived repos are excluded by default, and the fork is exactly where copied code usually lives. Index is ~2M repos, so absence is not evidence.

GitHub code search works but is sharper-edged than it looks: authentication required, its own 10 requests/minute bucket, a 1000-result cap, files over 384 KB unindexed, default branch only, no regex — and scoping qualifiers silently return zero (repo:, user:, org: and path: queries returned 0 even for GitHub's own documented examples). Use it for unscoped global searches; do not use it to confirm a hit inside a named repository. Expect occasional HTTP 408 on broad phrases and retry.

gh search code 'DISTINCTIVE LITERAL' --limit 50 --json repository,path,textMatches
gh api /rate_limit --jq '.resources.code_search'

ScanOSS matches snippets against an OSS knowledge base. It transmits fingerprints, so use scanoss-py scan --obfuscate on client or NDA'd code, and generate fingerprints locally first (scanoss-py fingerprint -o) if you want to inspect what leaves the machine.

Pass 3 — Local Forensics

Signals that survive when search finds nothing:

• Git pickaxe. git log -S'<literal>' finds additions/removals of a string; git log -G'<regex>'

matches diff text and therefore catches blocks that were edited afterwards. -S silently misses an edited copy. Add --all --full-history -- <path> for files deleted and re-added, --find-copies-harder for moves.

• Import events. A single commit adding thousands of lines with no incremental history, no

authorship trail, and a message like "add feature" is the shape of a paste.

• Style divergence. A file whose brace style, naming, comment voice, error handling, or tab width

disagrees with everything around it. Formatters mask this — check the commit that introduced it.

• Header archaeology. Copyright lines, SPDX tags, license blocks, @author tags, TODOs naming a

different product, non-English comments, URLs in comments, and — the loudest of all — a stripped header (git log -p the file's first commit).

• AST-level diff. difft --display side-by-side ours.c theirs.c compares structure, so it

survives reformatting and reordering that defeat git diff --ignore-all-space.

• Stack Overflow vintage. SO answers are CC BY-SA, and the version is fixed by the answer's post

date: 2.5 before 2011-02-08, 3.0 from 2011-02-08 to 2018-05-02, 4.0 after. That is a share-alike attribution license landing in a proprietary codebase — a real finding, not a nitpick.

Triage — Do Not Manufacture Findings

A false accusation is worse than a missed one. Before a match becomes a finding, rule out:

• Not copyrightable in the first place: short snippets, standard algorithms, API names and

signatures, facts, and expression dictated by function (merger) or by convention (scènes à faire).

• Common idiom: the pattern every project in that framework produces. Test it — search the

literal; tens of thousands of hits means idiom, not theft.

• Generated or scaffolded: create-react-app, protobuf output, Room/Core Data boilerplate, IDE

templates, Gradle wrappers.

• Direction of travel: the user's project may be the upstream. Compare first-commit dates

(gh api graphql for createdAt) before assuming which way it flowed.

• Same author: the user may have written it elsewhere first — which raises an ownership question

(employment, contractor terms, IP assignment) rather than a copying one, and that is often the real issue.

Classify every survivor:

The Report

One file, .audit/provenance-report.md, ordered by severity. Per finding: file and line range, what matched, the evidence (hash, URL, search that found it, commit SHA), the suspected upstream and its license, the classification, and the remediation options with a real cost estimate for each.

State the coverage honestly at the top: what was scanned, what was excluded, which tools ran, and what those tools structurally cannot see. A provenance report that implies completeness it does not have is the failure mode.

Remediation Ladder

Stop at the first rung that clears the problem. A rewrite is the fifth option, not the first.

1. Ask for permission or a license. Cheapest and most overlooked. But check contributor count and

whether a CLA exists first — on a multi-contributor project with no assignment, no single maintainer can grant it.

1. Confirm it is a problem at all. Uncopyrightable, licensed-compatible, or already permitted?

Then there is nothing to fix.

1. Comply. Most permissive licenses want attribution and license text — add NOTICE /

THIRD-PARTY-LICENSES, restore the header, record it in the SBOM. Minutes of work.

1. Isolate. Move the code behind a boundary that changes the obligation (a separate process, a

dynamically-linked module, a service). Whether this works is license-specific and lawyer territory — propose it, do not promise it.

1. Replace. Swap in a differently-licensed equivalent library. Usually cheaper than a rewrite and

the result is maintained by someone else.

1. Clean-room rewrite. See below.
2. Remove the feature. Always on the table; sometimes correct.

What does not clear anything — and an agent will be tempted by all four: renaming variables, reordering functions, reformatting, translating to another language, or feeding the original to an LLM and shipping its output. Each produces a derivative work while destroying the evidence that you knew. The LLM path has a second edge: output with no human authorship may not be protectable as the user's own, so they can end up with code that is both derivative of the original and unownable.