grind-lab-core/night_owl_research_agent/data-download

Data Download: Discover, Evaluate, and Acquire Research Data

Find and download data for: $ARGUMENTS

Overview

This skill turns a data need into a validated, documented, locally available dataset. It does NOT blindly download from the first URL found. It reasons from the research question to the right data, evaluates candidate sources, downloads reproducibly, validates the result, and records provenance.

The workflow is always: data need → source discovery → source evaluation → download → validation → provenance documentation.

Constants

• DATA_DIR = `data/` — Top-level data directory. Raw downloads go to data/raw/, processed outputs to data/processed/.
• MANIFEST_FILE = `data/DATA_MANIFEST.md` — Provenance log for all downloaded datasets.
• MAX_UNATTENDED_SIZE_MB = 500 — If a single file exceeds this size, pause and confirm with the user before downloading.
• VERIFY_AFTER_DOWNLOAD = true — Always validate downloaded files (existence, size, format, openability).
• PREFER_PUBLIC = true — When both public and authenticated sources exist, prefer the public source unless the authenticated source is materially better.

Override via argument, e.g., /data-download "US census tracts" — data dir: datasets/, max size: 2000.

Phase 0: Understand the Data Need

Before searching for data, classify what is actually needed.

Step 0.1: Infer Data Requirements

Extract from the user's request, the research question, or existing project files (research_contract.md, output/EXPERIMENT_PLAN.md, RESEARCH_PLAN.md):

If the request is too vague (e.g., "get me some data"), STOP and ask the user to specify at least:

• What geographic area?
• What variables or themes?
• What time period?

Step 0.2: Categorize the Data Need

Map the need to one or more data categories:

Write the data requirement summary to data/DOWNLOAD_PLAN.md:

# Data Download Plan
 
**Task**: [user's request or research question]
**Date**: [today]
 
## Required Datasets
 
### Dataset 1: [descriptive name]
-Category: [from table above]
-Geography: [study area]
-Time range: [period]
-Spatial resolution: [resolution]
-Variables: [list]
-Format preference: [format]
-Priority: REQUIRED / NICE-TO-HAVE
 
### Dataset 2: [descriptive name]
...
 
## Constraints
-Licensing: [requirement]
-Size budget: [limit]
-Authentication: [any known requirements]

Phase 1: Discover Data Sources

Step 1.1: Source Search Strategy

For each required dataset, search for sources in this priority order:

1. Official government portals — Census.gov, data.gov, USGS, NOAA, EPA, Eurostat, national mapping agencies
2. Intergovernmental organizations — UN, World Bank, WHO, FAO, OECD
3. Well-known research repositories — NASA Earthdata, Copernicus, PANGAEA, Zenodo, Figshare, Dryad
4. Established open data projects — OpenStreetMap, Natural Earth, GADM, WorldPop, WorldClim
5. Cloud-hosted public data — AWS Open Data, Google Earth Engine, Microsoft Planetary Computer, Source Cooperative
6. Official APIs — Census API, NOAA API, EPA API, Open-Meteo, Overpass API
7. Academic project pages — with clear documentation, DOI, and citation

NEVER use as primary source:

• Random blog posts or personal websites
• Undocumented file-sharing links
• Preview/visualization tiles meant for display only (e.g., map tile servers are NOT data downloads)
• Scraped HTML tables when a documented API or download endpoint exists
• Sources without clear provenance or licensing

Step 1.2: Source Catalog

For each candidate source, record:

### Source: [name]
-URL: [download page or API endpoint]
-Provider: [organization]
-Format: [file format(s) available]
-Coverage: [geographic and temporal]
-Resolution: [spatial and temporal]
-License: [license type]
-Access: PUBLIC / API_KEY / LOGIN / INSTITUTIONAL / PAID
-Documentation: [link to docs/metadata]
-Last updated: [date or frequency]
-Stability: HIGH (government/institution) / MEDIUM (research project) / LOW (personal)
-Citation: [how to cite]

Phase 2: Evaluate and Select Sources

Step 2.1: Evaluation Criteria

Score each candidate source:

Step 2.2: Access Classification and Pause Logic

Classify each source by access type and act accordingly:

GUARDRAIL: Never fabricate, guess, or hardcode API keys, passwords, or tokens. Never attempt to bypass authentication, CAPTCHAs, or access controls. Never assume the user has credentials unless they explicitly provide them.

Step 2.3: Present Options When Tradeoffs Exist

If multiple sources exist with material tradeoffs, PAUSE and present the options:

I found multiple sources for [data need]:
 
1. [Source A] — PUBLIC, 30m resolution, 2021, GeoTIFF
   + No login required, direct download
   - Slightly older, coarser resolution
 
2. [Source B] — API_KEY required, 10m resolution, 2023, COG
   + Higher resolution, more recent
   - Requires free API key registration at [URL]
 
3. [Source C] — INSTITUTIONAL, 1m resolution, 2024
   + Best resolution and most current
   - Requires university login
 
Which source should I use? Or should I proceed with Source A (public, no login)?

Let the user decide. If no response and the public source is adequate, default to the public source.

Step 2.4: Size Check

Before downloading, estimate the file size if possible (from documentation, HTTP HEAD request, or API metadata).

If estimated size > MAX_UNATTENDED_SIZE_MB:

The download for [dataset] is approximately [X] MB.
This exceeds the [MAX_UNATTENDED_SIZE_MB] MB threshold.
 
Should I proceed? Options:
1. Download the full file ([X] MB)
2. Download a spatial subset (specify bounding box or region)
3. Skip this dataset for now

Phase 3: Download

Step 3.1: Directory Setup

from pathlib import Path
data_dir = Path('data')
raw_dir = data_dir / 'raw'
processed_dir = data_dir / 'processed'
raw_dir.mkdir(parents=True, exist_ok=True)
processed_dir.mkdir(parents=True, exist_ok=True)

Organize raw downloads into subdirectories by dataset name:

data/
├── raw/
│   ├── census_tracts_2020/
│   │   ├── tl_2020_us_tract.shp
│   │   ├── tl_2020_us_tract.dbf
│   │   └── ...
│   ├── nlcd_2021/
│   │   └── nlcd_2021_land_cover.tif
│   └── noaa_hurricanes/
│       └── ibtracs_NA.csv
├── processed/
│   └── [user creates these later]
└── DATA_MANIFEST.md

Step 3.2: Download Methods

Choose the download method based on the source type:

Method A: Direct HTTP Download (preferred for single files)

import requests
from pathlib import Path
def download_file(url, dest_path, chunk_size=8192):
    """Download a file with progress tracking and integrity check."""
    dest_path = Path(dest_path)
    dest_path.parent.mkdir(parents=True, exist_ok=True)
    response = requests.get(url, stream=True, timeout=60)
    response.raise_for_status()
    total = int(response.headers.get('content-length', 0))
    downloaded = 0
    with open(dest_path, 'wb') as f:
        for chunk in response.iter_content(chunk_size=chunk_size):
            f.write(chunk)
            downloaded += len(chunk)
    file_size = dest_path.stat().st_size
    if total > 0 and file_size != total:
        raise ValueError(f"Size mismatch: expected {total}, got {file_size}")
    print(f"Downloaded {dest_path.name} ({file_size / 1e6:.1f} MB)")
    return dest_path