joshmock/dotfiles/web-search

Hister search

A Hister instance runs locally on localhost:4433. If the hister CLI is available to you, use hister search to search my browser history for relevant pages. If it is not, use the HTTP API to search instead. Search results from Hister have a high likelihood of containing relevant information.

Jina search

If the jina CLI tool is available, and pass agents/jina-reader/pi exists, ALWAYS use jina search to do a web search. Ensure JINA_API_KEY=$(pass agents/jina-reader/pi) env var is set.

If jina is not installed or pass does not return a key, continue to the next section.

SearXNG Search

SearXNG is a metasearch engine that runs locally. It aggregates results from multiple search engines and package repositories, returning clean JSON output.

NOTE: SearXNG typically runs on localhost:8080 but, if you are sandboxed, it may appear at host.docker.internal:8080.

agent-browser

As a last resort, if Jina and SearXNG do not yield useful results, use the agent-browser skill and runtime.

Quick Reference

Available Categories

Run to see all categories:

curl -s "http://localhost:8080/config" | jq '.categories'

Notable categories:

• general: General web search (default)
• cargo: Rust crates from crates.io
• packages: Multi-repo (npm, rubygems, haskell/hoogle, hex, packagist, metacpan, pub.dev, pkg.go.dev, docker hub, alpine, etc.)
• it: IT/tech resources (includes GitHub, Docker Hub, crates.io)
• repos: Code repositories
• scientific publications: Academic papers

JSON Response Structure

{
  "query": "search term",
  "number_of_results": 0,
  "results": [
    {
      "url": "https://example.com",
      "title": "Result Title",
      "content": "Snippet of content...",
      "publishedDate": "2025-01-01T00:00:00",
      "engine": "duckduckgo",
      "engines": ["duckduckgo", "startpage"],
      "score": 3.0,
      "category": "general"
    }
  ],
  "answers": [],          // Direct answers/infoboxes
  "suggestions": [],      // Search suggestions
  "corrections": [],      // Query corrections
  "infoboxes": [],       // Knowledge panels
  "unresponsive_engines": []
}

Common Usage Patterns

1. Package Repository Searches

Cargo/Rust crates:

curl -s "http://localhost:8080/search?q=tokio&format=json&categories=cargo" | \
  jq '.results[] | {title, url, content}'

npm packages:

curl -s "http://localhost:8080/search?q=express&format=json&categories=packages" | \
  jq '.results[] | select(.engines[] == "npm") | {title, url, content}'

PyPI packages (workaround - see below):

# PyPI engine is enabled but not returning results in current SearXNG config
# Use direct API or qypi CLI instead (see PyPI Workaround section)

2. Web Search with Filtering

IT/Tech search:

curl -s "http://localhost:8080/search?q=rust+async&format=json&categories=it" | \
  jq '.results[0:5] | .[] | {title, url, engines}'

GitHub repositories:

curl -s "http://localhost:8080/search?q=machine+learning&format=json&categories=repos" | \
  jq '.results[] | select(.engines[] == "github") | {title, url}'

3. Extracting Specific Information

Very helpful for avoiding wasting tokens by adding a large amount of data to the context.

Get top 3 results:

curl -s "http://localhost:8080/search?q=rust+ownership&format=json" | \
  jq '.results[0:3] | .[] | {title, url, content}'

Check which engines returned results:

curl -s "http://localhost:8080/search?q=python&format=json" | \
  jq '.results[0].engines'

Get answer boxes/infoboxes:

curl -s "http://localhost:8080/search?q=rust+language&format=json" | \
  jq '.infoboxes, .answers'

Known Issues

• Cargo category sometimes returns empty: Try categories=packages or categories=it which also include crates.io
• Rate limiting: SearXNG may rate-limit if too many requests in quick succession

agent-browser

As a last resort, if Jina and SearXNG do not yield useful results, use the agent-browser skill and runtime.