zaoqu-liu/scienceclaw/browser-automation

bash: python3 -c "
try:
    import browser_use
    print('✅ browser-use installed')
except ImportError:
    print('❌ browser-use not installed')
    print('   Install: pip install browser-use')
 
import shutil
if shutil.which('chromium') or shutil.which('chromium-browser') or shutil.which('google-chrome'):
    print('✅ Chromium/Chrome found')
else:
    print('⚠️  No Chromium/Chrome found')
    print('   Install: apt-get install chromium-browser (Linux)')
    print('   Or: brew install --cask chromium (macOS)')
 
try:
    import playwright
    print('✅ Playwright installed')
except ImportError:
    print('❌ Playwright not installed')
    print('   Install: pip install playwright && python -m playwright install chromium')
"

pip install -q browser-use playwright && python -m playwright install chromium

from browser_use import Agent, Browser, BrowserConfig
from langchain_openai import ChatOpenAI
import asyncio
async def extract_gepia2_data(gene: str, cancer: str):
    """Extract gene expression data from GEPIA2 (no API available)."""
    browser = Browser(config=BrowserConfig(headless=True))
    llm = ChatOpenAI(model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"])
    agent = Agent(
        task=f"""Go to http://gepia2.cancer-pku.cn/#analysis
        1. Click on 'Expression DIY' in the left menu
        2. In the gene input box, type '{gene}'
        3. Select '{cancer}' from the cancer type dropdown
        4. Click 'Plot' button
        5. Wait for the plot to load
        6. Extract the median expression values for Tumor and Normal from the plot
        7. Return the values as JSON: {{"gene": "{gene}", "cancer": "{cancer}", "tumor_median": X, "normal_median": Y}}
        """,
        llm=llm,
        browser=browser,
    )
    result = await agent.run()
    await browser.close()
    return result
result = asyncio.run(extract_gepia2_data("THBS2", "PAAD"))
print(result)

async def collect_multi_source(gene: str):
    """Collect gene info from multiple web-only sources."""
    browser = Browser(config=BrowserConfig(headless=True))
    llm = ChatOpenAI(model="gpt-4o")
    tasks = [
        {
            "source": "GeneCards",
            "url": f"https://www.genecards.org/cgi-bin/carddisp.pl?gene={gene}",
            "extract": "Gene summary, aliases, protein class, pathways, diseases"
        },
        {
            "source": "GEPIA2",
            "url": "http://gepia2.cancer-pku.cn/#analysis",
            "extract": f"Expression of {gene} across TCGA cancer types"
        }
    ]
    results = {}
    for task in tasks:
        agent = Agent(
            task=f"Navigate to {task['url']} and extract: {task['extract']}. Return as structured JSON.",
            llm=llm,
            browser=browser,
        )
        results[task["source"]] = await agent.run()
    await browser.close()
    return results

Task: Go to http://gepia2.cancer-pku.cn/#analysis
1. Select 'Expression DIY' → 'Box Plot'
2. Enter gene symbol: {GENE}
3. Select cancer types or 'All'
4. Click Plot
5. Extract expression values from the resulting visualization

Task: Navigate to https://www.genecards.org/cgi-bin/carddisp.pl?gene={GENE}
1. Extract: Gene summary paragraph
2. Extract: Protein expression table (tissues)
3. Extract: Pathways & interactions section
4. Extract: Disorders associated section
5. Return all as structured JSON

Task: Go to http://timer.cistrome.org/
1. Select 'Gene' module
2. Enter gene symbol: {GENE}
3. Select cancer type: {CANCER}
4. Select immune cell types: all
5. Click Submit
6. Extract correlation coefficients and p-values from the result table

Task: Navigate to https://www.proteinatlas.org/{ENSEMBL_ID}-{GENE}/pathology
1. Extract cancer expression data table
2. Extract prognostic significance across cancer types
3. Extract immunohistochemistry images metadata

curl API call for [DATABASE] failed (404/no API).
Attempting browser-based extraction via browser-use...