Backend Issues

Hardcoded Debug File Paths (CRITICAL)

Previous AI session added open(r"c:\Users\Ibrah\...\debug.log") in service.py. Path doesn't exist → FileNotFoundError → 500 on every scrape. ROOT CAUSE of "jobs not appearing in UI" (March 2026). Fix: Use logging module.

Proxy Pool Mode Mismatch

Webshare residential proxies but code used mode=direct (datacenter) → HTTP 400. Fix: mode=backbone.

CSRF Breaks Incognito

credentials: "include" in CSRF fetch triggers cross-origin restrictions in incognito. Fix: Removed — CSRF is header-based, no cookies needed.

Render 60-Min Timeout

Long jobs killed. Needs RQ Workers for async processing.

Scraping Job Timeout Issue

DB pool recycles every 5 min (pool_recycle=300). Using Transaction pooler (6543) instead of Session pooler (5432).