Manage Planning Data

⚠️ Scraper controls offline. Stats and registry load from Oracle regardless. To enable scraper controls, start the local server: python3 local_pipeline/admin_server.py

👥 User Activity & Saved Leads (who saved what)

Click to load…

🔄 Manage Planning Data — Scrapers

🚀 Run All Scrapers Idle

Runs every scraper in sequence: IDOX (211 councils) + BCP → Northgate → ROI → NI → Scotland → Wales.
Uses gentle speed (1 worker) by default. Safe to close the browser tab.

IDOX mode: Recent (14 days) Recent (30 days) Full backfill Speed: 🐢 Gentle (1 worker) 🚶 Normal (2 workers) 🚀 Fast (3 workers)

Progress:

🚀 Run All Now 🌙 Tonight ⏹ Stop All

🏴󠁧󠁢󠁥󠁮󠁧󠁿 IDOX (211 councils) Idle —

Mode Last 7 days Last 14 days Last 30 days Last 60 days Full 2-year backfill ⚠ Probe portals Council Speed 🐢 Gentle (1 worker) 🚶 Normal (2 workers) 🚀 Fast (3 workers) ▶ Run now 🌙 Tonight ⏹ Stop

Log

🏖 BCP (Bournemouth, Christchurch & Poole) Idle —

Mode Last 7 days Last 30 days Full 2-year backfill ⚠ ▶ Run now ⏹ Stop

Custom ASP.NET portal — CSV export via disclaimer session. Runs automatically after IDOX.

Log

🏙 Northgate (24 councils) Idle —

Mode: Recent (14 days) Recent (30 days) Full backfill ▶ Run 🌙 Tonight ⏹ Stop

Log

🇮🇪 Republic of Ireland Idle —

Full backfill (slow) ▶ Run ⏹ Stop

Log

🇬🇧 Northern Ireland Idle —

▶ Run ⏹ Stop planningni.gov.uk — currently unreachable

Log

🏴󠁧󠁢󠁳󠁣󠁴󠁿 Scotland Idle —

Spatial Hub API key ▶ Run ⏹ Stop

Log

🏴󠁧󠁢󠁷󠁬󠁳󠁿 Wales — National Portal Idle —

▶ Run setup_wales.py ⏹ Stop Wales has its own national planning portal (planningwales.gov.uk) — separate scraper needed. Cardiff & Swansea already covered via IDOX above.

Log