add_page_to_job
Register an individual page into an existing cloning job
Overview
Register a single source/clone URL pair into an existing cloning job when a page wasn't found during discover_all_pages. Common cases include dynamically routed pages, pages behind authentication, crawler misses, or pages added to the site after initial discovery. Upserts cleanly — calling it again on an already-tracked page updates the clone_url without duplicating the row.
How It Works
- Looks up the most recent cloning job for the current user that matches the source_url's domain.
- If clone_url is omitted and the job has a clone_domain, the clone URL is derived automatically from the source path.
- Both URLs are validated and SSRF-guarded before any database write.
- Upserts the page row: insert if new, update clone_url if already tracked.
- If queue_comparison: true, immediately queues the page for background comparison — same path as compare_page_pair.
- Returns the registered page_id, resolved clone_url, and whether a comparison was queued.
Input Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
source_url |
string |
required | URL of the source (original) page to register |
clone_url |
string |
optional | URL of the cloned page. If omitted and the job has a clone_domain, it is derived automatically from the source path. |
queue_comparison |
boolean |
optional | If true, immediately queue the page for comparison after registering it (default: false) |
What You Get Back
- page_id — the database ID of the registered page
- job_id — the cloning job the page was added to
- clone_url — the resolved clone URL (whether provided or derived)
- clone_url_derived — true if the clone URL was derived from the job's clone_domain
- comparison_queued — whether a comparison was immediately queued
- comparison_status — 'queued' or 'already_queued' when queue_comparison is true
- next_action — guidance on what to call next
Example Use Case
Your discover_all_pages run missed a React-rendered product page that requires JavaScript interaction to reach. Call add_page_to_job with the source URL and queue_comparison: true — the page is registered and immediately queued for analysis without re-crawling the entire site.
Tips
Use this instead of a full re-crawl when just a handful of pages were missed.
Omit clone_url if the job already has a clone_domain — the tool maps the source path onto the clone domain automatically.
Calling it on an already-tracked page is safe — it upserts, so no duplicate rows are created.
After registering, you can queue the page immediately with queue_comparison: true, or let run_full_comparison pick it up on the next pass.
Use discover_all_pages when most pages are missing; use this tool for targeted individual-page registration.