# Website Sync

**Website Sync** lets you import content directly from your company website, help center, or online catalog into a [Collection](/knowledge-base/collections/) without manual copy-pasting.

:::note[Required permission]
You need **`ai:manage`** or **`workspace:manage`** to discover, import, edit, or resync websites in the Knowledge Base.
:::

## 2-Step Website Import

1. Open a collection under **Build → Knowledge** (`/{slug}/ai-kb/{collectionId}`), click the add button, and select **Sync Website**.
2. **Step 1 — Paste website URL**:
   - Choose the target **Collection** (if not preselected).
   - Enter a valid `http://` or `https://` URL.
   - Start discovery to scan linked pages (default discovery limit: up to **1,000 pages**).
3. **Step 2 — Review & select pages**:
   - Discovered URLs are automatically organized into standalone pages and path-based folders.
   - Customize the **display title** and **icon/avatar**, then check the exact pages you want to import.
   - Confirm import to enqueue all selected pages for background processing.

:::caution[Duplicate domains & security restrictions]
- Each domain (normalized without `www.`) can only be added **once per collection**. To refresh an existing domain, open its detail page and resync.
- For SSRF protection, URLs targeting `localhost`, `.local`, or private IP ranges (`127.x`, `10.x`, `192.168.x`, `172.16-31.x`, `169.254.x`) are blocked.
:::

## Background Crawl & Indexing Pipeline

Imported and resynced pages run asynchronously through two stages:

1. **Stage 1 — Fetch & parse (`CRAWLING`)**: The worker fetches each URL (throttled to at least **600 ms** between requests to the same domain) and converts HTML into clean Markdown (stripping `script`, `style`, `iframe`, and `svg` elements). Pages with fewer than 20 characters of extracted text or requests exceeding the 8-second timeout are marked as failed.
2. **Stage 2 — Vector indexing (`WAITING_INDEX` → `INDEXING` → `READY`)**: Content is hashed with **SHA-256**. If a page's content is unchanged and already `READY`, vector re-indexing is skipped automatically.

## Manage & Resync Websites

Open a website's detail page (`/{slug}/ai-kb/{collectionId}/website-sync/{websiteId}` or from inside a bot's Knowledge tab) to manage its synced pages:

- **Aggregated website status**: Automatically derived from child pages — **`SYNCING`** while any page is in `PENDING`, `CRAWLING`, `SYNCING`, `WAITING_INDEX`, or `INDEXING`; **`FAILED`** if any page errored; and **`READY`** when all pages are ready.
- **Resync a page or the entire site**: Trigger a resync on a single page (`POST /kb/{id}/resync`) or the whole website (`POST /kb/websites/{id}/resync`) whenever your live site changes.
- **Toggle page availability**: Use the active switch (`isActive`) to exclude specific pages from AI retrieval without deleting them.
- **Rename or delete**: Update the website's display title (the root domain is read-only) or delete the website and its synced pages.

:::note[Read-only sources]
**Web Page (`WEB_PAGE`)** sources are read-only in the detail viewer so they stay faithful to the live URL. To update their content, edit your website and click **Resync**.
:::