Parse a website

Crawl and index a public website starting from the given URL

1 min read

posthttps://your-instance.example.com/api/parse-website

Request example

curl --request POST \
  --url https://your-instance.example.com/api/parse-website \
  --header 'Content-Type: application/json' \
  --data '{
  "websiteUrl": "<string>",
  "projectTitle": "<string>",
  "description": "<string>",
  "maxPages": 100,
  "maxDepth": 3,
  "excludePatterns": [
    "<string>"
  ],
  "force": false
}'

Response

{
  "message": "Parse queued",
  "project": "/your-org/your-repo",
  "queueId": 5,
  "position": 1
}

posthttps://your-instance.example.com/api/parse-website

Body

application/json

websiteUrlstringrequired#

Root URL to start crawling from

projectTitlestring#

Display name for the project

descriptionstring#

Short description shown in the library list

maxPagesinteger#

Maximum number of pages to crawl

Default: 100

maxDepthinteger#

Maximum link depth from the root URL

Default: 3

excludePatternsstring[]#

URL path patterns to skip (e.g., /blog/*)

forceboolean#

Re-crawl even if already indexed

Default: false

Responses

application/json

Parse job accepted and queued

messagestringrequired#

projectstringrequired#

Project identifier assigned to this library (e.g. /your-org/your-repo)

queueIdintegerrequired#

Numeric ID for this parse job

positioninteger#

Position in the queue. null if the job started immediately

Import a library bundle Refresh a library