Datasets and RAG

Datasets are tenant-scoped knowledge stores. Deeplinq persists source objects, processes them through durable jobs, stores organization-isolated vectors, and authorizes every dataset before retrieval.

Before starting, configure:

S3-compatible object storage;
Weaviate;
a kind: "embedding" connector compatible with the platform's 1,024 dimension retrieval contract;
Docling for PDF and office-document extraction;
an optional reranker connector for cross-encoder relevance scoring.

Create a dataset

curl -H "Authorization: Bearer $DEEPLINQ_API_KEY" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets" \
  -d '{"name":"HR policies","shared":false}'

Private datasets belong to their creator. Shared datasets can be governed at organization scope. Lists use opaque keyset cursors:

GET /v1/datasets?limit=50&cursor=<opaque>

Upload a file

curl -H "Authorization: Bearer $DEEPLINQ_API_KEY" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/files" \
  -F "file=@employee-handbook.pdf"

The source is stored before processing. Poll:

GET /v1/datasets/{dataset_id}/files

Status moves through:

pending → extracting → embedding → ready
                                  ↘ failed

Committed ingestion survives restarts and retries with backoff. Inspect job history and progress:

GET /v1/datasets/{dataset_id}/ingest/runs
GET /v1/datasets/{dataset_id}/ingest/runs/{run_id}

Ingest a website

curl -H "Authorization: Bearer $DEEPLINQ_API_KEY" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/resources" -d '{
  "type":"website",
  "name":"Product documentation",
  "url":"https://docs.example.com",
  "max_pages":50,
  "max_depth":2,
  "rate_limit_ms":250
}'

The crawler is same-host, depth-bounded, page-bounded, robots-aware, and protected against private, loopback, link-local, and metadata destinations. Redirect destinations are revalidated.

Schedule content refresh:

curl -H "Authorization: Bearer $DEEPLINQ_API_KEY" \
  -H "Content-Type: application/json" \
  -X PUT "$BASE_URL/v1/datasets/$DATASET_ID/ingest/schedule" \
  -d '{"interval":"1d"}'

Allowed intervals are 1h, 6h, 12h, 1d, and 7d. Content hashes avoid re-embedding unchanged pages.

One-shot retrieval

Add dataset_ids to an OpenAI-compatible chat request:

curl -H "Authorization: Bearer $DEEPLINQ_API_KEY" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/chat/completions" -d "{
  \"model\":\"gpt-5-mini\",
  \"messages\":[
    {\"role\":\"user\",\"content\":\"What is our parental leave policy?\"}
  ],
  \"dataset_ids\":[\"$DATASET_ID\"]
}"

Deeplinq combines BM25 and vector retrieval, optionally reranks candidates, screens retrieved chunks, and injects authorized context. Retrieval failure is 503; it never silently becomes plain chat.

Governed knowledge agent

When KNOWLEDGE_AGENT_ENABLED=1, a non-streaming request can ask the knowledge agent to search and read evidence iteratively:

{
  "model": "auto",
  "messages": [
    {"role": "user", "content": "Compare our parental and caregiver leave."}
  ],
  "agent_mode": "knowledge",
  "dataset_ids": ["<dataset-id>"],
  "stream": false
}

The normal completion response gains a deeplinq extension containing:

agent.status: completed, insufficient_evidence, or budget_exhausted;
citations: dataset, document, filename, and excerpt provenance.

Those three statuses are normal 200 outcomes. Runtime unavailability or a required dependency failure is 503.

Grant read, write, or admin permission to a user, role, or team:

curl -H "Authorization: Bearer $DEEPLINQ_API_KEY" \
  -H "Content-Type: application/json" \
  -X POST "$BASE_URL/v1/datasets/$DATASET_ID/grants" -d '{
  "principal_type":"team",
  "principal_id":"<team-id>",
  "permission":"read"
}'

Team membership is resolved live, so removing a member affects the next access check.

Tabular files

With the DuckDB sidecar configured, CSV and XLSX files are converted to Parquet and expose a schema summary for retrieval. Analytical questions can generate read-only DuckDB SQL. SQL, rows, execution time, memory, and result sizes are bounded; execution is interrupted after 30 seconds.

Delete safely

DELETE /v1/datasets/{id} returns 202. A durable cleanup job removes vectors, blobs, documents, and metadata. Continue polling your inventory until the dataset disappears; do not assume the response means every backing store has already been erased.

On this page