Connect to External Knowledge

Integrate external knowledge sources with ClickAI applications through API connections to leverage custom RAG systems or third-party knowledge services.

Table of Contents

· [Overview](#overview)

· [Step 1: Build the Retrieval API](#step-1-build-the-retrieval-api)

· [Step 2: Register an External Knowledge API](#step-2-register-an-external-knowledge-api)

· [Step 3: Create an External Knowledge Base](#step-3-create-an-external-knowledge-base)

· [Troubleshoot](#troubleshoot)

Overview

Beyond creating Knowledge Bases directly on ClickAI, you can connect to external knowledge sources — for example:

· Custom-built RAG systems

· AWS Bedrock Knowledge Base

· Vector databases: Pinecone, Weaviate, Qdrant, Milvus

· Third-party knowledge services

The connection process has 3 steps:

1. Build an API service that ClickAI can query

2. Register the API endpoint in ClickAI

3. Connect a specific knowledge source through the registered API

Step 1: Build the Retrieval API

You need to build an API service that complies with ClickAI's External Knowledge API specification.

Endpoint format:

POST {your-endpoint}/retrieval Content-Type: application/json Authorization: Bearer {API_KEY}

Request body:

{ "knowledge_id": "your-knowledge-id", "query": "User question", "retrieval_setting": { "top_k": 3, "score_threshold": 0.5 } }

Response format:

{ "records": [ { "content": "Retrieved chunk content.", "score": 0.98, "title": "document.txt", "metadata": { "path": "s3://bucket/document.txt", "description": "Document description" } } ] }

⚠️ IMPORTANT: The `metadata` field in each record must be an object (`{}`), not `null`. A `null` value causes errors in the retrieval pipeline. The `content` and `score` fields must be present in every record.

Step 2: Register an External Knowledge API

4. Go to Knowledge, click External Knowledge API in the upper-right corner

5. Click Add an External Knowledge API

6. Fill in the fields:

Field	Description
Name	A label to distinguish this API connection from others
API Endpoint	The base URL of your external knowledge service. ClickAI appends /retrieval automatically when sending requests
API Key	The authentication credential for your service. ClickAI sends this as a Bearer token in the Authorization header

Step 3: Create an External Knowledge Base

7. Go to Knowledge and click Connect to an External Knowledge Base

8. Fill in the fields:

· External Knowledge Name and Knowledge Description (optional)

· External Knowledge API: Select the API connection you registered

· External Knowledge ID: The identifier of the specific knowledge source within your external system, passed to your API as the knowledge_id field

🚨 WARNING: The External Knowledge API and External Knowledge ID **cannot be changed** after creation. To use a different API or knowledge source, create a new external knowledge base.

9. Configure Retrieval Settings:

Setting	Description
Top K	Maximum number of chunks to retrieve per query. Higher values return more results but may include less relevant content
Score Threshold	Minimum similarity score for returned chunks. Enable this to filter out low-relevance results. When disabled, all results up to the Top K limit are returned

Troubleshoot

Connection Refused or Timeout (Self-Hosted)

If you're self-hosting ClickAI, check the allowed_domains configuration in docker/ssrf_proxy/squid.conf.template:

acl allowed_domains dstdomain .marketplace.dify.ai .your-kb-service.com

Add your external knowledge service domain to the allowed list.

API Response Format Issues

Ensure response format complies with the specification:

· The metadata field in each record must be an object ({}), not null

· The content and score fields are required in every record

📖 Previous: [Knowledge Pipeline](./05-knowledge-pipeline.md) · Next: [External Knowledge API Spec](./07-external-knowledge-api.md)