Source connectors

What is implemented

The ingest worker implements six source families: Amazon S3 (s3), Google Cloud Storage (google_cloud_storage), Azure Blob Storage (azure_blob_storage), Google Drive (google_drive), PostgreSQL (postgresql), and MySQL (mysql).

Registration, a successful read probe, and a queued ingest are different. In Extraction Studio and workflow import steps, you can import from your saved connections. The manual HTTP trigger POST /api/w/:workspaceId/connections/:connectionId/source-ingest accepts s3 and postgresql. SharePoint, Gmail, Confluence, and older marketing lists of many connectors are not implemented source setups.

Amazon S3

Source fields: bucketUri (required), prefix, region, recursive (default true), customEndpointUrl. Auth is access keys (accessKeyId, secretAccessKey, optional sessionToken), or the public anonymous mode when the bucket policy allows it. Prefer a prefix-scoped read policy. Do not use an account-wide admin key.

A read or list check means the worker could list or fetch the scope you configured. It does not grant destination write, and it does not mean objects were copied into a collection.

Google Cloud Storage

Source fields: bucketName (required), prefix, recursive (default true), and secret serviceAccountKeyJson. Grant the service account object read on that bucket and prefix only. A container access check is the spec validation for this connector. It is not permission to write, and it is not the S3/PostgreSQL ingest route.

Azure Blob Storage

Source fields: accountName, containerName, prefix, recursive (default true). Secret depends on auth: accountKey, sasToken, or connectionString. Scope the SAS to read and list on one container. container_access_check is a read of that container, not a write.

Google Drive

Open Add Data → Google Drive from a signed-in browser session in a stored workspace (requires deployment connector phase 2+ and Google OAuth configured):

• Read-only OAuth with PKCE: Requests only https://www.googleapis.com/auth/drive.readonly with offline access in a top-level window returning to /auth/google-drive/callback. Refresh tokens are stored encrypted on the connection (refreshToken); access tokens stay in memory only and never appear in logs or responses.
• Folders and files: Choose up to 20 folders in My Drive or shared drives, with optional includeSubfolders (up to 20 levels deep). Google Docs arrive as .docx, Google Sheets as .xlsx, and Google Slides as .pdf.
• Bounds and sync: Up to 25 MiB per downloaded file and 10 MB per exported Google file, up to 500 new files per import run, and continuation jobs up to 50 listing pages. Renamed or moved files update in place by Drive file ID. If access is revoked at Google, the connection shows Reconnect (must use the same Google account).

PostgreSQL and MySQL

PostgreSQL source fields: host, port, dbName, username, password, optional schemaName (default public), and required tableName. Use a read-only database user on that table.

MySQL source fields: host, port (default 3306), database, username, password, ssl (default false), and optional tableName. You can also read metadata schemas for PostgreSQL and MySQL connections in Context Studio → Data.

The HTTP source-ingest body is { collectionId, maxObjects?, prefix?, progressiveDiscovery? }. maxObjects is an integer from 1 to 50. The response is { accepted: true, jobType: "connection.source.ingest", jobId, summary } when the job is queued. jobId null becomes a service-unavailable error. Queued is not inserted. Poll the collection documents for new rows. Database ingest stores row JSON as extracted text. It does not prove a destination write.

Recovery

If the test stays failed, check host reachability, the prefix or table name, and that the secret matches the auth method you selected. Rotate the secret on the connection. Do not paste it into a ticket. If a Google Drive connection reports source_reauthorization_required, click Reconnect in Add Data or Settings → Connections with the same Google account. A failed read does not delete documents already in the collection.